2025-09-16 17:31:41,016 INFO StreamThr :539590 [internal.py:wandb_internal():86] W&B internal server running at pid: 539590, started at: 2025-09-16 17:31:41.014224 2025-09-16 17:31:41,017 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status 2025-09-16 17:31:41,048 INFO WriterThread:539590 [datastore.py:open_for_write():85] open: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/run-78ke0sc9.wandb 2025-09-16 17:31:41,050 DEBUG SenderThread:539590 [sender.py:send():382] send: header 2025-09-16 17:31:46,051 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:31:46,098 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:31:51,051 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:31:51,098 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:31:56,052 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:31:56,100 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:32:01,053 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:32:01,101 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:32:06,054 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:32:06,102 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:32:09,740 DEBUG SenderThread:539590 [sender.py:send():382] send: run 2025-09-16 17:32:10,385 INFO SenderThread:539590 [dir_watcher.py:__init__():211] watching files in: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files 2025-09-16 17:32:10,385 INFO SenderThread:539590 [sender.py:_start_run_threads():1136] run started: 78ke0sc9 with start time 1758043901.013404 2025-09-16 17:32:10,504 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: check_version 2025-09-16 17:32:10,504 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: check_version 2025-09-16 17:32:10,669 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: run_start 2025-09-16 17:32:11,032 DEBUG HandlerThread:539590 [system_info.py:__init__():32] System info init 2025-09-16 17:32:11,032 DEBUG HandlerThread:539590 [system_info.py:__init__():47] System info init done 2025-09-16 17:32:11,032 INFO HandlerThread:539590 [system_monitor.py:start():194] Starting system monitor 2025-09-16 17:32:11,032 INFO SystemMonitor:539590 [system_monitor.py:_start():158] Starting system asset monitoring threads 2025-09-16 17:32:11,033 INFO HandlerThread:539590 [system_monitor.py:probe():214] Collecting system info 2025-09-16 17:32:11,033 INFO SystemMonitor:539590 [interfaces.py:start():190] Started cpu monitoring 2025-09-16 17:32:11,034 INFO SystemMonitor:539590 [interfaces.py:start():190] Started disk monitoring 2025-09-16 17:32:11,034 INFO SystemMonitor:539590 [interfaces.py:start():190] Started gpu monitoring 2025-09-16 17:32:11,037 INFO SystemMonitor:539590 [interfaces.py:start():190] Started memory monitoring 2025-09-16 17:32:11,038 INFO SystemMonitor:539590 [interfaces.py:start():190] Started network monitoring 2025-09-16 17:32:11,077 DEBUG HandlerThread:539590 [system_info.py:probe():196] Probing system 2025-09-16 17:32:11,084 DEBUG HandlerThread:539590 [system_info.py:_probe_git():181] Probing git 2025-09-16 17:32:11,137 DEBUG HandlerThread:539590 [system_info.py:_probe_git():189] Probing git done 2025-09-16 17:32:11,137 DEBUG HandlerThread:539590 [system_info.py:probe():244] Probing system done 2025-09-16 17:32:11,137 DEBUG HandlerThread:539590 [system_monitor.py:probe():223] {'os': 'Linux-5.14.21-150500.55.97_13.0.78-cray_shasta_c-x86_64-with-glibc2.35', 'python': '3.10.12', 'heartbeatAt': '2025-09-16T17:32:11.078065', 'startedAt': '2025-09-16T17:31:40.977557', 'docker': None, 'cuda': None, 'args': ('--yaml_config=./config/sfnonet.yaml', '--config=multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2', '--run_num=v0.1.0-seed102', '--amp_mode=bf16', '--checkpointing_level=0', '--multistep_count=2', '--h_parallel_size=4', '--w_parallel_size=1', '--load_checkpoint=legacy', '--batch_size=4'), 'state': 'running', 'program': '-m makani.train', 'codePathLocal': None, 'git': {'remote': 'git@github.com:ankurmahesh/modulus-makani-fork.git', 'commit': '4b7dcddea9084b60c41957440a9f3e14f7d2567d'}, 'email': 'mahesh.ankur10@gmail.com', 'root': '/global/u2/a/amahesh/ms_finetune/modulus-makani-fork', 'host': 'nid008421', 'username': 'amahesh', 'executable': '/usr/bin/python', 'cpu_count': 64, 'cpu_count_logical': 128, 'cpu_freq': {'current': 2694.737874999999, 'min': 1500.0, 'max': 2450.0}, 'cpu_freq_per_core': [{'current': 3142.623, 'min': 1500.0, 'max': 2450.0}, {'current': 3141.23, 'min': 1500.0, 'max': 2450.0}, {'current': 3142.565, 'min': 1500.0, 'max': 2450.0}, {'current': 3102.305, 'min': 1500.0, 'max': 2450.0}, {'current': 3144.082, 'min': 1500.0, 'max': 2450.0}, {'current': 2769.752, 'min': 1500.0, 'max': 2450.0}, {'current': 3141.75, 'min': 1500.0, 'max': 2450.0}, {'current': 3092.947, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3142.221, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3143.176, 'min': 1500.0, 'max': 2450.0}, {'current': 3082.769, 'min': 1500.0, 'max': 2450.0}, {'current': 2617.393, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3018.254, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2683.336, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3140.905, 'min': 1500.0, 'max': 2450.0}, {'current': 2963.543, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3141.599, 'min': 1500.0, 'max': 2450.0}, {'current': 2890.51, 'min': 1500.0, 'max': 2450.0}, {'current': 3141.122, 'min': 1500.0, 'max': 2450.0}, {'current': 3141.733, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3141.444, 'min': 1500.0, 'max': 2450.0}, {'current': 2513.054, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3141.439, 'min': 1500.0, 'max': 2450.0}, {'current': 3141.506, 'min': 1500.0, 'max': 2450.0}, {'current': 2790.106, 'min': 1500.0, 'max': 2450.0}, {'current': 2877.487, 'min': 1500.0, 'max': 2450.0}, {'current': 2955.303, 'min': 1500.0, 'max': 2450.0}, {'current': 2620.254, 'min': 1500.0, 'max': 2450.0}, {'current': 3112.111, 'min': 1500.0, 'max': 2450.0}, {'current': 2912.329, 'min': 1500.0, 'max': 2450.0}, {'current': 2869.11, 'min': 1500.0, 'max': 2450.0}, {'current': 3143.166, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3044.707, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2316.985, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2681.425, 'min': 1500.0, 'max': 2450.0}, {'current': 3142.846, 'min': 1500.0, 'max': 2450.0}, {'current': 2870.104, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2923.822, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3143.402, 'min': 1500.0, 'max': 2450.0}, {'current': 3064.286, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2879.017, 'min': 1500.0, 'max': 2450.0}, {'current': 2891.889, 'min': 1500.0, 'max': 2450.0}, {'current': 3004.548, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3137.829, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3106.768, 'min': 1500.0, 'max': 2450.0}, {'current': 3141.881, 'min': 1500.0, 'max': 2450.0}, {'current': 3140.778, 'min': 1500.0, 'max': 2450.0}, {'current': 3124.846, 'min': 1500.0, 'max': 2450.0}, {'current': 3139.964, 'min': 1500.0, 'max': 2450.0}, {'current': 3120.739, 'min': 1500.0, 'max': 2450.0}, {'current': 3125.623, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3142.094, 'min': 1500.0, 'max': 2450.0}, {'current': 3026.78, 'min': 1500.0, 'max': 2450.0}, {'current': 3141.629, 'min': 1500.0, 'max': 2450.0}, {'current': 3119.451, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3140.957, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 3139.433, 'min': 1500.0, 'max': 2450.0}, {'current': 2512.106, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}, {'current': 2450.0, 'min': 1500.0, 'max': 2450.0}], 'disk': {'/': {'total': 125.5986557006836, 'used': 0.5811920166015625}}, 'gpu': 'NVIDIA A100-SXM4-80GB', 'gpu_count': 4, 'gpu_devices': [{'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}, {'name': 'NVIDIA A100-SXM4-80GB', 'memory_total': 85899345920}], 'memory': {'total': 251.1973114013672}} 2025-09-16 17:32:11,137 INFO HandlerThread:539590 [system_monitor.py:probe():224] Finished collecting system info 2025-09-16 17:32:11,137 INFO HandlerThread:539590 [system_monitor.py:probe():227] Publishing system info 2025-09-16 17:32:11,137 DEBUG HandlerThread:539590 [system_info.py:_save_pip():52] Saving list of pip packages installed into the current environment 2025-09-16 17:32:11,139 DEBUG HandlerThread:539590 [system_info.py:_save_pip():68] Saving pip packages done 2025-09-16 17:32:11,389 INFO Thread-12 :539590 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/wandb-metadata.json 2025-09-16 17:32:11,389 INFO Thread-12 :539590 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/requirements.txt 2025-09-16 17:32:22,525 INFO HandlerThread:539590 [system_monitor.py:probe():229] Finished publishing system info 2025-09-16 17:32:22,537 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:32:22,537 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:32:22,537 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:32:22,537 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:32:22,538 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:32:22,538 DEBUG SenderThread:539590 [sender.py:send():382] send: files 2025-09-16 17:32:22,538 INFO SenderThread:539590 [sender.py:_save_file():1392] saving file wandb-metadata.json with policy now 2025-09-16 17:32:22,552 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:32:22,552 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:32:22,552 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:32:22,831 DEBUG SenderThread:539590 [sender.py:send():382] send: telemetry 2025-09-16 17:32:23,291 DEBUG SenderThread:539590 [sender.py:send():382] send: use_artifact 2025-09-16 17:32:23,394 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/wandb-metadata.json 2025-09-16 17:32:23,394 INFO Thread-12 :539590 [dir_watcher.py:_on_file_created():271] file/dir created: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:32:25,395 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:32:27,395 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:32:27,404 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:32:32,405 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:32:37,405 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:32:37,551 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:32:37,552 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:32:37,552 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:32:42,756 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:32:43,402 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/config.yaml 2025-09-16 17:32:48,100 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:32:51,405 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:32:52,551 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:32:52,552 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:32:52,552 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:32:53,898 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:32:58,899 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:33:03,900 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:33:07,551 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:33:07,552 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:33:07,594 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:33:09,717 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:33:11,039 DEBUG SystemMonitor:539590 [system_monitor.py:_start():172] Starting system metrics aggregation loop 2025-09-16 17:33:11,043 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:33:13,414 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:33:15,642 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:33:20,334 DEBUG SenderThread:539590 [sender.py:send():382] send: telemetry 2025-09-16 17:33:20,755 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:33:21,418 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:33:22,552 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:33:22,552 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:33:22,553 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:33:23,419 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:33:26,703 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:33:29,421 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:33:31,765 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:33:36,765 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:33:37,552 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:33:37,552 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:33:37,553 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:33:39,426 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:33:41,048 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:33:41,427 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:33:42,049 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:33:43,428 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:33:45,429 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:33:47,300 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:33:47,429 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:33:48,431 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/config.yaml 2025-09-16 17:33:49,432 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:33:51,433 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:33:52,552 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:33:52,553 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:33:52,553 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:33:52,709 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:33:53,434 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:33:55,434 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:33:57,435 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:33:58,512 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:33:59,436 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:01,437 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:03,438 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:03,627 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:34:05,438 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:07,439 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:07,553 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:34:07,553 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:34:07,554 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:34:08,753 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:34:09,440 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:11,049 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:34:11,441 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:13,442 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:13,894 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:34:15,443 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:17,444 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:19,000 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:34:19,444 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:21,445 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:22,553 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:34:22,553 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:34:22,554 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:34:23,446 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:24,123 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:34:25,447 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:27,448 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:29,259 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:34:29,449 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:31,449 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:33,148 INFO wandb-upload_0:539590 [upload_job.py:push():131] Uploaded file /tmp/tmpkeab4k9vwandb/t9b6ypxb-wandb-metadata.json 2025-09-16 17:34:34,393 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:34:34,451 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:36,452 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:37,553 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:34:37,553 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:34:37,555 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:34:38,453 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:39,513 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:34:40,454 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:41,052 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:34:42,455 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:44,456 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:44,632 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:34:46,457 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:48,458 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:49,764 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:34:50,459 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:52,459 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:52,553 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:34:52,553 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:34:52,555 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:34:54,460 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:54,882 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:34:56,461 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:34:58,462 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:00,003 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:35:00,463 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:02,464 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:04,464 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:05,139 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:35:06,465 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:07,553 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:35:07,553 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:35:07,556 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:35:08,466 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:10,272 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:35:10,467 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:11,056 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:35:12,468 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:14,469 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:15,398 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:35:16,470 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:18,471 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:20,472 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:20,507 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:35:22,474 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:22,553 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:35:22,554 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:35:22,556 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:35:24,474 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:25,628 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:35:26,475 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:28,476 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:30,477 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:30,762 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:35:32,478 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:34,479 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:35,896 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:35:36,480 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:37,553 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:35:37,553 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:35:37,555 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:35:38,481 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:40,482 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:41,008 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:35:41,061 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:35:42,483 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:44,484 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:46,130 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:35:46,485 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:48,486 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:50,487 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:51,243 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:35:52,488 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:52,553 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:35:52,554 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:35:52,557 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:35:54,489 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:56,349 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:35:56,490 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:35:58,491 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:00,492 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:01,476 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:36:02,492 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:04,493 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:06,494 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:06,594 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:36:07,554 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:36:07,554 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:36:07,556 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:36:08,495 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:10,496 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:11,095 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:36:11,996 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:36:12,497 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:14,498 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:16,499 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:17,108 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:36:18,500 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:20,501 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:22,239 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:36:22,501 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:22,553 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:36:22,554 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:36:22,556 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:36:24,503 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:26,504 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:27,378 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:36:28,504 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:30,505 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:32,484 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:36:32,506 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:34,507 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:36,508 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:37,554 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:36:37,555 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:36:37,556 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:36:37,728 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:36:38,509 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:40,510 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:41,098 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:36:42,511 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:42,729 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:36:44,512 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:46,513 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:47,853 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:36:48,514 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:50,514 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:52,516 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:52,554 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:36:52,555 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:36:52,555 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:36:52,977 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:36:54,516 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:56,517 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:36:58,101 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:36:58,518 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:00,519 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:02,520 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:03,222 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:37:04,521 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:06,522 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:07,554 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:37:07,554 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:37:07,556 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:37:08,344 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:37:08,523 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:10,524 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:11,101 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:37:12,525 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:13,469 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:37:14,526 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:16,527 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:18,528 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:18,589 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:37:20,529 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:22,530 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:22,554 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:37:22,555 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:37:22,555 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:37:23,717 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:37:24,531 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:26,532 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:28,533 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:28,841 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:37:30,534 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:32,534 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:33,952 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:37:34,535 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:36,536 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:37,554 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:37:37,554 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:37:37,557 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:37:38,537 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:39,062 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:37:40,538 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:41,106 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:37:42,539 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:44,178 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:37:44,540 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:46,541 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:48,542 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:49,296 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:37:50,543 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:52,544 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:52,554 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:37:52,555 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:37:52,555 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:37:54,403 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:37:54,545 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:56,546 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:58,547 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:37:59,534 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:38:00,548 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:02,549 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:04,550 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:04,664 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:38:06,551 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:07,555 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:38:07,556 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:38:07,556 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:38:08,552 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:09,772 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:38:10,553 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:11,107 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:38:12,554 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:14,554 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:14,897 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:38:16,555 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:18,556 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:20,013 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:38:20,557 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:22,555 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:38:22,555 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:38:22,556 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:38:22,558 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:24,559 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:25,145 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:38:26,560 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:28,561 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:30,261 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:38:30,562 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:32,563 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:34,564 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:35,393 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:38:36,565 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:37,555 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:38:37,555 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:38:37,556 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:38:38,565 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:40,510 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:38:40,566 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:41,110 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:38:42,567 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:44,568 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:45,630 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:38:46,569 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:48,570 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:50,571 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:50,745 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:38:52,555 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:38:52,556 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:38:52,556 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:38:52,572 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:54,572 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:55,855 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:38:56,574 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:38:58,574 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:39:00,575 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:39:00,984 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:39:02,576 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:39:04,577 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:39:06,090 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:39:06,578 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:39:07,555 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:39:07,555 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:39:07,556 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:39:08,579 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:39:10,580 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:39:13,256 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:39:13,256 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:39:14,581 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:39:16,582 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:39:18,583 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:39:18,608 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:39:20,584 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:39:22,555 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:39:22,556 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:39:22,556 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:39:22,585 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:39:24,586 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:39:28,557 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:39:33,558 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:39:34,118 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:39:36,591 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:39:37,555 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: stop_status 2025-09-16 17:39:37,556 DEBUG SenderThread:539590 [sender.py:send_request():409] send_request: stop_status 2025-09-16 17:39:37,557 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:39:38,592 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:39:43,557 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:39:48,558 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:39:52,556 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:39:56,740 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:39:56,741 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:39:57,560 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:39:58,601 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:00,602 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:02,062 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:40:02,561 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:40:02,602 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:04,603 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:06,604 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:07,161 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:40:07,557 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:40:08,605 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:10,606 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:11,119 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:40:12,284 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:40:12,562 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:40:12,607 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:14,608 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:16,609 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:17,399 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:40:17,563 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:40:18,609 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:20,610 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:22,511 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:40:22,557 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:40:22,611 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:24,612 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:26,613 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:27,565 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:40:27,620 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:40:28,614 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:30,615 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:32,566 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:40:32,616 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:32,740 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:40:34,616 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:36,617 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:37,557 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:40:37,850 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:40:38,618 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:40,619 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:41,121 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:40:42,567 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:40:42,620 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:42,972 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:40:44,621 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:46,622 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:47,568 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:40:48,083 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:40:48,623 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:50,623 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:52,557 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:40:52,624 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:53,199 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:40:54,625 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:56,626 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:40:57,570 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:40:58,306 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:40:58,627 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:00,628 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:02,571 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:41:02,628 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:03,416 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:41:04,629 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:06,630 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:07,557 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:41:08,531 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:41:08,631 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:10,632 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:11,126 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:41:12,572 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:41:12,633 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:13,648 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:41:14,634 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:16,634 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:17,573 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:41:18,636 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:18,769 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:41:20,637 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:22,557 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:41:22,638 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:23,887 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:41:24,638 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:26,639 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:27,575 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:41:28,640 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:28,994 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:41:30,641 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:32,575 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:41:32,642 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:34,124 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:41:34,643 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:36,643 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:37,557 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:41:38,644 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:39,249 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:41:40,645 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:41,127 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:41:42,576 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:41:42,646 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:44,358 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:41:44,647 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:46,648 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:47,578 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:41:48,649 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:49,479 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:41:50,650 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:52,557 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:41:52,651 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:54,602 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:41:54,652 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:56,652 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:57,579 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:41:58,653 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:41:59,730 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:42:00,654 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:02,580 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:42:02,656 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:04,657 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:04,849 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:42:06,658 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:07,557 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:42:08,658 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:09,984 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:42:10,659 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:11,130 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:42:12,581 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:42:12,660 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:14,661 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:15,102 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:42:16,662 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:17,582 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:42:18,663 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:20,226 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:42:20,664 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:22,557 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:42:22,665 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:24,666 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:25,342 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:42:26,667 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:27,584 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:42:28,669 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:30,465 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:42:30,671 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:32,585 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:42:32,673 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:34,675 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:35,599 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:42:36,677 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:37,558 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:42:38,679 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:40,681 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:40,712 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:42:41,133 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:42:42,586 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:42:42,682 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:44,683 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:45,836 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:42:46,684 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:47,587 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:42:48,685 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:50,686 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:50,962 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:42:52,558 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:42:52,687 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:54,687 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:56,096 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:42:56,688 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:42:57,589 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:42:58,689 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:00,690 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:01,215 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:43:02,590 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:43:02,691 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:04,692 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:06,331 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:43:06,692 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:07,558 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:43:08,693 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:10,694 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:11,138 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:43:11,752 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:43:12,591 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:43:12,695 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:14,696 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:16,697 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:16,871 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:43:17,592 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:43:18,698 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:20,699 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:21,995 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:43:22,558 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:43:22,700 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:24,701 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:26,702 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:27,118 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:43:27,594 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:43:28,702 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:30,703 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:32,245 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:43:32,595 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:43:32,704 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:34,705 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:36,706 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:37,374 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:43:37,558 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:43:38,707 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:40,708 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:41,138 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:43:42,485 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:43:42,596 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:43:42,709 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:44,710 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:46,710 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:47,597 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:43:47,674 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:43:48,711 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:50,712 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:52,558 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:43:52,713 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:52,806 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:43:54,714 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:56,714 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:43:57,599 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:43:57,933 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:43:58,715 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:00,716 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:02,600 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:44:02,717 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:03,064 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:44:04,718 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:06,719 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:07,558 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:44:08,192 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:44:08,720 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:10,721 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:11,142 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:44:12,601 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:44:12,722 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:13,314 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:44:14,722 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:16,723 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:17,602 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:44:18,429 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:44:18,724 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:20,725 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:22,558 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:44:22,726 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:23,565 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:44:24,727 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:26,728 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:27,604 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:44:28,680 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:44:28,729 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:30,730 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:32,605 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:44:32,730 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:33,800 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:44:34,732 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:36,733 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:37,558 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:44:38,734 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:38,914 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:44:40,734 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:41,145 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:44:42,606 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:44:42,735 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:44,034 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:44:44,736 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:46,737 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:47,607 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:44:48,738 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:49,165 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:44:50,739 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:52,559 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:44:52,740 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:54,292 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:44:54,741 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:56,742 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:57,609 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:44:58,743 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:44:59,414 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:45:00,744 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:02,609 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:45:02,745 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:04,534 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:45:04,746 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:06,747 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:07,559 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:45:08,748 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:09,658 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:45:10,749 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:11,148 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:45:12,611 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:45:12,749 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:14,750 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:14,784 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:45:16,751 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:17,612 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:45:18,752 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:19,913 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:45:20,753 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:22,559 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:45:22,754 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:24,755 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:25,034 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:45:26,756 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:27,613 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:45:28,756 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:30,147 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:45:30,758 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:32,614 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:45:32,759 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:34,759 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:35,279 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:45:36,760 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:37,559 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:45:38,762 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:40,393 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:45:40,763 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:41,151 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:45:42,616 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:45:42,764 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:44,764 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:45,522 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:45:46,765 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:47,617 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:45:48,766 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:50,635 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:45:50,767 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:52,559 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:45:52,768 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:54,769 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:55,751 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:45:56,770 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:45:57,618 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:45:58,771 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:00,771 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:00,867 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:46:02,619 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:46:02,772 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:04,773 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:05,994 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:46:06,774 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:07,559 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:46:08,775 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:10,776 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:11,118 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:46:11,154 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:46:12,621 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:46:12,777 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:14,778 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:16,235 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:46:16,779 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:17,622 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:46:18,780 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:20,781 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:21,357 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:46:22,559 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:46:22,781 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:24,782 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:26,457 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:46:26,783 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:27,624 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:46:28,784 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:30,785 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:31,577 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:46:32,625 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:46:32,786 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:34,787 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:36,701 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:46:36,788 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:37,559 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:46:38,788 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:40,790 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:41,158 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:46:41,825 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:46:42,627 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:46:42,791 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:44,792 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:46,793 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:46,969 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:46:47,628 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:46:48,794 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:50,795 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:52,090 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:46:52,560 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:46:52,795 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:54,797 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:56,797 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:46:57,221 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:46:57,629 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:46:58,798 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:00,799 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:02,343 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:47:02,630 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:47:02,800 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:04,801 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:06,801 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:07,462 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:47:07,560 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:47:08,802 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:10,803 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:11,160 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:47:12,570 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:47:12,631 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:47:12,804 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:14,805 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:16,806 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:17,633 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:47:17,693 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:47:18,807 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:20,808 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:22,560 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:47:22,808 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:22,814 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:47:24,809 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:26,810 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:27,634 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:47:27,926 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:47:28,811 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:30,812 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:32,635 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:47:32,813 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:33,044 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:47:34,814 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:36,815 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:37,560 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:47:38,167 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:47:38,816 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:40,816 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:41,163 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:47:42,637 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:47:42,817 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:43,295 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:47:44,818 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:46,819 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:47,638 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:47:48,414 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:47:48,820 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:50,821 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:52,560 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:47:52,822 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:53,525 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:47:54,822 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:56,823 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:47:57,640 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:47:58,655 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:47:58,824 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:00,825 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:02,641 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:48:02,826 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:03,762 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:48:04,827 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:06,827 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:07,560 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:48:08,828 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:08,878 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:48:10,829 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:11,165 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:48:12,643 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:48:12,830 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:13,988 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:48:14,831 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:16,832 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:17,643 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:48:18,833 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:19,109 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:48:20,834 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:22,561 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:48:22,834 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:24,230 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:48:24,835 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:26,836 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:27,645 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:48:28,837 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:29,339 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:48:30,838 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:32,646 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:48:32,839 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:34,441 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:48:34,839 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:36,840 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:37,561 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:48:38,841 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:39,551 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:48:40,842 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:41,172 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:48:42,648 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:48:42,843 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:44,682 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:48:44,844 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:46,845 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:47,648 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:48:48,845 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:49,792 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:48:50,846 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:52,561 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:48:52,847 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:54,848 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:54,919 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:48:56,849 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:48:57,650 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:48:58,850 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:00,041 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:49:00,851 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:02,651 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:49:02,852 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:04,853 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:05,155 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:49:06,853 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:07,561 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:49:08,854 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:10,268 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:49:10,855 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:11,171 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:49:12,653 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:49:12,856 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:14,857 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:15,386 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:49:16,858 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:17,653 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:49:18,859 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:20,518 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:49:20,859 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:22,562 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:49:22,860 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:24,861 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:25,637 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:49:26,862 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:27,655 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:49:28,863 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:30,754 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:49:30,864 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:32,656 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:49:32,864 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:34,865 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:35,871 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:49:36,866 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:37,562 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:49:38,867 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:40,868 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:41,038 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:49:41,175 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:49:42,658 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:49:42,869 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:44,870 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:46,147 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:49:46,871 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:47,658 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:49:48,872 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:50,873 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:51,274 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:49:52,562 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:49:52,874 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:54,874 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:56,388 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:49:56,875 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:49:57,660 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:49:58,876 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:00,877 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:01,499 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:50:02,661 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:50:02,878 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:04,879 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:06,614 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:50:06,880 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:07,562 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:50:08,880 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:10,881 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:11,178 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:50:11,727 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:50:12,662 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:50:12,882 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:14,883 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:16,841 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:50:16,884 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:17,663 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:50:18,885 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:20,886 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:21,947 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:50:22,562 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:50:22,887 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:24,888 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:26,889 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:27,058 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:50:27,665 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:50:28,890 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:30,890 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:32,177 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:50:32,666 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:50:32,891 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:34,892 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:36,893 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:37,283 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:50:37,562 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:50:38,894 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:40,895 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:41,184 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:50:42,392 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:50:42,668 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:50:42,896 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:44,896 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:46,897 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:47,507 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:50:47,669 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:50:48,898 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:50,899 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:52,562 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:50:52,614 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:50:52,900 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:54,901 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:56,902 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:50:57,670 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:50:57,730 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:50:58,902 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:00,903 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:02,671 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:51:02,834 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:51:02,904 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:04,905 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:06,906 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:07,562 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:51:07,948 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:51:08,907 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:10,908 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:11,184 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:51:12,673 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:51:12,909 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:13,066 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:51:14,910 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:16,911 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:17,674 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:51:18,182 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:51:18,912 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:20,913 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:22,562 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:51:22,913 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:23,298 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:51:24,915 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:26,915 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:27,675 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:51:28,410 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:51:28,916 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:30,917 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:32,676 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:51:32,918 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:33,530 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:51:34,919 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:36,920 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:37,562 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:51:38,634 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:51:38,920 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:40,922 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:41,187 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:51:42,678 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:51:42,922 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:43,760 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:51:44,923 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:46,924 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:47,679 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:51:48,871 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:51:48,925 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:50,926 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:52,563 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:51:52,927 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:53,993 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:51:54,927 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:56,928 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:57,681 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:51:58,929 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:51:59,103 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:52:00,930 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:02,682 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:52:02,931 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:04,215 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:52:04,932 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:06,933 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:07,562 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:52:08,933 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:09,337 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:52:10,934 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:11,191 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:52:12,683 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:52:12,935 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:14,452 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:52:14,936 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:16,937 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:17,684 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:52:18,938 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:19,564 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:52:20,939 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:22,563 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:52:22,940 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:24,683 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:52:24,941 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:26,942 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:27,686 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:52:28,943 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:29,802 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:52:30,944 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:32,687 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:52:32,945 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:34,916 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:52:34,946 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:36,947 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:37,563 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:52:38,947 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:40,032 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:52:40,948 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:41,196 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:52:42,688 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:52:42,949 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:44,950 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:45,156 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:52:46,951 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:47,690 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:52:48,952 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:50,260 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:52:50,953 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:52,563 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:52:52,954 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:54,954 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:55,383 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:52:56,955 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:52:57,691 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:52:58,956 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:00,502 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:53:00,957 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:02,692 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:53:02,958 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:04,959 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:05,633 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:53:06,960 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:07,563 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:53:08,961 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:10,758 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:53:10,962 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:11,197 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:53:12,694 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:53:12,963 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:14,963 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:15,886 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:53:16,965 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:17,695 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:53:18,965 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:20,966 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:21,008 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:53:22,563 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:53:22,967 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:24,968 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:26,123 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:53:26,969 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:27,696 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:53:28,970 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:30,971 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:31,249 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:53:32,697 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:53:32,972 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:34,972 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:36,368 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:53:36,973 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:37,563 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:53:38,974 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:40,975 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:41,199 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:53:41,486 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:53:42,699 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:53:42,976 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:44,977 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:46,618 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:53:46,978 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:47,700 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:53:48,978 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:50,979 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:51,740 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:53:52,563 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:53:52,980 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:54,981 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:56,864 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:53:56,982 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:53:57,702 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:53:58,983 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:00,984 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:01,985 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:54:02,703 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:54:02,986 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:04,987 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:06,987 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:07,133 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:54:07,564 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:54:08,988 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:10,989 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:11,203 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:54:12,205 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:54:12,704 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:54:12,990 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:14,991 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:16,992 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:17,394 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:54:17,705 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:54:18,993 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:20,994 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:22,512 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:54:22,564 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:54:22,995 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:24,996 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:26,997 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:27,633 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:54:27,707 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:54:28,998 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:30,998 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:32,708 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:54:32,764 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:54:32,999 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:35,000 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:37,001 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:37,564 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:54:37,884 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:54:39,002 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:41,003 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:41,206 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:54:42,710 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:54:43,003 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:43,007 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:54:45,004 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:47,005 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:47,711 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:54:48,133 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:54:49,006 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:51,007 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:52,564 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:54:53,008 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:53,258 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:54:55,008 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:57,009 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:54:57,713 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:54:58,813 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:54:59,961 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:54:59,963 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:54:59,969 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:54:59,970 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:54:59,976 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:54:59,977 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:54:59,981 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:54:59,996 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:54:59,996 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,006 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,006 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,009 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,011 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,046 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,086 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,087 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,088 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,093 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,094 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,097 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,098 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,099 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,100 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,121 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,131 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,132 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,134 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,138 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,139 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,142 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,143 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,144 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,145 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,164 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,174 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,175 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,176 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,181 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,182 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,185 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,186 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,186 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,188 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,208 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,218 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,219 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,220 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,225 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,226 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,229 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,229 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,230 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,231 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,253 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,263 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,264 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,265 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,270 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,271 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,274 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,274 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,275 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,276 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,297 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,307 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,308 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,309 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,314 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,315 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,318 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,318 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,319 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,321 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,342 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,352 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,353 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,355 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,359 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,360 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,363 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,363 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,364 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,366 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,387 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,593 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,594 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:00,595 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: partial_history 2025-09-16 17:55:01,011 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:02,727 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:55:03,011 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:03,849 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:55:05,012 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:07,013 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:07,564 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:55:08,976 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:55:09,014 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:11,015 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:11,209 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:55:12,728 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:55:13,015 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:14,114 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:55:15,016 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:17,017 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:17,730 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:55:19,018 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:19,226 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:55:21,019 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:22,564 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:55:23,020 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:24,363 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:55:25,021 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:27,022 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:27,732 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:55:29,023 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:29,494 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:55:31,024 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:32,733 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:55:33,024 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:34,622 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:55:35,025 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:37,026 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:37,564 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:55:39,027 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:39,743 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:55:41,028 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:41,212 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:55:42,734 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:55:43,029 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:44,853 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:55:45,030 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:47,030 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:47,735 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:55:49,031 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:49,976 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:55:51,033 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:52,564 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:55:53,033 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:55,034 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:55,098 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:55:57,035 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:55:57,737 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:55:59,036 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:00,218 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:56:01,037 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:02,738 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:56:03,038 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:05,039 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:05,348 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:56:07,040 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:07,564 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:56:09,040 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:10,468 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:56:11,041 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:11,216 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:56:12,740 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:56:13,042 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:15,043 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:15,585 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:56:17,044 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:17,741 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:56:19,045 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:20,700 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:56:21,046 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:22,565 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:56:23,047 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:25,047 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:25,845 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:56:27,048 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:27,743 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:56:29,049 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:30,962 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:56:31,050 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:32,744 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:56:33,051 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:35,052 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:36,087 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:56:37,053 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:37,564 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:56:39,054 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:41,055 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:41,203 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:56:41,217 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:56:42,746 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:56:43,055 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:45,056 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:46,333 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:56:47,057 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:47,747 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:56:49,058 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:51,059 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:51,467 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:56:52,565 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:56:53,060 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:55,061 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:56,586 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:56:57,062 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:56:57,748 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:56:59,062 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:01,063 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:01,712 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:57:02,749 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:57:03,064 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:05,065 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:06,832 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:57:07,066 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:07,565 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:57:09,067 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:11,068 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:11,220 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:57:12,222 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:57:12,751 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:57:13,069 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:15,070 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:17,070 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:17,341 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:57:17,752 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:57:19,071 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:21,072 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:22,463 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:57:22,565 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:57:23,074 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:25,074 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:27,075 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:27,599 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:57:27,753 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:57:29,076 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:31,077 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:32,712 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:57:32,755 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:57:33,078 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:35,079 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:37,080 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:37,565 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:57:37,846 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:57:39,080 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:41,081 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:41,223 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:57:42,757 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:57:42,964 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:57:43,082 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:45,083 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:47,084 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:47,758 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:57:48,093 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:57:49,084 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:51,085 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:52,565 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:57:53,087 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:53,216 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:57:56,088 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:57,759 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:57:58,089 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:57:58,334 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:58:00,090 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:02,091 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:02,760 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:58:03,469 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:58:04,092 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:06,092 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:07,565 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:58:08,093 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:08,583 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:58:10,094 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:11,228 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:58:12,095 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:12,762 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:58:13,712 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:58:14,096 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:16,097 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:17,763 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:58:18,098 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:18,849 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:58:20,099 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:22,099 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:22,566 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:58:23,974 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:58:24,100 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:26,101 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:27,765 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:58:28,102 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:29,083 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:58:30,103 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:32,104 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:32,766 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:58:34,105 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:34,190 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:58:36,106 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:37,566 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:58:38,107 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:39,331 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:58:40,108 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:41,228 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:58:42,108 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:42,767 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:58:44,110 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:44,448 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:58:46,110 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:47,768 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:58:48,111 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:49,568 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:58:50,112 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:52,113 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:52,566 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:58:54,114 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:54,676 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:58:56,115 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:57,770 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:58:58,115 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:58:59,789 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:59:00,116 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:02,117 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:02,771 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:59:04,118 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:04,921 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:59:06,119 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:07,566 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:59:08,120 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:10,028 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:59:10,121 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:11,231 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:59:12,122 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:12,772 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:59:14,122 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:15,149 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:59:16,123 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:17,773 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:59:18,124 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:20,125 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:20,249 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:59:22,126 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:22,566 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:59:24,127 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:25,360 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:59:26,128 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:27,775 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:59:28,129 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:30,129 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:30,464 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:59:32,130 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:32,776 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:59:34,131 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:35,580 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:59:36,132 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:37,566 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:59:38,133 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:40,134 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:40,725 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:59:41,234 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 17:59:42,134 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:42,777 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:59:44,135 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:45,836 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:59:46,136 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:47,778 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:59:48,137 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:50,138 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:50,958 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:59:52,139 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:52,566 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 17:59:54,140 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:56,079 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 17:59:56,140 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 17:59:57,780 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 17:59:58,141 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:00,142 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:01,202 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:00:02,143 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:02,781 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:00:04,144 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:06,144 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:06,401 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:00:07,567 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 18:00:08,145 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:10,146 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:11,236 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 18:00:11,517 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:00:12,147 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:12,782 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:00:14,148 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:16,149 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:16,639 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:00:17,783 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:00:18,150 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:20,150 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:21,765 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:00:22,151 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:22,567 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 18:00:24,152 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:26,153 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:26,881 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:00:27,785 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:00:28,154 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:30,155 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:32,002 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:00:32,155 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:32,786 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:00:34,156 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:36,157 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:37,116 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:00:37,567 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 18:00:38,158 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:40,159 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:41,239 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 18:00:42,160 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:42,225 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:00:42,787 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:00:44,161 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:46,162 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:47,336 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:00:47,788 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:00:48,163 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:50,163 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:52,164 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:52,455 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:00:52,567 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 18:00:54,165 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:56,166 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:00:57,573 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:00:57,789 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:00:58,167 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:00,168 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:02,169 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:02,695 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:01:02,791 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:01:04,169 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:06,170 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:07,567 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 18:01:07,808 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:01:08,172 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:10,172 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:11,242 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 18:01:12,173 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:12,792 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:01:12,929 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:01:14,174 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:16,175 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:17,793 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:01:18,041 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:01:18,176 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:20,177 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:22,178 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:22,567 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 18:01:23,146 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:01:24,178 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:26,180 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:27,795 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:01:28,180 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:28,249 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:01:30,181 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:32,182 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:32,795 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:01:33,358 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:01:34,183 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:36,184 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:37,567 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 18:01:38,185 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:38,480 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:01:40,185 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:41,245 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 18:01:42,186 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:42,797 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:01:43,587 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:01:44,187 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:46,188 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:47,799 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:01:48,189 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:48,699 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:01:50,190 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:52,191 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:52,567 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 18:01:53,805 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:01:54,192 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:56,193 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:57,800 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:01:58,194 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:01:58,914 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:02:00,195 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:02,196 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:02,801 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:02:04,023 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:02:04,196 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:06,197 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:07,567 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 18:02:08,198 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:09,137 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:02:10,199 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:11,248 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 18:02:12,200 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:12,803 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:02:14,201 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:14,262 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:02:16,202 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:17,804 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:02:18,203 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:19,366 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:02:20,204 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:22,204 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:22,568 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 18:02:24,205 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:24,478 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:02:26,206 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:27,805 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:02:28,207 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:29,585 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:02:30,208 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:32,208 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:32,806 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:02:34,209 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:34,704 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:02:36,210 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:37,568 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 18:02:38,211 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:39,807 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:02:40,212 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:41,250 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 18:02:42,213 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:42,809 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:02:44,215 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:44,932 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:02:46,215 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:47,810 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:02:48,216 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:50,047 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:02:50,217 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:52,218 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:52,568 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 18:02:54,219 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:55,149 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:02:56,220 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:02:57,811 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:02:58,220 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:00,221 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:00,256 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:03:02,222 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:02,813 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:03:04,223 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:05,370 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:03:06,224 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:07,568 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 18:03:08,225 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:10,226 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:10,491 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:03:11,253 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 18:03:12,227 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:12,815 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:03:14,228 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:15,602 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:03:16,229 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:17,816 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:03:18,229 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:20,230 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:20,715 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:03:22,231 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:22,568 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 18:03:24,232 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:25,825 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:03:26,233 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:27,818 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:03:28,233 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:30,234 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:30,946 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:03:32,235 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:32,818 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:03:34,236 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:36,054 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:03:36,237 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:37,568 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 18:03:38,238 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:40,239 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:41,166 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:03:41,257 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 18:03:42,240 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:42,820 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:03:44,241 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:46,242 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:46,290 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:03:47,821 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:03:48,243 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:50,243 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:51,428 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:03:52,244 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:52,568 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 18:03:54,245 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:56,246 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:03:56,541 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:03:57,823 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:03:58,247 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:04:00,248 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:04:01,662 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:04:02,248 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:04:02,824 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:04:04,249 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:04:06,250 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:04:06,794 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:04:07,568 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: internal_messages 2025-09-16 18:04:08,251 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:04:10,252 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:04:11,261 DEBUG SenderThread:539590 [sender.py:send():382] send: stats 2025-09-16 18:04:11,912 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:04:12,253 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:04:12,826 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:04:14,254 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:04:16,254 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:04:17,025 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: status_report 2025-09-16 18:04:17,827 DEBUG HandlerThread:539590 [handler.py:handle_request():146] handle_request: keepalive 2025-09-16 18:04:18,255 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log 2025-09-16 18:04:20,256 INFO Thread-12 :539590 [dir_watcher.py:_on_file_modified():288] file/dir modified: /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed102/wandb/run-20250916_173140-78ke0sc9/files/output.log