2025-09-17 21:58:17,566 INFO    MainThread:1208058 [wandb_setup.py:_flush():76] Current SDK version is 0.16.1
2025-09-17 21:58:17,567 INFO    MainThread:1208058 [wandb_setup.py:_flush():76] Configure stats pid to 1208058
2025-09-17 21:58:17,567 INFO    MainThread:1208058 [wandb_setup.py:_flush():76] Loading settings from /global/homes/a/amahesh/.config/wandb/settings
2025-09-17 21:58:17,567 INFO    MainThread:1208058 [wandb_setup.py:_flush():76] Loading settings from /global/u2/a/amahesh/ms_finetune/modulus-makani-fork/wandb/settings
2025-09-17 21:58:17,567 INFO    MainThread:1208058 [wandb_setup.py:_flush():76] Loading settings from environment variables: {'start_method': 'thread'}
2025-09-17 21:58:17,567 WARNING MainThread:1208058 [wandb_setup.py:_flush():76] Could not find program at -m makani.train
2025-09-17 21:58:17,567 INFO    MainThread:1208058 [wandb_setup.py:_flush():76] Inferring run settings from compute environment: {'program_relpath': None, 'program': '-m makani.train'}
2025-09-17 21:58:17,567 INFO    MainThread:1208058 [wandb_init.py:_log_setup():524] Logging user logs to /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/logs/debug.log
2025-09-17 21:58:17,568 INFO    MainThread:1208058 [wandb_init.py:_log_setup():525] Logging internal logs to /pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/wandb/run-20250917_215817-jdwsvk36/logs/debug-internal.log
2025-09-17 21:58:17,568 INFO    MainThread:1208058 [wandb_init.py:init():564] calling init triggers
2025-09-17 21:58:17,569 INFO    MainThread:1208058 [wandb_init.py:init():571] wandb.init called with sweep_config: {}
config: {'_original_attrs': ['_original_attrs', 'params'], 'params': {'wandb_group': 'multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2-0.1.0', 'scheduler': 'CosineAnnealingLR', 'max_epochs': 20, 'scheduler_T_max': 20, 'lr': 0.0001, 'load_counters': False, 'load_optimizer': False, 'load_scheduler': False, 'finetune': True, 'pretrained_checkpoint_path': '/pscratch/sd/a/amahesh/recovered_fcn_training/modulus-makani_runs-0.1.0-fcndev_stats/sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/training_checkpoints/best_ckpt_mp0.tar', 'embed_dim': 620, 'num_layers': 8, 'scale_factor': 2, 'hard_thresholding_fraction': 1.0, 'loss': 'weighted squared temp-std geometric l2', 'valid_autoreg_steps': 1, 'metadata_json_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/data.json', 'train_data_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/train', 'valid_data_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/valid', 'exp_dir': '/pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/', 'n_years': 1, 'img_shape_x': 721, 'img_shape_y': 1440, 'min_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/stats/mins.npy', 'max_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/stats/maxs.npy', 'time_means_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/stats_fcndev/time_means.npy', 'global_means_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/stats_fcndev/global_means.npy', 'global_stds_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/stats_fcndev/global_stds.npy', 'time_diff_means_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/stats_fcndev/time_diff_means.npy', 'time_diff_stds_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/stats_fcndev/time_diff_stds.npy', 'nettype': 'SFNO', 'model_grid_type': 'equiangular', 'sht_grid_type': 'legendre-gauss', 'filter_type': 'linear', 'complex_activation': 'real', 'normalization_layer': 'instance_norm', 'use_mlp': True, 'mlp_mode': 'serial', 'mlp_ratio': 2, 'separable': False, 'operator_type': 'dhconv', 'activation_function': 'gelu', 'pos_embed': 'none', 'channel_weights': 'auto', 'n_eval_samples': 8760, 'batch_size': 1, 'weight_decay': 0.0, 'scheduler_factor': 0.1, 'scheduler_patience': 10, 'scheduler_step_size': 100, 'scheduler_gamma': 0.5, 'lr_warmup_steps': 0, 'verbose': False, 'wireup_info': 'mpi', 'wireup_store': 'tcp', 'num_data_workers': 2, 'num_visualization_workers': 2, 'dt': 1, 'n_history': 0, 'prediction_type': 'iterative', 'prediction_length': 35, 'n_initial_conditions': 5, 'n_train_samples_per_epoch': 54000, 'ics_type': 'specify_number', 'save_raw_forecasts': True, 'save_channel': False, 'masked_acc': False, 'maskpath': None, 'perturb': False, 'add_noise': False, 'noise_std': 0.0, 'target': 'default', 'normalize_residual': False, 'channel_names': ['u10m', 'v10m', 'u100m', 'v100m', 't2m', 'sp', 'msl', 'tcwv', 'd2m', 'u50', 'u100', 'u150', 'u200', 'u250', 'u300', 'u400', 'u500', 'u600', 'u700', 'u850', 'u925', 'u1000', 'v50', 'v100', 'v150', 'v200', 'v250', 'v300', 'v400', 'v500', 'v600', 'v700', 'v850', 'v925', 'v1000', 'z50', 'z100', 'z150', 'z200', 'z250', 'z300', 'z400', 'z500', 'z600', 'z700', 'z850', 'z925', 'z1000', 't50', 't100', 't150', 't200', 't250', 't300', 't400', 't500', 't600', 't700', 't850', 't925', 't1000', 'q50', 'q100', 'q150', 'q200', 'q250', 'q300', 'q400', 'q500', 'q600', 'q700', 'q850', 'q925', 'q1000'], 'normalization': 'zscore', 'add_grid': True, 'gridtype': 'sinusoidal', 'grid_num_frequencies': 16, 'roll': False, 'add_zenith': True, 'add_orography': True, 'orography_path': '/global/cfs/cdirs/m3522/cmip6/ERA5/e5.oper.invariant/197901/e5.oper.invariant.128_129_z.ll025sc.1979010100_1979010100.nc', 'add_landmask': True, 'landmask_path': '/global/cfs/cdirs/m3522/cmip6/ERA5/e5.oper.invariant/197901/e5.oper.invariant.128_172_lsm.ll025sc.1979010100_1979010100.nc', 'log_to_screen': True, 'log_to_wandb': True, 'log_video': 20, 'save_checkpoint': 'legacy', 'optimizer_type': 'AdamW', 'optimizer_beta1': 0.9, 'optimizer_beta2': 0.95, 'optimizer_max_grad_norm': 32, 'crop_size_x': None, 'crop_size_y': None, 'inf_data_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/out_of_sample', 'wandb_name': None, 'wandb_project': 'ERA5_sfno', 'wandb_entity': 'weatherbenching', 'pos_drop_rate': 0.1, 'initialization_seed': 77, 'epsilon_factor': 0, 'fin_parallel_size': 1, 'fout_parallel_size': 1, 'h_parallel_size': 4, 'w_parallel_size': 1, 'model_parallel_sizes': [4, 1, 1, 1], 'model_parallel_names': ['h', 'w', 'fin', 'fout'], 'parameters_reduction_buffer_count': 1, 'load_checkpoint': 'legacy', 'world_size': 128, 'global_batch_size': 32, 'experiment_dir': '/pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77', 'checkpoint_path': '/pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/training_checkpoints/ckpt_mp{mp_rank}.tar', 'best_checkpoint_path': '/pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/training_checkpoints/best_ckpt_mp{mp_rank}.tar', 'resuming': False, 'amp_mode': 'bf16', 'jit_mode': 'none', 'cuda_graph_mode': 'none', 'skip_validation': False, 'enable_odirect': False, 'checkpointing': 0, 'enable_synthetic_data': False, 'split_data_channels': False, 'print_timings_frequency': -1, 'multistep_count': 2, 'n_future': 1, 'enable_benchy': False, 'disable_ddp': False, 'enable_grad_anomaly_detection': False, 'wandb_dir': '/pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77', 'h5_path': 'fields', 'dhours': 6, 'lat': [90.0, 89.75, 89.5, 89.25, 89.0, 88.75, 88.5, 88.25, 88.0, 87.75, 87.5, 87.25, 87.0, 86.75, 86.5, 86.25, 86.0, 85.75, 85.5, 85.25, 85.0, 84.75, 84.5, 84.25, 84.0, 83.75, 83.5, 83.25, 83.0, 82.75, 82.5, 82.25, 82.0, 81.75, 81.5, 81.25, 81.0, 80.75, 80.5, 80.25, 80.0, 79.75, 79.5, 79.25, 79.0, 78.75, 78.5, 78.25, 78.0, 77.75, 77.5, 77.25, 77.0, 76.75, 76.5, 76.25, 76.0, 75.75, 75.5, 75.25, 75.0, 74.75, 74.5, 74.25, 74.0, 73.75, 73.5, 73.25, 73.0, 72.75, 72.5, 72.25, 72.0, 71.75, 71.5, 71.25, 71.0, 70.75, 70.5, 70.25, 70.0, 69.75, 69.5, 69.25, 69.0, 68.75, 68.5, 68.25, 68.0, 67.75, 67.5, 67.25, 67.0, 66.75, 66.5, 66.25, 66.0, 65.75, 65.5, 65.25, 65.0, 64.75, 64.5, 64.25, 64.0, 63.75, 63.5, 63.25, 63.0, 62.75, 62.5, 62.25, 62.0, 61.75, 61.5, 61.25, 61.0, 60.75, 60.5, 60.25, 60.0, 59.75, 59.5, 59.25, 59.0, 58.75, 58.5, 58.25, 58.0, 57.75, 57.5, 57.25, 57.0, 56.75, 56.5, 56.25, 56.0, 55.75, 55.5, 55.25, 55.0, 54.75, 54.5, 54.25, 54.0, 53.75, 53.5, 53.25, 53.0, 52.75, 52.5, 52.25, 52.0, 51.75, 51.5, 51.25, 51.0, 50.75, 50.5, 50.25, 50.0, 49.75, 49.5, 49.25, 49.0, 48.75, 48.5, 48.25, 48.0, 47.75, 47.5, 47.25, 47.0, 46.75, 46.5, 46.25, 46.0, 45.75, 45.5, 45.25, 45.0, 44.75, 44.5, 44.25, 44.0, 43.75, 43.5, 43.25, 43.0, 42.75, 42.5, 42.25, 42.0, 41.75, 41.5, 41.25, 41.0, 40.75, 40.5, 40.25, 40.0, 39.75, 39.5, 39.25, 39.0, 38.75, 38.5, 38.25, 38.0, 37.75, 37.5, 37.25, 37.0, 36.75, 36.5, 36.25, 36.0, 35.75, 35.5, 35.25, 35.0, 34.75, 34.5, 34.25, 34.0, 33.75, 33.5, 33.25, 33.0, 32.75, 32.5, 32.25, 32.0, 31.75, 31.5, 31.25, 31.0, 30.75, 30.5, 30.25, 30.0, 29.75, 29.5, 29.25, 29.0, 28.75, 28.5, 28.25, 28.0, 27.75, 27.5, 27.25, 27.0, 26.75, 26.5, 26.25, 26.0, 25.75, 25.5, 25.25, 25.0, 24.75, 24.5, 24.25, 24.0, 23.75, 23.5, 23.25, 23.0, 22.75, 22.5, 22.25, 22.0, 21.75, 21.5, 21.25, 21.0, 20.75, 20.5, 20.25, 20.0, 19.75, 19.5, 19.25, 19.0, 18.75, 18.5, 18.25, 18.0, 17.75, 17.5, 17.25, 17.0, 16.75, 16.5, 16.25, 16.0, 15.75, 15.5, 15.25, 15.0, 14.75, 14.5, 14.25, 14.0, 13.75, 13.5, 13.25, 13.0, 12.75, 12.5, 12.25, 12.0, 11.75, 11.5, 11.25, 11.0, 10.75, 10.5, 10.25, 10.0, 9.75, 9.5, 9.25, 9.0, 8.75, 8.5, 8.25, 8.0, 7.75, 7.5, 7.25, 7.0, 6.75, 6.5, 6.25, 6.0, 5.75, 5.5, 5.25, 5.0, 4.75, 4.5, 4.25, 4.0, 3.75, 3.5, 3.25, 3.0, 2.75, 2.5, 2.25, 2.0, 1.75, 1.5, 1.25, 1.0, 0.75, 0.5, 0.25, 0.0, -0.25, -0.5, -0.75, -1.0, -1.25, -1.5, -1.75, -2.0, -2.25, -2.5, -2.75, -3.0, -3.25, -3.5, -3.75, -4.0, -4.25, -4.5, -4.75, -5.0, -5.25, -5.5, -5.75, -6.0, -6.25, -6.5, -6.75, -7.0, -7.25, -7.5, -7.75, -8.0, -8.25, -8.5, -8.75, -9.0, -9.25, -9.5, -9.75, -10.0, -10.25, -10.5, -10.75, -11.0, -11.25, -11.5, -11.75, -12.0, -12.25, -12.5, -12.75, -13.0, -13.25, -13.5, -13.75, -14.0, -14.25, -14.5, -14.75, -15.0, -15.25, -15.5, -15.75, -16.0, -16.25, -16.5, -16.75, -17.0, -17.25, -17.5, -17.75, -18.0, -18.25, -18.5, -18.75, -19.0, -19.25, -19.5, -19.75, -20.0, -20.25, -20.5, -20.75, -21.0, -21.25, -21.5, -21.75, -22.0, -22.25, -22.5, -22.75, -23.0, -23.25, -23.5, -23.75, -24.0, -24.25, -24.5, -24.75, -25.0, -25.25, -25.5, -25.75, -26.0, -26.25, -26.5, -26.75, -27.0, -27.25, -27.5, -27.75, -28.0, -28.25, -28.5, -28.75, -29.0, -29.25, -29.5, -29.75, -30.0, -30.25, -30.5, -30.75, -31.0, -31.25, -31.5, -31.75, -32.0, -32.25, -32.5, -32.75, -33.0, -33.25, -33.5, -33.75, -34.0, -34.25, -34.5, -34.75, -35.0, -35.25, -35.5, -35.75, -36.0, -36.25, -36.5, -36.75, -37.0, -37.25, -37.5, -37.75, -38.0, -38.25, -38.5, -38.75, -39.0, -39.25, -39.5, -39.75, -40.0, -40.25, -40.5, -40.75, -41.0, -41.25, -41.5, -41.75, -42.0, -42.25, -42.5, -42.75, -43.0, -43.25, -43.5, -43.75, -44.0, -44.25, -44.5, -44.75, -45.0, -45.25, -45.5, -45.75, -46.0, -46.25, -46.5, -46.75, -47.0, -47.25, -47.5, -47.75, -48.0, -48.25, -48.5, -48.75, -49.0, -49.25, -49.5, -49.75, -50.0, -50.25, -50.5, -50.75, -51.0, -51.25, -51.5, -51.75, -52.0, -52.25, -52.5, -52.75, -53.0, -53.25, -53.5, -53.75, -54.0, -54.25, -54.5, -54.75, -55.0, -55.25, -55.5, -55.75, -56.0, -56.25, -56.5, -56.75, -57.0, -57.25, -57.5, -57.75, -58.0, -58.25, -58.5, -58.75, -59.0, -59.25, -59.5, -59.75, -60.0, -60.25, -60.5, -60.75, -61.0, -61.25, -61.5, -61.75, -62.0, -62.25, -62.5, -62.75, -63.0, -63.25, -63.5, -63.75, -64.0, -64.25, -64.5, -64.75, -65.0, -65.25, -65.5, -65.75, -66.0, -66.25, -66.5, -66.75, -67.0, -67.25, -67.5, -67.75, -68.0, -68.25, -68.5, -68.75, -69.0, -69.25, -69.5, -69.75, -70.0, -70.25, -70.5, -70.75, -71.0, -71.25, -71.5, -71.75, -72.0, -72.25, -72.5, -72.75, -73.0, -73.25, -73.5, -73.75, -74.0, -74.25, -74.5, -74.75, -75.0, -75.25, -75.5, -75.75, -76.0, -76.25, -76.5, -76.75, -77.0, -77.25, -77.5, -77.75, -78.0, -78.25, -78.5, -78.75, -79.0, -79.25, -79.5, -79.75, -80.0, -80.25, -80.5, -80.75, -81.0, -81.25, -81.5, -81.75, -82.0, -82.25, -82.5, -82.75, -83.0, -83.25, -83.5, -83.75, -84.0, -84.25, -84.5, -84.75, -85.0, -85.25, -85.5, -85.75, -86.0, -86.25, -86.5, -86.75, -87.0, -87.25, -87.5, -87.75, -88.0, -88.25, -88.5, -88.75, -89.0, -89.25, -89.5, -89.75, -90.0], 'lon': [0.0, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2.0, 2.25, 2.5, 2.75, 3.0, 3.25, 3.5, 3.75, 4.0, 4.25, 4.5, 4.75, 5.0, 5.25, 5.5, 5.75, 6.0, 6.25, 6.5, 6.75, 7.0, 7.25, 7.5, 7.75, 8.0, 8.25, 8.5, 8.75, 9.0, 9.25, 9.5, 9.75, 10.0, 10.25, 10.5, 10.75, 11.0, 11.25, 11.5, 11.75, 12.0, 12.25, 12.5, 12.75, 13.0, 13.25, 13.5, 13.75, 14.0, 14.25, 14.5, 14.75, 15.0, 15.25, 15.5, 15.75, 16.0, 16.25, 16.5, 16.75, 17.0, 17.25, 17.5, 17.75, 18.0, 18.25, 18.5, 18.75, 19.0, 19.25, 19.5, 19.75, 20.0, 20.25, 20.5, 20.75, 21.0, 21.25, 21.5, 21.75, 22.0, 22.25, 22.5, 22.75, 23.0, 23.25, 23.5, 23.75, 24.0, 24.25, 24.5, 24.75, 25.0, 25.25, 25.5, 25.75, 26.0, 26.25, 26.5, 26.75, 27.0, 27.25, 27.5, 27.75, 28.0, 28.25, 28.5, 28.75, 29.0, 29.25, 29.5, 29.75, 30.0, 30.25, 30.5, 30.75, 31.0, 31.25, 31.5, 31.75, 32.0, 32.25, 32.5, 32.75, 33.0, 33.25, 33.5, 33.75, 34.0, 34.25, 34.5, 34.75, 35.0, 35.25, 35.5, 35.75, 36.0, 36.25, 36.5, 36.75, 37.0, 37.25, 37.5, 37.75, 38.0, 38.25, 38.5, 38.75, 39.0, 39.25, 39.5, 39.75, 40.0, 40.25, 40.5, 40.75, 41.0, 41.25, 41.5, 41.75, 42.0, 42.25, 42.5, 42.75, 43.0, 43.25, 43.5, 43.75, 44.0, 44.25, 44.5, 44.75, 45.0, 45.25, 45.5, 45.75, 46.0, 46.25, 46.5, 46.75, 47.0, 47.25, 47.5, 47.75, 48.0, 48.25, 48.5, 48.75, 49.0, 49.25, 49.5, 49.75, 50.0, 50.25, 50.5, 50.75, 51.0, 51.25, 51.5, 51.75, 52.0, 52.25, 52.5, 52.75, 53.0, 53.25, 53.5, 53.75, 54.0, 54.25, 54.5, 54.75, 55.0, 55.25, 55.5, 55.75, 56.0, 56.25, 56.5, 56.75, 57.0, 57.25, 57.5, 57.75, 58.0, 58.25, 58.5, 58.75, 59.0, 59.25, 59.5, 59.75, 60.0, 60.25, 60.5, 60.75, 61.0, 61.25, 61.5, 61.75, 62.0, 62.25, 62.5, 62.75, 63.0, 63.25, 63.5, 63.75, 64.0, 64.25, 64.5, 64.75, 65.0, 65.25, 65.5, 65.75, 66.0, 66.25, 66.5, 66.75, 67.0, 67.25, 67.5, 67.75, 68.0, 68.25, 68.5, 68.75, 69.0, 69.25, 69.5, 69.75, 70.0, 70.25, 70.5, 70.75, 71.0, 71.25, 71.5, 71.75, 72.0, 72.25, 72.5, 72.75, 73.0, 73.25, 73.5, 73.75, 74.0, 74.25, 74.5, 74.75, 75.0, 75.25, 75.5, 75.75, 76.0, 76.25, 76.5, 76.75, 77.0, 77.25, 77.5, 77.75, 78.0, 78.25, 78.5, 78.75, 79.0, 79.25, 79.5, 79.75, 80.0, 80.25, 80.5, 80.75, 81.0, 81.25, 81.5, 81.75, 82.0, 82.25, 82.5, 82.75, 83.0, 83.25, 83.5, 83.75, 84.0, 84.25, 84.5, 84.75, 85.0, 85.25, 85.5, 85.75, 86.0, 86.25, 86.5, 86.75, 87.0, 87.25, 87.5, 87.75, 88.0, 88.25, 88.5, 88.75, 89.0, 89.25, 89.5, 89.75, 90.0, 90.25, 90.5, 90.75, 91.0, 91.25, 91.5, 91.75, 92.0, 92.25, 92.5, 92.75, 93.0, 93.25, 93.5, 93.75, 94.0, 94.25, 94.5, 94.75, 95.0, 95.25, 95.5, 95.75, 96.0, 96.25, 96.5, 96.75, 97.0, 97.25, 97.5, 97.75, 98.0, 98.25, 98.5, 98.75, 99.0, 99.25, 99.5, 99.75, 100.0, 100.25, 100.5, 100.75, 101.0, 101.25, 101.5, 101.75, 102.0, 102.25, 102.5, 102.75, 103.0, 103.25, 103.5, 103.75, 104.0, 104.25, 104.5, 104.75, 105.0, 105.25, 105.5, 105.75, 106.0, 106.25, 106.5, 106.75, 107.0, 107.25, 107.5, 107.75, 108.0, 108.25, 108.5, 108.75, 109.0, 109.25, 109.5, 109.75, 110.0, 110.25, 110.5, 110.75, 111.0, 111.25, 111.5, 111.75, 112.0, 112.25, 112.5, 112.75, 113.0, 113.25, 113.5, 113.75, 114.0, 114.25, 114.5, 114.75, 115.0, 115.25, 115.5, 115.75, 116.0, 116.25, 116.5, 116.75, 117.0, 117.25, 117.5, 117.75, 118.0, 118.25, 118.5, 118.75, 119.0, 119.25, 119.5, 119.75, 120.0, 120.25, 120.5, 120.75, 121.0, 121.25, 121.5, 121.75, 122.0, 122.25, 122.5, 122.75, 123.0, 123.25, 123.5, 123.75, 124.0, 124.25, 124.5, 124.75, 125.0, 125.25, 125.5, 125.75, 126.0, 126.25, 126.5, 126.75, 127.0, 127.25, 127.5, 127.75, 128.0, 128.25, 128.5, 128.75, 129.0, 129.25, 129.5, 129.75, 130.0, 130.25, 130.5, 130.75, 131.0, 131.25, 131.5, 131.75, 132.0, 132.25, 132.5, 132.75, 133.0, 133.25, 133.5, 133.75, 134.0, 134.25, 134.5, 134.75, 135.0, 135.25, 135.5, 135.75, 136.0, 136.25, 136.5, 136.75, 137.0, 137.25, 137.5, 137.75, 138.0, 138.25, 138.5, 138.75, 139.0, 139.25, 139.5, 139.75, 140.0, 140.25, 140.5, 140.75, 141.0, 141.25, 141.5, 141.75, 142.0, 142.25, 142.5, 142.75, 143.0, 143.25, 143.5, 143.75, 144.0, 144.25, 144.5, 144.75, 145.0, 145.25, 145.5, 145.75, 146.0, 146.25, 146.5, 146.75, 147.0, 147.25, 147.5, 147.75, 148.0, 148.25, 148.5, 148.75, 149.0, 149.25, 149.5, 149.75, 150.0, 150.25, 150.5, 150.75, 151.0, 151.25, 151.5, 151.75, 152.0, 152.25, 152.5, 152.75, 153.0, 153.25, 153.5, 153.75, 154.0, 154.25, 154.5, 154.75, 155.0, 155.25, 155.5, 155.75, 156.0, 156.25, 156.5, 156.75, 157.0, 157.25, 157.5, 157.75, 158.0, 158.25, 158.5, 158.75, 159.0, 159.25, 159.5, 159.75, 160.0, 160.25, 160.5, 160.75, 161.0, 161.25, 161.5, 161.75, 162.0, 162.25, 162.5, 162.75, 163.0, 163.25, 163.5, 163.75, 164.0, 164.25, 164.5, 164.75, 165.0, 165.25, 165.5, 165.75, 166.0, 166.25, 166.5, 166.75, 167.0, 167.25, 167.5, 167.75, 168.0, 168.25, 168.5, 168.75, 169.0, 169.25, 169.5, 169.75, 170.0, 170.25, 170.5, 170.75, 171.0, 171.25, 171.5, 171.75, 172.0, 172.25, 172.5, 172.75, 173.0, 173.25, 173.5, 173.75, 174.0, 174.25, 174.5, 174.75, 175.0, 175.25, 175.5, 175.75, 176.0, 176.25, 176.5, 176.75, 177.0, 177.25, 177.5, 177.75, 178.0, 178.25, 178.5, 178.75, 179.0, 179.25, 179.5, 179.75, 180.0, 180.25, 180.5, 180.75, 181.0, 181.25, 181.5, 181.75, 182.0, 182.25, 182.5, 182.75, 183.0, 183.25, 183.5, 183.75, 184.0, 184.25, 184.5, 184.75, 185.0, 185.25, 185.5, 185.75, 186.0, 186.25, 186.5, 186.75, 187.0, 187.25, 187.5, 187.75, 188.0, 188.25, 188.5, 188.75, 189.0, 189.25, 189.5, 189.75, 190.0, 190.25, 190.5, 190.75, 191.0, 191.25, 191.5, 191.75, 192.0, 192.25, 192.5, 192.75, 193.0, 193.25, 193.5, 193.75, 194.0, 194.25, 194.5, 194.75, 195.0, 195.25, 195.5, 195.75, 196.0, 196.25, 196.5, 196.75, 197.0, 197.25, 197.5, 197.75, 198.0, 198.25, 198.5, 198.75, 199.0, 199.25, 199.5, 199.75, 200.0, 200.25, 200.5, 200.75, 201.0, 201.25, 201.5, 201.75, 202.0, 202.25, 202.5, 202.75, 203.0, 203.25, 203.5, 203.75, 204.0, 204.25, 204.5, 204.75, 205.0, 205.25, 205.5, 205.75, 206.0, 206.25, 206.5, 206.75, 207.0, 207.25, 207.5, 207.75, 208.0, 208.25, 208.5, 208.75, 209.0, 209.25, 209.5, 209.75, 210.0, 210.25, 210.5, 210.75, 211.0, 211.25, 211.5, 211.75, 212.0, 212.25, 212.5, 212.75, 213.0, 213.25, 213.5, 213.75, 214.0, 214.25, 214.5, 214.75, 215.0, 215.25, 215.5, 215.75, 216.0, 216.25, 216.5, 216.75, 217.0, 217.25, 217.5, 217.75, 218.0, 218.25, 218.5, 218.75, 219.0, 219.25, 219.5, 219.75, 220.0, 220.25, 220.5, 220.75, 221.0, 221.25, 221.5, 221.75, 222.0, 222.25, 222.5, 222.75, 223.0, 223.25, 223.5, 223.75, 224.0, 224.25, 224.5, 224.75, 225.0, 225.25, 225.5, 225.75, 226.0, 226.25, 226.5, 226.75, 227.0, 227.25, 227.5, 227.75, 228.0, 228.25, 228.5, 228.75, 229.0, 229.25, 229.5, 229.75, 230.0, 230.25, 230.5, 230.75, 231.0, 231.25, 231.5, 231.75, 232.0, 232.25, 232.5, 232.75, 233.0, 233.25, 233.5, 233.75, 234.0, 234.25, 234.5, 234.75, 235.0, 235.25, 235.5, 235.75, 236.0, 236.25, 236.5, 236.75, 237.0, 237.25, 237.5, 237.75, 238.0, 238.25, 238.5, 238.75, 239.0, 239.25, 239.5, 239.75, 240.0, 240.25, 240.5, 240.75, 241.0, 241.25, 241.5, 241.75, 242.0, 242.25, 242.5, 242.75, 243.0, 243.25, 243.5, 243.75, 244.0, 244.25, 244.5, 244.75, 245.0, 245.25, 245.5, 245.75, 246.0, 246.25, 246.5, 246.75, 247.0, 247.25, 247.5, 247.75, 248.0, 248.25, 248.5, 248.75, 249.0, 249.25, 249.5, 249.75, 250.0, 250.25, 250.5, 250.75, 251.0, 251.25, 251.5, 251.75, 252.0, 252.25, 252.5, 252.75, 253.0, 253.25, 253.5, 253.75, 254.0, 254.25, 254.5, 254.75, 255.0, 255.25, 255.5, 255.75, 256.0, 256.25, 256.5, 256.75, 257.0, 257.25, 257.5, 257.75, 258.0, 258.25, 258.5, 258.75, 259.0, 259.25, 259.5, 259.75, 260.0, 260.25, 260.5, 260.75, 261.0, 261.25, 261.5, 261.75, 262.0, 262.25, 262.5, 262.75, 263.0, 263.25, 263.5, 263.75, 264.0, 264.25, 264.5, 264.75, 265.0, 265.25, 265.5, 265.75, 266.0, 266.25, 266.5, 266.75, 267.0, 267.25, 267.5, 267.75, 268.0, 268.25, 268.5, 268.75, 269.0, 269.25, 269.5, 269.75, 270.0, 270.25, 270.5, 270.75, 271.0, 271.25, 271.5, 271.75, 272.0, 272.25, 272.5, 272.75, 273.0, 273.25, 273.5, 273.75, 274.0, 274.25, 274.5, 274.75, 275.0, 275.25, 275.5, 275.75, 276.0, 276.25, 276.5, 276.75, 277.0, 277.25, 277.5, 277.75, 278.0, 278.25, 278.5, 278.75, 279.0, 279.25, 279.5, 279.75, 280.0, 280.25, 280.5, 280.75, 281.0, 281.25, 281.5, 281.75, 282.0, 282.25, 282.5, 282.75, 283.0, 283.25, 283.5, 283.75, 284.0, 284.25, 284.5, 284.75, 285.0, 285.25, 285.5, 285.75, 286.0, 286.25, 286.5, 286.75, 287.0, 287.25, 287.5, 287.75, 288.0, 288.25, 288.5, 288.75, 289.0, 289.25, 289.5, 289.75, 290.0, 290.25, 290.5, 290.75, 291.0, 291.25, 291.5, 291.75, 292.0, 292.25, 292.5, 292.75, 293.0, 293.25, 293.5, 293.75, 294.0, 294.25, 294.5, 294.75, 295.0, 295.25, 295.5, 295.75, 296.0, 296.25, 296.5, 296.75, 297.0, 297.25, 297.5, 297.75, 298.0, 298.25, 298.5, 298.75, 299.0, 299.25, 299.5, 299.75, 300.0, 300.25, 300.5, 300.75, 301.0, 301.25, 301.5, 301.75, 302.0, 302.25, 302.5, 302.75, 303.0, 303.25, 303.5, 303.75, 304.0, 304.25, 304.5, 304.75, 305.0, 305.25, 305.5, 305.75, 306.0, 306.25, 306.5, 306.75, 307.0, 307.25, 307.5, 307.75, 308.0, 308.25, 308.5, 308.75, 309.0, 309.25, 309.5, 309.75, 310.0, 310.25, 310.5, 310.75, 311.0, 311.25, 311.5, 311.75, 312.0, 312.25, 312.5, 312.75, 313.0, 313.25, 313.5, 313.75, 314.0, 314.25, 314.5, 314.75, 315.0, 315.25, 315.5, 315.75, 316.0, 316.25, 316.5, 316.75, 317.0, 317.25, 317.5, 317.75, 318.0, 318.25, 318.5, 318.75, 319.0, 319.25, 319.5, 319.75, 320.0, 320.25, 320.5, 320.75, 321.0, 321.25, 321.5, 321.75, 322.0, 322.25, 322.5, 322.75, 323.0, 323.25, 323.5, 323.75, 324.0, 324.25, 324.5, 324.75, 325.0, 325.25, 325.5, 325.75, 326.0, 326.25, 326.5, 326.75, 327.0, 327.25, 327.5, 327.75, 328.0, 328.25, 328.5, 328.75, 329.0, 329.25, 329.5, 329.75, 330.0, 330.25, 330.5, 330.75, 331.0, 331.25, 331.5, 331.75, 332.0, 332.25, 332.5, 332.75, 333.0, 333.25, 333.5, 333.75, 334.0, 334.25, 334.5, 334.75, 335.0, 335.25, 335.5, 335.75, 336.0, 336.25, 336.5, 336.75, 337.0, 337.25, 337.5, 337.75, 338.0, 338.25, 338.5, 338.75, 339.0, 339.25, 339.5, 339.75, 340.0, 340.25, 340.5, 340.75, 341.0, 341.25, 341.5, 341.75, 342.0, 342.25, 342.5, 342.75, 343.0, 343.25, 343.5, 343.75, 344.0, 344.25, 344.5, 344.75, 345.0, 345.25, 345.5, 345.75, 346.0, 346.25, 346.5, 346.75, 347.0, 347.25, 347.5, 347.75, 348.0, 348.25, 348.5, 348.75, 349.0, 349.25, 349.5, 349.75, 350.0, 350.25, 350.5, 350.75, 351.0, 351.25, 351.5, 351.75, 352.0, 352.25, 352.5, 352.75, 353.0, 353.25, 353.5, 353.75, 354.0, 354.25, 354.5, 354.75, 355.0, 355.25, 355.5, 355.75, 356.0, 356.25, 356.5, 356.75, 357.0, 357.25, 357.5, 357.75, 358.0, 358.25, 358.5, 358.75, 359.0, 359.25, 359.5, 359.75], 'data_grid_type': 'equiangular', 'in_channels': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73], 'out_channels': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73], 'dataset': {'name': '74ch-6hourly', 'description': 'ERA5 data at 6 hourly frequency with snapshots at 0000, 0600, 1200, 1800, UTC. First snapshot in each file is Jan 01 0000 UTC. ', 'metadata_file': '/pscratch/sd/p/pharring/74var-6hourly/staging/data.json'}, 'wandb_run_id': 'jdwsvk36'}, '_yaml_filename': '/global/u2/a/amahesh/ms_finetune/modulus-makani-fork/config/sfnonet.yaml', '_config_name': 'multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2', 'wandb_group': 'multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2-0.1.0', 'scheduler': 'CosineAnnealingLR', 'max_epochs': 20, 'scheduler_T_max': 20, 'lr': 0.0001, 'load_counters': False, 'load_optimizer': False, 'load_scheduler': False, 'finetune': True, 'pretrained_checkpoint_path': '/pscratch/sd/a/amahesh/recovered_fcn_training/modulus-makani_runs-0.1.0-fcndev_stats/sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/training_checkpoints/best_ckpt_mp0.tar', 'embed_dim': 620, 'num_layers': 8, 'scale_factor': 2, 'hard_thresholding_fraction': 1.0, 'loss': 'weighted squared temp-std geometric l2', 'valid_autoreg_steps': 1, 'metadata_json_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/data.json', 'train_data_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/train', 'valid_data_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/valid', 'exp_dir': '/pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/', 'n_years': 1, 'img_shape_x': 721, 'img_shape_y': 1440, 'min_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/stats/mins.npy', 'max_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/stats/maxs.npy', 'time_means_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/stats_fcndev/time_means.npy', 'global_means_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/stats_fcndev/global_means.npy', 'global_stds_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/stats_fcndev/global_stds.npy', 'time_diff_means_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/stats_fcndev/time_diff_means.npy', 'time_diff_stds_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/stats_fcndev/time_diff_stds.npy', 'nettype': 'SFNO', 'model_grid_type': 'equiangular', 'sht_grid_type': 'legendre-gauss', 'filter_type': 'linear', 'complex_activation': 'real', 'normalization_layer': 'instance_norm', 'use_mlp': True, 'mlp_mode': 'serial', 'mlp_ratio': 2, 'separable': False, 'operator_type': 'dhconv', 'activation_function': 'gelu', 'pos_embed': 'none', 'channel_weights': 'auto', 'n_eval_samples': 8760, 'batch_size': 1, 'weight_decay': 0.0, 'scheduler_factor': 0.1, 'scheduler_patience': 10, 'scheduler_step_size': 100, 'scheduler_gamma': 0.5, 'lr_warmup_steps': 0, 'verbose': False, 'wireup_info': 'mpi', 'wireup_store': 'tcp', 'num_data_workers': 2, 'num_visualization_workers': 2, 'dt': 1, 'n_history': 0, 'prediction_type': 'iterative', 'prediction_length': 35, 'n_initial_conditions': 5, 'n_train_samples_per_epoch': 54000, 'ics_type': 'specify_number', 'save_raw_forecasts': True, 'save_channel': False, 'masked_acc': False, 'maskpath': None, 'perturb': False, 'add_noise': False, 'noise_std': 0.0, 'target': 'default', 'normalize_residual': False, 'channel_names': ['u10m', 'v10m', 'u100m', 'v100m', 't2m', 'sp', 'msl', 'tcwv', 'd2m', 'u50', 'u100', 'u150', 'u200', 'u250', 'u300', 'u400', 'u500', 'u600', 'u700', 'u850', 'u925', 'u1000', 'v50', 'v100', 'v150', 'v200', 'v250', 'v300', 'v400', 'v500', 'v600', 'v700', 'v850', 'v925', 'v1000', 'z50', 'z100', 'z150', 'z200', 'z250', 'z300', 'z400', 'z500', 'z600', 'z700', 'z850', 'z925', 'z1000', 't50', 't100', 't150', 't200', 't250', 't300', 't400', 't500', 't600', 't700', 't850', 't925', 't1000', 'q50', 'q100', 'q150', 'q200', 'q250', 'q300', 'q400', 'q500', 'q600', 'q700', 'q850', 'q925', 'q1000'], 'normalization': 'zscore', 'add_grid': True, 'gridtype': 'sinusoidal', 'grid_num_frequencies': 16, 'roll': False, 'add_zenith': True, 'add_orography': True, 'orography_path': '/global/cfs/cdirs/m3522/cmip6/ERA5/e5.oper.invariant/197901/e5.oper.invariant.128_129_z.ll025sc.1979010100_1979010100.nc', 'add_landmask': True, 'landmask_path': '/global/cfs/cdirs/m3522/cmip6/ERA5/e5.oper.invariant/197901/e5.oper.invariant.128_172_lsm.ll025sc.1979010100_1979010100.nc', 'log_to_screen': True, 'log_to_wandb': True, 'log_video': 20, 'save_checkpoint': 'legacy', 'optimizer_type': 'AdamW', 'optimizer_beta1': 0.9, 'optimizer_beta2': 0.95, 'optimizer_max_grad_norm': 32, 'crop_size_x': None, 'crop_size_y': None, 'inf_data_path': '/pscratch/sd/p/pharring/74var-6hourly/staging/out_of_sample', 'wandb_name': None, 'wandb_project': 'ERA5_sfno', 'wandb_entity': 'weatherbenching', 'pos_drop_rate': 0.1, 'initialization_seed': 77, 'epsilon_factor': 0, 'fin_parallel_size': 1, 'fout_parallel_size': 1, 'h_parallel_size': 4, 'w_parallel_size': 1, 'model_parallel_sizes': [4, 1, 1, 1], 'model_parallel_names': ['h', 'w', 'fin', 'fout'], 'parameters_reduction_buffer_count': 1, 'load_checkpoint': 'legacy', 'world_size': 128, 'global_batch_size': 32, 'experiment_dir': '/pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77', 'checkpoint_path': '/pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/training_checkpoints/ckpt_mp{mp_rank}.tar', 'best_checkpoint_path': '/pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77/training_checkpoints/best_ckpt_mp{mp_rank}.tar', 'resuming': False, 'amp_mode': 'bf16', 'jit_mode': 'none', 'cuda_graph_mode': 'none', 'skip_validation': False, 'enable_odirect': False, 'checkpointing': 0, 'enable_synthetic_data': False, 'split_data_channels': False, 'print_timings_frequency': -1, 'multistep_count': 2, 'n_future': 1, 'enable_benchy': False, 'disable_ddp': False, 'enable_grad_anomaly_detection': False, 'wandb_dir': '/pscratch/sd/a/amahesh/fcn_training/modulus-makani_runs-0.1.0gmd-fcndev_stats/multistep_sfno_linear_74chq_sc2_layers8_edim620_wstgl2/v0.1.0-seed77', 'h5_path': 'fields', 'dhours': 6, 'lat': [90.0, 89.75, 89.5, 89.25, 89.0, 88.75, 88.5, 88.25, 88.0, 87.75, 87.5, 87.25, 87.0, 86.75, 86.5, 86.25, 86.0, 85.75, 85.5, 85.25, 85.0, 84.75, 84.5, 84.25, 84.0, 83.75, 83.5, 83.25, 83.0, 82.75, 82.5, 82.25, 82.0, 81.75, 81.5, 81.25, 81.0, 80.75, 80.5, 80.25, 80.0, 79.75, 79.5, 79.25, 79.0, 78.75, 78.5, 78.25, 78.0, 77.75, 77.5, 77.25, 77.0, 76.75, 76.5, 76.25, 76.0, 75.75, 75.5, 75.25, 75.0, 74.75, 74.5, 74.25, 74.0, 73.75, 73.5, 73.25, 73.0, 72.75, 72.5, 72.25, 72.0, 71.75, 71.5, 71.25, 71.0, 70.75, 70.5, 70.25, 70.0, 69.75, 69.5, 69.25, 69.0, 68.75, 68.5, 68.25, 68.0, 67.75, 67.5, 67.25, 67.0, 66.75, 66.5, 66.25, 66.0, 65.75, 65.5, 65.25, 65.0, 64.75, 64.5, 64.25, 64.0, 63.75, 63.5, 63.25, 63.0, 62.75, 62.5, 62.25, 62.0, 61.75, 61.5, 61.25, 61.0, 60.75, 60.5, 60.25, 60.0, 59.75, 59.5, 59.25, 59.0, 58.75, 58.5, 58.25, 58.0, 57.75, 57.5, 57.25, 57.0, 56.75, 56.5, 56.25, 56.0, 55.75, 55.5, 55.25, 55.0, 54.75, 54.5, 54.25, 54.0, 53.75, 53.5, 53.25, 53.0, 52.75, 52.5, 52.25, 52.0, 51.75, 51.5, 51.25, 51.0, 50.75, 50.5, 50.25, 50.0, 49.75, 49.5, 49.25, 49.0, 48.75, 48.5, 48.25, 48.0, 47.75, 47.5, 47.25, 47.0, 46.75, 46.5, 46.25, 46.0, 45.75, 45.5, 45.25, 45.0, 44.75, 44.5, 44.25, 44.0, 43.75, 43.5, 43.25, 43.0, 42.75, 42.5, 42.25, 42.0, 41.75, 41.5, 41.25, 41.0, 40.75, 40.5, 40.25, 40.0, 39.75, 39.5, 39.25, 39.0, 38.75, 38.5, 38.25, 38.0, 37.75, 37.5, 37.25, 37.0, 36.75, 36.5, 36.25, 36.0, 35.75, 35.5, 35.25, 35.0, 34.75, 34.5, 34.25, 34.0, 33.75, 33.5, 33.25, 33.0, 32.75, 32.5, 32.25, 32.0, 31.75, 31.5, 31.25, 31.0, 30.75, 30.5, 30.25, 30.0, 29.75, 29.5, 29.25, 29.0, 28.75, 28.5, 28.25, 28.0, 27.75, 27.5, 27.25, 27.0, 26.75, 26.5, 26.25, 26.0, 25.75, 25.5, 25.25, 25.0, 24.75, 24.5, 24.25, 24.0, 23.75, 23.5, 23.25, 23.0, 22.75, 22.5, 22.25, 22.0, 21.75, 21.5, 21.25, 21.0, 20.75, 20.5, 20.25, 20.0, 19.75, 19.5, 19.25, 19.0, 18.75, 18.5, 18.25, 18.0, 17.75, 17.5, 17.25, 17.0, 16.75, 16.5, 16.25, 16.0, 15.75, 15.5, 15.25, 15.0, 14.75, 14.5, 14.25, 14.0, 13.75, 13.5, 13.25, 13.0, 12.75, 12.5, 12.25, 12.0, 11.75, 11.5, 11.25, 11.0, 10.75, 10.5, 10.25, 10.0, 9.75, 9.5, 9.25, 9.0, 8.75, 8.5, 8.25, 8.0, 7.75, 7.5, 7.25, 7.0, 6.75, 6.5, 6.25, 6.0, 5.75, 5.5, 5.25, 5.0, 4.75, 4.5, 4.25, 4.0, 3.75, 3.5, 3.25, 3.0, 2.75, 2.5, 2.25, 2.0, 1.75, 1.5, 1.25, 1.0, 0.75, 0.5, 0.25, 0.0, -0.25, -0.5, -0.75, -1.0, -1.25, -1.5, -1.75, -2.0, -2.25, -2.5, -2.75, -3.0, -3.25, -3.5, -3.75, -4.0, -4.25, -4.5, -4.75, -5.0, -5.25, -5.5, -5.75, -6.0, -6.25, -6.5, -6.75, -7.0, -7.25, -7.5, -7.75, -8.0, -8.25, -8.5, -8.75, -9.0, -9.25, -9.5, -9.75, -10.0, -10.25, -10.5, -10.75, -11.0, -11.25, -11.5, -11.75, -12.0, -12.25, -12.5, -12.75, -13.0, -13.25, -13.5, -13.75, -14.0, -14.25, -14.5, -14.75, -15.0, -15.25, -15.5, -15.75, -16.0, -16.25, -16.5, -16.75, -17.0, -17.25, -17.5, -17.75, -18.0, -18.25, -18.5, -18.75, -19.0, -19.25, -19.5, -19.75, -20.0, -20.25, -20.5, -20.75, -21.0, -21.25, -21.5, -21.75, -22.0, -22.25, -22.5, -22.75, -23.0, -23.25, -23.5, -23.75, -24.0, -24.25, -24.5, -24.75, -25.0, -25.25, -25.5, -25.75, -26.0, -26.25, -26.5, -26.75, -27.0, -27.25, -27.5, -27.75, -28.0, -28.25, -28.5, -28.75, -29.0, -29.25, -29.5, -29.75, -30.0, -30.25, -30.5, -30.75, -31.0, -31.25, -31.5, -31.75, -32.0, -32.25, -32.5, -32.75, -33.0, -33.25, -33.5, -33.75, -34.0, -34.25, -34.5, -34.75, -35.0, -35.25, -35.5, -35.75, -36.0, -36.25, -36.5, -36.75, -37.0, -37.25, -37.5, -37.75, -38.0, -38.25, -38.5, -38.75, -39.0, -39.25, -39.5, -39.75, -40.0, -40.25, -40.5, -40.75, -41.0, -41.25, -41.5, -41.75, -42.0, -42.25, -42.5, -42.75, -43.0, -43.25, -43.5, -43.75, -44.0, -44.25, -44.5, -44.75, -45.0, -45.25, -45.5, -45.75, -46.0, -46.25, -46.5, -46.75, -47.0, -47.25, -47.5, -47.75, -48.0, -48.25, -48.5, -48.75, -49.0, -49.25, -49.5, -49.75, -50.0, -50.25, -50.5, -50.75, -51.0, -51.25, -51.5, -51.75, -52.0, -52.25, -52.5, -52.75, -53.0, -53.25, -53.5, -53.75, -54.0, -54.25, -54.5, -54.75, -55.0, -55.25, -55.5, -55.75, -56.0, -56.25, -56.5, -56.75, -57.0, -57.25, -57.5, -57.75, -58.0, -58.25, -58.5, -58.75, -59.0, -59.25, -59.5, -59.75, -60.0, -60.25, -60.5, -60.75, -61.0, -61.25, -61.5, -61.75, -62.0, -62.25, -62.5, -62.75, -63.0, -63.25, -63.5, -63.75, -64.0, -64.25, -64.5, -64.75, -65.0, -65.25, -65.5, -65.75, -66.0, -66.25, -66.5, -66.75, -67.0, -67.25, -67.5, -67.75, -68.0, -68.25, -68.5, -68.75, -69.0, -69.25, -69.5, -69.75, -70.0, -70.25, -70.5, -70.75, -71.0, -71.25, -71.5, -71.75, -72.0, -72.25, -72.5, -72.75, -73.0, -73.25, -73.5, -73.75, -74.0, -74.25, -74.5, -74.75, -75.0, -75.25, -75.5, -75.75, -76.0, -76.25, -76.5, -76.75, -77.0, -77.25, -77.5, -77.75, -78.0, -78.25, -78.5, -78.75, -79.0, -79.25, -79.5, -79.75, -80.0, -80.25, -80.5, -80.75, -81.0, -81.25, -81.5, -81.75, -82.0, -82.25, -82.5, -82.75, -83.0, -83.25, -83.5, -83.75, -84.0, -84.25, -84.5, -84.75, -85.0, -85.25, -85.5, -85.75, -86.0, -86.25, -86.5, -86.75, -87.0, -87.25, -87.5, -87.75, -88.0, -88.25, -88.5, -88.75, -89.0, -89.25, -89.5, -89.75, -90.0], 'lon': [0.0, 0.25, 0.5, 0.75, 1.0, 1.25, 1.5, 1.75, 2.0, 2.25, 2.5, 2.75, 3.0, 3.25, 3.5, 3.75, 4.0, 4.25, 4.5, 4.75, 5.0, 5.25, 5.5, 5.75, 6.0, 6.25, 6.5, 6.75, 7.0, 7.25, 7.5, 7.75, 8.0, 8.25, 8.5, 8.75, 9.0, 9.25, 9.5, 9.75, 10.0, 10.25, 10.5, 10.75, 11.0, 11.25, 11.5, 11.75, 12.0, 12.25, 12.5, 12.75, 13.0, 13.25, 13.5, 13.75, 14.0, 14.25, 14.5, 14.75, 15.0, 15.25, 15.5, 15.75, 16.0, 16.25, 16.5, 16.75, 17.0, 17.25, 17.5, 17.75, 18.0, 18.25, 18.5, 18.75, 19.0, 19.25, 19.5, 19.75, 20.0, 20.25, 20.5, 20.75, 21.0, 21.25, 21.5, 21.75, 22.0, 22.25, 22.5, 22.75, 23.0, 23.25, 23.5, 23.75, 24.0, 24.25, 24.5, 24.75, 25.0, 25.25, 25.5, 25.75, 26.0, 26.25, 26.5, 26.75, 27.0, 27.25, 27.5, 27.75, 28.0, 28.25, 28.5, 28.75, 29.0, 29.25, 29.5, 29.75, 30.0, 30.25, 30.5, 30.75, 31.0, 31.25, 31.5, 31.75, 32.0, 32.25, 32.5, 32.75, 33.0, 33.25, 33.5, 33.75, 34.0, 34.25, 34.5, 34.75, 35.0, 35.25, 35.5, 35.75, 36.0, 36.25, 36.5, 36.75, 37.0, 37.25, 37.5, 37.75, 38.0, 38.25, 38.5, 38.75, 39.0, 39.25, 39.5, 39.75, 40.0, 40.25, 40.5, 40.75, 41.0, 41.25, 41.5, 41.75, 42.0, 42.25, 42.5, 42.75, 43.0, 43.25, 43.5, 43.75, 44.0, 44.25, 44.5, 44.75, 45.0, 45.25, 45.5, 45.75, 46.0, 46.25, 46.5, 46.75, 47.0, 47.25, 47.5, 47.75, 48.0, 48.25, 48.5, 48.75, 49.0, 49.25, 49.5, 49.75, 50.0, 50.25, 50.5, 50.75, 51.0, 51.25, 51.5, 51.75, 52.0, 52.25, 52.5, 52.75, 53.0, 53.25, 53.5, 53.75, 54.0, 54.25, 54.5, 54.75, 55.0, 55.25, 55.5, 55.75, 56.0, 56.25, 56.5, 56.75, 57.0, 57.25, 57.5, 57.75, 58.0, 58.25, 58.5, 58.75, 59.0, 59.25, 59.5, 59.75, 60.0, 60.25, 60.5, 60.75, 61.0, 61.25, 61.5, 61.75, 62.0, 62.25, 62.5, 62.75, 63.0, 63.25, 63.5, 63.75, 64.0, 64.25, 64.5, 64.75, 65.0, 65.25, 65.5, 65.75, 66.0, 66.25, 66.5, 66.75, 67.0, 67.25, 67.5, 67.75, 68.0, 68.25, 68.5, 68.75, 69.0, 69.25, 69.5, 69.75, 70.0, 70.25, 70.5, 70.75, 71.0, 71.25, 71.5, 71.75, 72.0, 72.25, 72.5, 72.75, 73.0, 73.25, 73.5, 73.75, 74.0, 74.25, 74.5, 74.75, 75.0, 75.25, 75.5, 75.75, 76.0, 76.25, 76.5, 76.75, 77.0, 77.25, 77.5, 77.75, 78.0, 78.25, 78.5, 78.75, 79.0, 79.25, 79.5, 79.75, 80.0, 80.25, 80.5, 80.75, 81.0, 81.25, 81.5, 81.75, 82.0, 82.25, 82.5, 82.75, 83.0, 83.25, 83.5, 83.75, 84.0, 84.25, 84.5, 84.75, 85.0, 85.25, 85.5, 85.75, 86.0, 86.25, 86.5, 86.75, 87.0, 87.25, 87.5, 87.75, 88.0, 88.25, 88.5, 88.75, 89.0, 89.25, 89.5, 89.75, 90.0, 90.25, 90.5, 90.75, 91.0, 91.25, 91.5, 91.75, 92.0, 92.25, 92.5, 92.75, 93.0, 93.25, 93.5, 93.75, 94.0, 94.25, 94.5, 94.75, 95.0, 95.25, 95.5, 95.75, 96.0, 96.25, 96.5, 96.75, 97.0, 97.25, 97.5, 97.75, 98.0, 98.25, 98.5, 98.75, 99.0, 99.25, 99.5, 99.75, 100.0, 100.25, 100.5, 100.75, 101.0, 101.25, 101.5, 101.75, 102.0, 102.25, 102.5, 102.75, 103.0, 103.25, 103.5, 103.75, 104.0, 104.25, 104.5, 104.75, 105.0, 105.25, 105.5, 105.75, 106.0, 106.25, 106.5, 106.75, 107.0, 107.25, 107.5, 107.75, 108.0, 108.25, 108.5, 108.75, 109.0, 109.25, 109.5, 109.75, 110.0, 110.25, 110.5, 110.75, 111.0, 111.25, 111.5, 111.75, 112.0, 112.25, 112.5, 112.75, 113.0, 113.25, 113.5, 113.75, 114.0, 114.25, 114.5, 114.75, 115.0, 115.25, 115.5, 115.75, 116.0, 116.25, 116.5, 116.75, 117.0, 117.25, 117.5, 117.75, 118.0, 118.25, 118.5, 118.75, 119.0, 119.25, 119.5, 119.75, 120.0, 120.25, 120.5, 120.75, 121.0, 121.25, 121.5, 121.75, 122.0, 122.25, 122.5, 122.75, 123.0, 123.25, 123.5, 123.75, 124.0, 124.25, 124.5, 124.75, 125.0, 125.25, 125.5, 125.75, 126.0, 126.25, 126.5, 126.75, 127.0, 127.25, 127.5, 127.75, 128.0, 128.25, 128.5, 128.75, 129.0, 129.25, 129.5, 129.75, 130.0, 130.25, 130.5, 130.75, 131.0, 131.25, 131.5, 131.75, 132.0, 132.25, 132.5, 132.75, 133.0, 133.25, 133.5, 133.75, 134.0, 134.25, 134.5, 134.75, 135.0, 135.25, 135.5, 135.75, 136.0, 136.25, 136.5, 136.75, 137.0, 137.25, 137.5, 137.75, 138.0, 138.25, 138.5, 138.75, 139.0, 139.25, 139.5, 139.75, 140.0, 140.25, 140.5, 140.75, 141.0, 141.25, 141.5, 141.75, 142.0, 142.25, 142.5, 142.75, 143.0, 143.25, 143.5, 143.75, 144.0, 144.25, 144.5, 144.75, 145.0, 145.25, 145.5, 145.75, 146.0, 146.25, 146.5, 146.75, 147.0, 147.25, 147.5, 147.75, 148.0, 148.25, 148.5, 148.75, 149.0, 149.25, 149.5, 149.75, 150.0, 150.25, 150.5, 150.75, 151.0, 151.25, 151.5, 151.75, 152.0, 152.25, 152.5, 152.75, 153.0, 153.25, 153.5, 153.75, 154.0, 154.25, 154.5, 154.75, 155.0, 155.25, 155.5, 155.75, 156.0, 156.25, 156.5, 156.75, 157.0, 157.25, 157.5, 157.75, 158.0, 158.25, 158.5, 158.75, 159.0, 159.25, 159.5, 159.75, 160.0, 160.25, 160.5, 160.75, 161.0, 161.25, 161.5, 161.75, 162.0, 162.25, 162.5, 162.75, 163.0, 163.25, 163.5, 163.75, 164.0, 164.25, 164.5, 164.75, 165.0, 165.25, 165.5, 165.75, 166.0, 166.25, 166.5, 166.75, 167.0, 167.25, 167.5, 167.75, 168.0, 168.25, 168.5, 168.75, 169.0, 169.25, 169.5, 169.75, 170.0, 170.25, 170.5, 170.75, 171.0, 171.25, 171.5, 171.75, 172.0, 172.25, 172.5, 172.75, 173.0, 173.25, 173.5, 173.75, 174.0, 174.25, 174.5, 174.75, 175.0, 175.25, 175.5, 175.75, 176.0, 176.25, 176.5, 176.75, 177.0, 177.25, 177.5, 177.75, 178.0, 178.25, 178.5, 178.75, 179.0, 179.25, 179.5, 179.75, 180.0, 180.25, 180.5, 180.75, 181.0, 181.25, 181.5, 181.75, 182.0, 182.25, 182.5, 182.75, 183.0, 183.25, 183.5, 183.75, 184.0, 184.25, 184.5, 184.75, 185.0, 185.25, 185.5, 185.75, 186.0, 186.25, 186.5, 186.75, 187.0, 187.25, 187.5, 187.75, 188.0, 188.25, 188.5, 188.75, 189.0, 189.25, 189.5, 189.75, 190.0, 190.25, 190.5, 190.75, 191.0, 191.25, 191.5, 191.75, 192.0, 192.25, 192.5, 192.75, 193.0, 193.25, 193.5, 193.75, 194.0, 194.25, 194.5, 194.75, 195.0, 195.25, 195.5, 195.75, 196.0, 196.25, 196.5, 196.75, 197.0, 197.25, 197.5, 197.75, 198.0, 198.25, 198.5, 198.75, 199.0, 199.25, 199.5, 199.75, 200.0, 200.25, 200.5, 200.75, 201.0, 201.25, 201.5, 201.75, 202.0, 202.25, 202.5, 202.75, 203.0, 203.25, 203.5, 203.75, 204.0, 204.25, 204.5, 204.75, 205.0, 205.25, 205.5, 205.75, 206.0, 206.25, 206.5, 206.75, 207.0, 207.25, 207.5, 207.75, 208.0, 208.25, 208.5, 208.75, 209.0, 209.25, 209.5, 209.75, 210.0, 210.25, 210.5, 210.75, 211.0, 211.25, 211.5, 211.75, 212.0, 212.25, 212.5, 212.75, 213.0, 213.25, 213.5, 213.75, 214.0, 214.25, 214.5, 214.75, 215.0, 215.25, 215.5, 215.75, 216.0, 216.25, 216.5, 216.75, 217.0, 217.25, 217.5, 217.75, 218.0, 218.25, 218.5, 218.75, 219.0, 219.25, 219.5, 219.75, 220.0, 220.25, 220.5, 220.75, 221.0, 221.25, 221.5, 221.75, 222.0, 222.25, 222.5, 222.75, 223.0, 223.25, 223.5, 223.75, 224.0, 224.25, 224.5, 224.75, 225.0, 225.25, 225.5, 225.75, 226.0, 226.25, 226.5, 226.75, 227.0, 227.25, 227.5, 227.75, 228.0, 228.25, 228.5, 228.75, 229.0, 229.25, 229.5, 229.75, 230.0, 230.25, 230.5, 230.75, 231.0, 231.25, 231.5, 231.75, 232.0, 232.25, 232.5, 232.75, 233.0, 233.25, 233.5, 233.75, 234.0, 234.25, 234.5, 234.75, 235.0, 235.25, 235.5, 235.75, 236.0, 236.25, 236.5, 236.75, 237.0, 237.25, 237.5, 237.75, 238.0, 238.25, 238.5, 238.75, 239.0, 239.25, 239.5, 239.75, 240.0, 240.25, 240.5, 240.75, 241.0, 241.25, 241.5, 241.75, 242.0, 242.25, 242.5, 242.75, 243.0, 243.25, 243.5, 243.75, 244.0, 244.25, 244.5, 244.75, 245.0, 245.25, 245.5, 245.75, 246.0, 246.25, 246.5, 246.75, 247.0, 247.25, 247.5, 247.75, 248.0, 248.25, 248.5, 248.75, 249.0, 249.25, 249.5, 249.75, 250.0, 250.25, 250.5, 250.75, 251.0, 251.25, 251.5, 251.75, 252.0, 252.25, 252.5, 252.75, 253.0, 253.25, 253.5, 253.75, 254.0, 254.25, 254.5, 254.75, 255.0, 255.25, 255.5, 255.75, 256.0, 256.25, 256.5, 256.75, 257.0, 257.25, 257.5, 257.75, 258.0, 258.25, 258.5, 258.75, 259.0, 259.25, 259.5, 259.75, 260.0, 260.25, 260.5, 260.75, 261.0, 261.25, 261.5, 261.75, 262.0, 262.25, 262.5, 262.75, 263.0, 263.25, 263.5, 263.75, 264.0, 264.25, 264.5, 264.75, 265.0, 265.25, 265.5, 265.75, 266.0, 266.25, 266.5, 266.75, 267.0, 267.25, 267.5, 267.75, 268.0, 268.25, 268.5, 268.75, 269.0, 269.25, 269.5, 269.75, 270.0, 270.25, 270.5, 270.75, 271.0, 271.25, 271.5, 271.75, 272.0, 272.25, 272.5, 272.75, 273.0, 273.25, 273.5, 273.75, 274.0, 274.25, 274.5, 274.75, 275.0, 275.25, 275.5, 275.75, 276.0, 276.25, 276.5, 276.75, 277.0, 277.25, 277.5, 277.75, 278.0, 278.25, 278.5, 278.75, 279.0, 279.25, 279.5, 279.75, 280.0, 280.25, 280.5, 280.75, 281.0, 281.25, 281.5, 281.75, 282.0, 282.25, 282.5, 282.75, 283.0, 283.25, 283.5, 283.75, 284.0, 284.25, 284.5, 284.75, 285.0, 285.25, 285.5, 285.75, 286.0, 286.25, 286.5, 286.75, 287.0, 287.25, 287.5, 287.75, 288.0, 288.25, 288.5, 288.75, 289.0, 289.25, 289.5, 289.75, 290.0, 290.25, 290.5, 290.75, 291.0, 291.25, 291.5, 291.75, 292.0, 292.25, 292.5, 292.75, 293.0, 293.25, 293.5, 293.75, 294.0, 294.25, 294.5, 294.75, 295.0, 295.25, 295.5, 295.75, 296.0, 296.25, 296.5, 296.75, 297.0, 297.25, 297.5, 297.75, 298.0, 298.25, 298.5, 298.75, 299.0, 299.25, 299.5, 299.75, 300.0, 300.25, 300.5, 300.75, 301.0, 301.25, 301.5, 301.75, 302.0, 302.25, 302.5, 302.75, 303.0, 303.25, 303.5, 303.75, 304.0, 304.25, 304.5, 304.75, 305.0, 305.25, 305.5, 305.75, 306.0, 306.25, 306.5, 306.75, 307.0, 307.25, 307.5, 307.75, 308.0, 308.25, 308.5, 308.75, 309.0, 309.25, 309.5, 309.75, 310.0, 310.25, 310.5, 310.75, 311.0, 311.25, 311.5, 311.75, 312.0, 312.25, 312.5, 312.75, 313.0, 313.25, 313.5, 313.75, 314.0, 314.25, 314.5, 314.75, 315.0, 315.25, 315.5, 315.75, 316.0, 316.25, 316.5, 316.75, 317.0, 317.25, 317.5, 317.75, 318.0, 318.25, 318.5, 318.75, 319.0, 319.25, 319.5, 319.75, 320.0, 320.25, 320.5, 320.75, 321.0, 321.25, 321.5, 321.75, 322.0, 322.25, 322.5, 322.75, 323.0, 323.25, 323.5, 323.75, 324.0, 324.25, 324.5, 324.75, 325.0, 325.25, 325.5, 325.75, 326.0, 326.25, 326.5, 326.75, 327.0, 327.25, 327.5, 327.75, 328.0, 328.25, 328.5, 328.75, 329.0, 329.25, 329.5, 329.75, 330.0, 330.25, 330.5, 330.75, 331.0, 331.25, 331.5, 331.75, 332.0, 332.25, 332.5, 332.75, 333.0, 333.25, 333.5, 333.75, 334.0, 334.25, 334.5, 334.75, 335.0, 335.25, 335.5, 335.75, 336.0, 336.25, 336.5, 336.75, 337.0, 337.25, 337.5, 337.75, 338.0, 338.25, 338.5, 338.75, 339.0, 339.25, 339.5, 339.75, 340.0, 340.25, 340.5, 340.75, 341.0, 341.25, 341.5, 341.75, 342.0, 342.25, 342.5, 342.75, 343.0, 343.25, 343.5, 343.75, 344.0, 344.25, 344.5, 344.75, 345.0, 345.25, 345.5, 345.75, 346.0, 346.25, 346.5, 346.75, 347.0, 347.25, 347.5, 347.75, 348.0, 348.25, 348.5, 348.75, 349.0, 349.25, 349.5, 349.75, 350.0, 350.25, 350.5, 350.75, 351.0, 351.25, 351.5, 351.75, 352.0, 352.25, 352.5, 352.75, 353.0, 353.25, 353.5, 353.75, 354.0, 354.25, 354.5, 354.75, 355.0, 355.25, 355.5, 355.75, 356.0, 356.25, 356.5, 356.75, 357.0, 357.25, 357.5, 357.75, 358.0, 358.25, 358.5, 358.75, 359.0, 359.25, 359.5, 359.75], 'data_grid_type': 'equiangular', 'in_channels': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73], 'out_channels': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73], 'dataset': {'name': '74ch-6hourly', 'description': 'ERA5 data at 6 hourly frequency with snapshots at 0000, 0600, 1200, 1800, UTC. First snapshot in each file is Jan 01 0000 UTC. ', 'metadata_file': '/pscratch/sd/p/pharring/74var-6hourly/staging/data.json'}, 'wandb_run_id': 'jdwsvk36'}
2025-09-17 21:58:17,569 INFO    MainThread:1208058 [wandb_init.py:init():614] starting backend
2025-09-17 21:58:17,569 INFO    MainThread:1208058 [wandb_init.py:init():618] setting up manager
2025-09-17 21:58:17,590 INFO    MainThread:1208058 [wandb_init.py:init():624] backend started and connected
2025-09-17 21:58:17,632 INFO    MainThread:1208058 [wandb_init.py:init():716] updated telemetry
2025-09-17 21:58:17,701 INFO    MainThread:1208058 [wandb_init.py:init():749] communicating run to backend with 90.0 second timeout
2025-09-17 21:58:18,544 INFO    MainThread:1208058 [wandb_run.py:_on_init():2254] communicating current version
2025-09-17 21:58:18,696 INFO    MainThread:1208058 [wandb_run.py:_on_init():2263] got version response upgrade_message: "wandb version 0.21.4 is available!  To upgrade, please run:\n $ pip install wandb --upgrade"

2025-09-17 21:58:18,696 INFO    MainThread:1208058 [wandb_init.py:init():800] starting run threads in backend
2025-09-17 21:58:18,874 INFO    MainThread:1208058 [wandb_run.py:_console_start():2233] atexit reg
2025-09-17 21:58:18,875 INFO    MainThread:1208058 [wandb_run.py:_redirect():2088] redirect: wrap_raw
2025-09-17 21:58:18,875 INFO    MainThread:1208058 [wandb_run.py:_redirect():2153] Wrapping output streams.
2025-09-17 21:58:18,875 INFO    MainThread:1208058 [wandb_run.py:_redirect():2178] Redirects installed.
2025-09-17 21:58:18,877 INFO    MainThread:1208058 [wandb_init.py:init():841] run started, returning control to user process
2025-09-17 21:59:33,642 INFO    MainThread:1208058 [wandb_watch.py:watch():51] Watching
2025-09-18 11:22:01,162 WARNING MsgRouterThr:1208058 [router.py:message_loop():77] message_loop has been closed