Started working on galaxy PGC2463611 at Wed Aug  5 20:36:03 2020
Bands touching this brick, g r z
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 253.04434799999999 54.2335358 --width 281 --height 281 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 

runbrick.py starting at 2020-08-05T20:36:04.945098
legacypipe git version: DR9.5.10-12-gf22e529f
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '253.04434799999999', '54.2335358', '--width', '281', '--height', '281', '--pixscale', '0.262', '--threads', '16', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 253.04434799999999 54.2335358 --width 281 --height 281 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0

Parsed RA,Dec 253.04434799999999 54.2335358
Starting process 8836 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059232 MB
Starting process 8839 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB
Starting process 8841 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB
Starting process 8843 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB
Starting process 8845 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 8847 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 8849 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 8850 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB
Starting process 8853 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB
Starting process 8855 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 8857 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 8859 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 8861 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 8864 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 8868 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 8869 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage fit_on_coadds
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-08-05T20:36:06.349079
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Keeping 11 CCDs:
  k4m_160615_064734_ooi_zd_ls9-CCD3 z expnum 95405 exptime 80.105 propid 2016A-0453 seeing 0.94 MJD 57554.283 object MzLS_666483_z 
    mosaic/CP/V4.3/CP20160614/k4m_160615_064734_ooi_zd_ls9.fits.fz
  k4m_170412_112301_ooi_zd_ls9-CCD1 z expnum 121439 exptime 107.105 propid 2016A-0453 seeing 1.06 MJD 57855.474 object MzLS_667217_z 
    mosaic/CP/V4.3/CP20170411/k4m_170412_112301_ooi_zd_ls9.fits.fz
  k4m_170523_095715_ooi_zd_ls9-CCD4 z expnum 129693 exptime 84.101 propid 2016A-0453 seeing 1.19 MJD 57896.415 object MzLS_665745_z 
    mosaic/CP/V4.3/CP20170522/k4m_170523_095715_ooi_zd_ls9.fits.fz
  ksb_170520_101640_ooi_r_ls9-CCD4 r expnum 78930197 exptime 181.5 propid 2015A-0801 seeing 2.04 MJD 57893.428 object 32623 
    90prime/CP/V2.3/CP20170519/ksb_170520_101640_ooi_r_ls9.fits.fz
  ksb_170523_080943_ooi_g_ls9-CCD4 g expnum 78960126 exptime 100.18 propid 2015A-0801 seeing 1.45 MJD 57896.340 object 32623 
    90prime/CP/V2.3/CP20170522/ksb_170523_080943_ooi_g_ls9.fits.fz
  ksb_160712_052408_ooi_r_ls9-CCD2 r expnum 75810061 exptime 233.3 propid BASS seeing 1.73 MJD 57581.225 object 31542 
    90prime/CP/V2.3/CP20160711/ksb_160712_052408_ooi_r_ls9.fits.fz
  ksb_170327_113133_ooi_g_ls9-CCD2 g expnum 78390275 exptime 91.542 propid BASS seeing 1.41 MJD 57839.480 object 31542 
    90prime/CP/V2.3/CP20170326/ksb_170327_113133_ooi_g_ls9.fits.fz
  ksb_170501_090154_ooi_r_ls9-CCD3 r expnum 78740200 exptime 62.028 propid BASS seeing 1.12 MJD 57874.376 object 31553 
    90prime/CP/V2.3/CP20170430/ksb_170501_090154_ooi_r_ls9.fits.fz
  ksb_170523_081418_ooi_g_ls9-CCD3 g expnum 78960128 exptime 103.558 propid 2015A-0801 seeing 1.52 MJD 57896.343 object 31553 
    90prime/CP/V2.3/CP20170522/ksb_170523_081418_ooi_g_ls9.fits.fz
  ksb_160531_080046_ooi_r_ls9-CCD1 r expnum 75390152 exptime 65.0 propid BASS seeing 1.16 MJD 57539.334 object 31541 
    90prime/CP/V2.3/CP20160530/ksb_160531_080046_ooi_r_ls9.fits.fz
  ksb_170602_101151_ooi_g_ls9-CCD1 g expnum 79060155 exptime 124.85 propid 2015A-0801 seeing 1.53 MJD 57906.425 object 31541 
    90prime/CP/V2.3/CP20170601/ksb_170602_101151_ooi_g_ls9.fits.fz
Faked up EXPNUM 78930197
Faked up EXPNUM 78930197
Faked up EXPNUM 78930197
90prime get_fwhm: self.fwhm = 4.49
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3960 4009 and amp range 2016 4031 and image y 15 76 and amp range 0 2047
Range in image: x 0 50 , y 0 62 (with image size 50 x 62)
dzp 0.02499258 -> scaling image by 1.0232859986636875
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 3.61
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3608 3890 and amp range 2006 4053 and image y 902 1185 and amp range 0 2039
Range in image: x 0 283 , y 0 284 (with image size 283 x 284)
dzp -0.00064781494 -> scaling image by 0.9994035183523652
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 4.09
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 840 1130 and amp range 0 2005 and image y 1021 1312 and amp range 0 2039
Range in image: x 0 291 , y 0 292 (with image size 291 x 292)
dzp 0.007397243 -> scaling image by 1.006836374469654
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 78960128
Faked up EXPNUM 78960128
Faked up EXPNUM 78960128
90prime get_fwhm: self.fwhm = 3.35
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 187 353 and amp range 0 2015 and image y 2960 3125 and amp range 2048 4095
Range in image: x 0 167 , y 0 166 (with image size 167 x 166)
dzp -0.045992933 -> scaling image by 0.958523636456738
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 4.59
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2053 2343 and amp range 2048 4095 and image y 3248 3536 and amp range 2022 4060
Range in image: x 0 291 , y 0 289 (with image size 291 x 289)
dzp -0.02499645 -> scaling image by 0.9772404160512723
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 78740200
Faked up EXPNUM 78740200
Faked up EXPNUM 78740200
90prime get_fwhm: self.fwhm = 2.47
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 186 352 and amp range 0 2015 and image y 2965 3130 and amp range 2048 4095
Range in image: x 0 167 , y 0 166 (with image size 167 x 166)
dzp -0.07725406 -> scaling image by 0.9313188157188347
Faked up EXPNUM 78390275
Faked up EXPNUM 78390275
Faked up EXPNUM 78390275
90prime get_fwhm: self.fwhm = 3.11
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 15 135 and amp range 0 2015 and image y 3422 3586 and amp range 2048 4095
Range in image: x 0 121 , y 0 165 (with image size 121 x 165)
dzp 0.051889952 -> scaling image by 1.0489528829333021
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 75390152
Faked up EXPNUM 75390152
Faked up EXPNUM 75390152
90prime get_fwhm: self.fwhm = 2.57
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2122 2288 and amp range 2016 4031 and image y 1442 1607 and amp range 0 2047
Range in image: x 0 167 , y 0 166 (with image size 167 x 166)
dzp -0.0041296566 -> scaling image by 0.9962036700684744
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 79060155
Faked up EXPNUM 79060155
Faked up EXPNUM 79060155
90prime get_fwhm: self.fwhm = 3.37
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2135 2301 and amp range 2016 4031 and image y 1450 1615 and amp range 0 2047
Range in image: x 0 167 , y 0 166 (with image size 167 x 166)
dzp -0.011406615 -> scaling image by 0.9895491130016149
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 78960126
Faked up EXPNUM 78960126
Faked up EXPNUM 78960126
90prime get_fwhm: self.fwhm = 3.19
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3967 4009 and amp range 2016 4031 and image y 15 65 and amp range 0 2047
Range in image: x 0 43 , y 0 51 (with image size 43 x 51)
dzp 0.016679538 -> scaling image by 1.0154810312115594
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 75810061
Faked up EXPNUM 75810061
Faked up EXPNUM 75810061
90prime get_fwhm: self.fwhm = 3.81
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 15 122 and amp range 0 2015 and image y 3421 3584 and amp range 2048 4095
Range in image: x 0 108 , y 0 164 (with image size 108 x 164)
dzp 0.016961161 -> scaling image by 1.0157444649385616
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Stage tims : Wall: 26.38 s, CPU: 1.22 s, VmPeak: 958 MB, VmSize: 958 MB, VmRSS: 176 MB, VmData: 207 MB, maxrss: 0.179768 MB
Stage tims finished: 2020-08-05T20:36:32.728545
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage tims : 6.114 s worker CPU, 65.550 s worker Wall, Wall: 26.382 s, Cores in use: 0.28, Total efficiency (on 16 cores): 1.7 %
Stage tims finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-tims.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-tims.p
Running stage refs
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage refs at 2020-08-05T20:36:32.813089
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-01171.fits
Matched 13 large galaxies to Gaia stars.
Matched 0 large galaxies to Tycho-2 stars.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/metrics/cus/reference-custom-253044p54233.fits
Stage refs : Wall: 10.54 s, CPU: 3.17 s, VmPeak: 988 MB, VmSize: 982 MB, VmRSS: 200 MB, VmData: 231 MB, maxrss: 0.20484 MB
Stage refs finished: 2020-08-05T20:36:43.352335
Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 10.541 s, Cores in use: 0.30, Total efficiency (on 16 cores): 1.9 %
Stage refs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-refs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-refs.p
Running stage outliers
Running stage outliers at 2020-08-05T20:36:43.383923
Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/metrics/cus/outlier-mask-custom-253044p54233.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/metrics/cus/outliers-pre-custom-253044p54233.jpg
Image ksb_170523_081418_ooi_g_ls9-CCD3 g : masking 89 positive outlier pixels and 0 negative outlier pixels
Image ksb_170501_090154_ooi_r_ls9-CCD3 r : masking 257 positive outlier pixels and 0 negative outlier pixels
Image ksb_160531_080046_ooi_r_ls9-CCD1 r : masking 351 positive outlier pixels and 0 negative outlier pixels
Image k4m_160615_064734_ooi_zd_ls9-CCD3 z : masking 1785 positive outlier pixels and 0 negative outlier pixels
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/metrics/cus/outlier-mask-custom-253044p54233.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/metrics/cus/outliers-post-custom-253044p54233.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/metrics/cus/outliers-masked-pos-custom-253044p54233.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/metrics/cus/outliers-masked-neg-custom-253044p54233.jpg
Stage outliers : Wall: 1.84 s, CPU: 1.54 s, VmPeak: 991 MB, VmSize: 991 MB, VmRSS: 218 MB, VmData: 245 MB, maxrss: 0.22276 MB
Stage outliers finished: 2020-08-05T20:36:45.220693
Resources for stage outliers : 0.786 s worker CPU, 0.870 s worker Wall, Wall: 1.838 s, Cores in use: 1.26, Total efficiency (on 16 cores): 7.9 %
Stage outliers finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-outliers.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-outliers.p
Running stage halos
Running stage halos at 2020-08-05T20:36:45.245096
Stage halos : Wall: 0.78 s, CPU: 0.76 s, VmPeak: 991 MB, VmSize: 991 MB, VmRSS: 218 MB, VmData: 246 MB, maxrss: 0.2229 MB
Stage halos finished: 2020-08-05T20:36:46.022477
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.779 s, Cores in use: 0.98, Total efficiency (on 16 cores): 6.1 %
Stage halos finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-halos.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-halos.p
Running stage fit_on_coadds
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Running stage fit_on_coadds at 2020-08-05T20:36:46.144532
PSF sigmas in coadd pixels: ['2.35', '2.29', '2.46', '2.48']
Using average PSF sigma 2.395566358689102
Gaussian PSF norm 0.11775703509555277 vs pixelized 0.096465595
average tim pixel scale / coadd scale: 1.729534137354211
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-copsf-g.fits.fz
PSF sigmas in coadd pixels: ['3.31', '2.80', '1.82', '1.89']
Using average PSF sigma 2.454527133479732
Gaussian PSF norm 0.11492836560089613 vs pixelized 0.09748186
average tim pixel scale / coadd scale: 1.7294610056489392
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-copsf-r.fits.fz
PSF sigmas in coadd pixels: ['1.52', '1.72', '1.94']
Using average PSF sigma 1.7273379876689077
Gaussian PSF norm 0.16331186703916192 vs pixelized 0.1371545
average tim pixel scale / coadd scale: 0.9908615580390256
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-copsf-z.fits.fz
Stage fit_on_coadds : Wall: 0.20 s, CPU: 0.17 s, VmPeak: 991 MB, VmSize: 991 MB, VmRSS: 219 MB, VmData: 246 MB, maxrss: 0.224652 MB
Stage fit_on_coadds finished: 2020-08-05T20:36:46.345739
Resources for stage fit_on_coadds : 0.172 s worker CPU, 0.172 s worker Wall, Wall: 0.203 s, Cores in use: 1.70, Total efficiency (on 16 cores): 10.6 %
Stage fit_on_coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-fit_on_coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-fit_on_coadds.p
Running stage srcs
Running stage srcs at 2020-08-05T20:36:46.363330
SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])]
Found 13 potential peaks
Of 13 potential peaks: 5 in veto map, 2 cut by saddle test, 0 cut by aper test, 6 kept
SED z : 6 new peaks
Found 18 potential peaks
Of 18 potential peaks: 14 in veto map, 3 cut by saddle test, 0 cut by aper test, 1 kept
SED r : 1 new peaks
Found 22 potential peaks
Of 22 potential peaks: 18 in veto map, 3 cut by saddle test, 0 cut by aper test, 1 kept
SED g : 1 new peaks
Found 25 potential peaks
Of 25 potential peaks: 20 in veto map, 2 cut by saddle test, 0 cut by aper test, 3 kept
SED Flat : 3 new peaks
Found 26 potential peaks
Of 26 potential peaks: 22 in veto map, 2 cut by saddle test, 0 cut by aper test, 2 kept
SED Red : 2 new peaks
Adding 13 new sources
Median coadd sky for g : 8.3848e-05
Median coadd sky for r : -8.7174005e-05
Median coadd sky for z : -7.6490105e-06
Stage srcs : Wall: 1.32 s, CPU: 1.27 s, VmPeak: 991 MB, VmSize: 991 MB, VmRSS: 220 MB, VmData: 246 MB, maxrss: 0.225156 MB
Stage srcs finished: 2020-08-05T20:36:47.680494
Resources for stage srcs : 0.082 s worker CPU, 0.083 s worker Wall, Wall: 1.318 s, Cores in use: 1.02, Total efficiency (on 16 cores): 6.4 %
Stage srcs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-srcs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-srcs.p
Running stage fitblobs
Running stage fitblobs at 2020-08-05T20:36:47.795580
Blob 1 of 15, id: 3, sources: 2, size: 92x159, npix 8941, brick X: 97,189, Y: 59,218, one pixel: 176 59
Blob 2 of 15, id: 10, sources: 3, size: 79x73, npix 3422, brick X: 202,281, Y: 203,276, one pixel: 258 203
Blob 3 of 15, id: 2, sources: 2, size: 62x52, npix 2084, brick X: 70,132, Y: 30,82, one pixel: 90 30
Blob 4 of 15, id: 12, sources: 3, size: 53x53, npix 1706, brick X: 36,89, Y: 228,281, one pixel: 62 228
Blob 5 of 15, id: 9, sources: 1, size: 32x53, npix 1224, brick X: 0,32, Y: 176,229, one pixel: 5 176
Blob 6 of 15, id: 6, sources: 1, size: 29x29, npix 554, brick X: 40,69, Y: 135,164, one pixel: 54 135
Blob 7 of 15, id: 4, sources: 1, size: 25x26, npix 425, brick X: 242,267, Y: 70,96, one pixel: 254 70
Blob 8 of 15, id: 14, sources: 1, size: 30x20, npix 424, brick X: 198,228, Y: 261,281, one pixel: 211 261
Blob 9 of 15, id: 0, sources: 1, size: 29x16, npix 345, brick X: 134,163, Y: 0,16, one pixel: 135 0
Blob 10 of 15, id: 5, sources: 1, size: 19x20, npix 227, brick X: 199,218, Y: 75,95, one pixel: 207 75
Blob 11 of 15, id: 11, sources: 1, size: 19x20, npix 227, brick X: 157,176, Y: 205,225, one pixel: 165 205
Blob 12 of 15, id: 13, sources: 1, size: 19x19, npix 199, brick X: 124,143, Y: 233,252, one pixel: 132 233
Blob 13 of 15, id: 1, sources: 1, size: 18x19, npix 198, brick X: 263,281, Y: 21,40, one pixel: 271 21
Blob 14 of 15, id: 8, sources: 1, size: 19x18, npix 189, brick X: 185,204, Y: 174,192, one pixel: 193 174
Blob 15 of 15, id: 7, sources: 1, size: 18x18, npix 171, brick X: 74,92, Y: 150,168, one pixel: 83 150
Blob 14 finished, total: Wall: 0.09 s, CPU: 0.08 s, VmPeak: 809 MB, VmSize: 731 MB, VmRSS: 147 MB, VmData: 183 MB, maxrss: 0.150968 MB
Blob 13 finished, total: Wall: 0.08 s, CPU: 0.08 s, VmPeak: 804 MB, VmSize: 731 MB, VmRSS: 145 MB, VmData: 183 MB, maxrss: 0.148976 MB
Blob 5 finished, total: Wall: 0.13 s, CPU: 0.13 s, VmPeak: 808 MB, VmSize: 732 MB, VmRSS: 148 MB, VmData: 183 MB, maxrss: 0.1513 MB
Blob 15 finished, total: Wall: 0.07 s, CPU: 0.07 s, VmPeak: 809 MB, VmSize: 731 MB, VmRSS: 147 MB, VmData: 183 MB, maxrss: 0.150848 MB
Blob 11 finished, total: Wall: 0.25 s, CPU: 0.23 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 93 MB, VmData: 139 MB, maxrss: 0.094884 MB
Blob 10 finished, total: Wall: 0.28 s, CPU: 0.28 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 93 MB, VmData: 139 MB, maxrss: 0.095168 MB
Blob 12 finished, total: Wall: 0.29 s, CPU: 0.29 s, VmPeak: 807 MB, VmSize: 731 MB, VmRSS: 147 MB, VmData: 183 MB, maxrss: 0.150888 MB
Blob 7 finished, total: Wall: 0.42 s, CPU: 0.42 s, VmPeak: 808 MB, VmSize: 732 MB, VmRSS: 148 MB, VmData: 183 MB, maxrss: 0.151316 MB
Blob 2 finished, total: Wall: 0.56 s, CPU: 0.56 s, VmPeak: 671 MB, VmSize: 671 MB, VmRSS: 95 MB, VmData: 142 MB, maxrss: 0.097004 MB
Blob 9 finished, total: Wall: 0.57 s, CPU: 0.56 s, VmPeak: 808 MB, VmSize: 732 MB, VmRSS: 146 MB, VmData: 183 MB, maxrss: 0.149472 MB
Blob 6 finished, total: Wall: 0.60 s, CPU: 0.60 s, VmPeak: 669 MB, VmSize: 669 MB, VmRSS: 92 MB, VmData: 140 MB, maxrss: 0.093964 MB
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.05403, 54.24420), pm (-4.8, -13.9), parallax 0.167 with NanoMaggies: g=16.3, r=16.5, z=15.5
Reset brightness to NanoMaggies: g=16.4, r=15.8, z=15.6
Blob 4 finished, total: Wall: 0.76 s, CPU: 0.76 s, VmPeak: 804 MB, VmSize: 731 MB, VmRSS: 147 MB, VmData: 183 MB, maxrss: 0.150488 MB
Blob 8 finished, total: Wall: 0.90 s, CPU: 0.90 s, VmPeak: 804 MB, VmSize: 731 MB, VmRSS: 147 MB, VmData: 183 MB, maxrss: 0.150988 MB
Blob 3 finished, total: Wall: 1.34 s, CPU: 1.33 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 93 MB, VmData: 139 MB, maxrss: 0.095152 MB
Known galaxy.  Initial shape: EllipseWithPriors(0.25): log r_e=1.97496, ee1=0.695251, ee2=0.399069
Measuring 5 iterative sources
Blob 1 finished, total: Wall: 7.14 s, CPU: 7.13 s, VmPeak: 809 MB, VmSize: 732 MB, VmRSS: 151 MB, VmData: 184 MB, maxrss: 0.154172 MB
Old catalog: 30
New catalog: 21
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/metrics/cus/blobs-custom-253044p54233.fits.gz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/metrics/cus/all-models-custom-253044p54233.fits
Stage fitblobs : Wall: 7.71 s, CPU: 0.58 s, VmPeak: 991 MB, VmSize: 991 MB, VmRSS: 220 MB, VmData: 246 MB, maxrss: 0.225512 MB
Stage fitblobs finished: 2020-08-05T20:36:55.501072
Resources for stage fitblobs : 13.649 s worker CPU, 13.781 s worker Wall, Wall: 7.707 s, Cores in use: 1.85, Total efficiency (on 16 cores): 11.5 %
Stage fitblobs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-fitblobs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-fitblobs.p
Running stage coadds
Running stage coadds at 2020-08-05T20:36:55.514664
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-ccds.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-image-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-invvar-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-nexp-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-depth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-galdepth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-psfsize-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-model-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-chi2-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-blobmodel-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-image-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-invvar-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-nexp-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-depth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-galdepth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-psfsize-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-model-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-chi2-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-blobmodel-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-image-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-invvar-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-nexp-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-depth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-galdepth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-psfsize-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-model-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-chi2-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-blobmodel-z.fits.fz
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-depth.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/tmp-legacysurvey-custom-253044p54233-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/tmp-legacysurvey-custom-253044p54233-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/tmp-legacysurvey-custom-253044p54233-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/tmp-legacysurvey-custom-253044p54233-resid.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-resid.jpg
Stage coadds : Wall: 3.25 s, CPU: 1.08 s, VmPeak: 991 MB, VmSize: 991 MB, VmRSS: 221 MB, VmData: 246 MB, maxrss: 0.22596 MB
Stage coadds finished: 2020-08-05T20:36:58.764215
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage coadds : 7.823 s worker CPU, 13.452 s worker Wall, Wall: 3.251 s, Cores in use: 2.74, Total efficiency (on 16 cores): 17.1 %
Stage coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-coadds.p
Running stage wise_forced
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage wise_forced at 2020-08-05T20:36:58.783591
Cut to 1 unWISE tiles
Reading WISE tile 2527p545 band 1
Reading WISE tile 2527p545 band 4
Reading WISE tile 2527p545 band 2
Reading WISE tile 2527p545 band 3
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.04 s, CPU: 0.04 s, VmPeak: 810 MB, VmSize: 810 MB, VmRSS: 152 MB, VmData: 178 MB, maxrss: 0.196488 MB
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.04 s, CPU: 0.04 s, VmPeak: 806 MB, VmSize: 804 MB, VmRSS: 144 MB, VmData: 171 MB, maxrss: 0.208108 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w2-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w2-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w2-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w2-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.03 s, CPU: 0.03 s, VmPeak: 809 MB, VmSize: 803 MB, VmRSS: 144 MB, VmData: 171 MB, maxrss: 0.208544 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w1-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w1-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w1-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w1-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.04 s, CPU: 0.04 s, VmPeak: 812 MB, VmSize: 812 MB, VmRSS: 153 MB, VmData: 180 MB, maxrss: 0.209324 MB
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-image-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-model-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-invvar-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-image-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-model-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-invvar-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-image-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-model-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-invvar-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-image-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-model-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-invvar-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/tmp-legacysurvey-custom-253044p54233-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/tmp-legacysurvey-custom-253044p54233-wisemodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-wisemodel.jpg
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Stage wise_forced : Wall: 2.82 s, CPU: 0.89 s, VmPeak: 991 MB, VmSize: 991 MB, VmRSS: 221 MB, VmData: 246 MB, maxrss: 0.226668 MB
Stage wise_forced finished: 2020-08-05T20:37:01.600206
Resources for stage wise_forced : 3.057 s worker CPU, 4.870 s worker Wall, Wall: 2.818 s, Cores in use: 1.40, Total efficiency (on 16 cores): 8.8 %
Stage wise_forced finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-wise_forced.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-wise_forced.p
Running stage writecat
Running stage writecat at 2020-08-05T20:37:01.683904
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-maskbits.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/tractor-i/cus/tractor-custom-253044p54233.fits
Reading SFD maps...
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/tractor/cus/tractor-custom-253044p54233.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/tractor/cus/brick-custom-253044p54233.sha256sum
Stage writecat : Wall: 0.73 s, CPU: 0.57 s, VmPeak: 1124 MB, VmSize: 996 MB, VmRSS: 223 MB, VmData: 246 MB, maxrss: 0.359532 MB
Stage writecat finished: 2020-08-05T20:37:02.415939
Resources for stage writecat : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 0.734 s, Cores in use: 0.78, Total efficiency (on 16 cores): 4.9 %
Stage writecat finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-writecat.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-writecat.p
All done: 31.688 s worker CPU, 98.783 s worker Wall, Wall: 56.087 s, Cores in use: 0.77, Total efficiency (on 16 cores): 4.8 %
Finished galaxy PGC2463611 in 0.985 minutes.
Started working on galaxy PGC2463611 at Wed Aug  5 22:14:32 2020
Bands touching this brick, g r z
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 253.04434799999999 54.2335358 --width 281 --height 281 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 

runbrick.py starting at 2020-08-05T22:14:34.357060
legacypipe git version: DR9.5.10-12-gf22e529f
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '253.04434799999999', '54.2335358', '--width', '281', '--height', '281', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 253.04434799999999 54.2335358 --width 281 --height 281 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0

Parsed RA,Dec 253.04434799999999 54.2335358
Starting process 28924 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059236 MB
Starting process 28925 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059236 MB
Starting process 28926 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB
Starting process 28927 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB
Starting process 28928 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 28929 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 28930 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 28931 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB
Starting process 28932 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB
Starting process 28933 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 28934 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 28935 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 28936 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 28937 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 28938 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 28939 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 28940 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 28941 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 28942 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05928 MB
Starting process 28943 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05928 MB
Starting process 28944 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05928 MB
Starting process 28945 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB
Starting process 28946 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB
Starting process 28947 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059292 MB
Starting process 28949 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB
Starting process 28950 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB
Starting process 28948 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB
Starting process 28951 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB
Starting process 28952 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Starting process 28954 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Starting process 28955 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059308 MB
Starting process 28953 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage fit_on_coadds
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-08-05T22:14:35.015713
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Keeping 11 CCDs:
  k4m_160615_064734_ooi_zd_ls9-CCD3 z expnum 95405 exptime 80.105 propid 2016A-0453 seeing 0.94 MJD 57554.283 object MzLS_666483_z 
    mosaic/CP/V4.3/CP20160614/k4m_160615_064734_ooi_zd_ls9.fits.fz
  k4m_170412_112301_ooi_zd_ls9-CCD1 z expnum 121439 exptime 107.105 propid 2016A-0453 seeing 1.06 MJD 57855.474 object MzLS_667217_z 
    mosaic/CP/V4.3/CP20170411/k4m_170412_112301_ooi_zd_ls9.fits.fz
  k4m_170523_095715_ooi_zd_ls9-CCD4 z expnum 129693 exptime 84.101 propid 2016A-0453 seeing 1.19 MJD 57896.415 object MzLS_665745_z 
    mosaic/CP/V4.3/CP20170522/k4m_170523_095715_ooi_zd_ls9.fits.fz
  ksb_170520_101640_ooi_r_ls9-CCD4 r expnum 78930197 exptime 181.5 propid 2015A-0801 seeing 2.04 MJD 57893.428 object 32623 
    90prime/CP/V2.3/CP20170519/ksb_170520_101640_ooi_r_ls9.fits.fz
  ksb_170523_080943_ooi_g_ls9-CCD4 g expnum 78960126 exptime 100.18 propid 2015A-0801 seeing 1.45 MJD 57896.340 object 32623 
    90prime/CP/V2.3/CP20170522/ksb_170523_080943_ooi_g_ls9.fits.fz
  ksb_160712_052408_ooi_r_ls9-CCD2 r expnum 75810061 exptime 233.3 propid BASS seeing 1.73 MJD 57581.225 object 31542 
    90prime/CP/V2.3/CP20160711/ksb_160712_052408_ooi_r_ls9.fits.fz
  ksb_170327_113133_ooi_g_ls9-CCD2 g expnum 78390275 exptime 91.542 propid BASS seeing 1.41 MJD 57839.480 object 31542 
    90prime/CP/V2.3/CP20170326/ksb_170327_113133_ooi_g_ls9.fits.fz
  ksb_170501_090154_ooi_r_ls9-CCD3 r expnum 78740200 exptime 62.028 propid BASS seeing 1.12 MJD 57874.376 object 31553 
    90prime/CP/V2.3/CP20170430/ksb_170501_090154_ooi_r_ls9.fits.fz
  ksb_170523_081418_ooi_g_ls9-CCD3 g expnum 78960128 exptime 103.558 propid 2015A-0801 seeing 1.52 MJD 57896.343 object 31553 
    90prime/CP/V2.3/CP20170522/ksb_170523_081418_ooi_g_ls9.fits.fz
  ksb_160531_080046_ooi_r_ls9-CCD1 r expnum 75390152 exptime 65.0 propid BASS seeing 1.16 MJD 57539.334 object 31541 
    90prime/CP/V2.3/CP20160530/ksb_160531_080046_ooi_r_ls9.fits.fz
  ksb_170602_101151_ooi_g_ls9-CCD1 g expnum 79060155 exptime 124.85 propid 2015A-0801 seeing 1.53 MJD 57906.425 object 31541 
    90prime/CP/V2.3/CP20170601/ksb_170602_101151_ooi_g_ls9.fits.fz
Faked up EXPNUM 78740200
Faked up EXPNUM 78740200
Faked up EXPNUM 78740200
90prime get_fwhm: self.fwhm = 2.47
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 186 352 and amp range 0 2015 and image y 2965 3130 and amp range 2048 4095
Range in image: x 0 167 , y 0 166 (with image size 167 x 166)
dzp -0.07725406 -> scaling image by 0.9313188157188347
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 78930197
Faked up EXPNUM 78930197
Faked up EXPNUM 78930197
90prime get_fwhm: self.fwhm = 4.49
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3960 4009 and amp range 2016 4031 and image y 15 76 and amp range 0 2047
Range in image: x 0 50 , y 0 62 (with image size 50 x 62)
dzp 0.02499258 -> scaling image by 1.0232859986636875
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 4.09
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 840 1130 and amp range 0 2005 and image y 1021 1312 and amp range 0 2039
Range in image: x 0 291 , y 0 292 (with image size 291 x 292)
dzp 0.007397243 -> scaling image by 1.006836374469654
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 75390152
Faked up EXPNUM 75390152
Faked up EXPNUM 75390152
90prime get_fwhm: self.fwhm = 2.57
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2122 2288 and amp range 2016 4031 and image y 1442 1607 and amp range 0 2047
Range in image: x 0 167 , y 0 166 (with image size 167 x 166)
dzp -0.0041296566 -> scaling image by 0.9962036700684744
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 4.59
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2053 2343 and amp range 2048 4095 and image y 3248 3536 and amp range 2022 4060
Range in image: x 0 291 , y 0 289 (with image size 291 x 289)
dzp -0.02499645 -> scaling image by 0.9772404160512723
Faked up EXPNUM 78960128
Faked up EXPNUM 78960128
Faked up EXPNUM 78960128
90prime get_fwhm: self.fwhm = 3.35
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 187 353 and amp range 0 2015 and image y 2960 3125 and amp range 2048 4095
Range in image: x 0 167 , y 0 166 (with image size 167 x 166)
dzp -0.045992933 -> scaling image by 0.958523636456738
Faked up EXPNUM 78390275
Faked up EXPNUM 78390275
Faked up EXPNUM 78390275
90prime get_fwhm: self.fwhm = 3.11
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 15 135 and amp range 0 2015 and image y 3422 3586 and amp range 2048 4095
Range in image: x 0 121 , y 0 165 (with image size 121 x 165)
dzp 0.051889952 -> scaling image by 1.0489528829333021
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 3.61
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3608 3890 and amp range 2006 4053 and image y 902 1185 and amp range 0 2039
Range in image: x 0 283 , y 0 284 (with image size 283 x 284)
dzp -0.00064781494 -> scaling image by 0.9994035183523652
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 79060155
Faked up EXPNUM 79060155
Faked up EXPNUM 79060155
90prime get_fwhm: self.fwhm = 3.37
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2135 2301 and amp range 2016 4031 and image y 1450 1615 and amp range 0 2047
Range in image: x 0 167 , y 0 166 (with image size 167 x 166)
dzp -0.011406615 -> scaling image by 0.9895491130016149
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 78960126
Faked up EXPNUM 78960126
Faked up EXPNUM 78960126
90prime get_fwhm: self.fwhm = 3.19
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3967 4009 and amp range 2016 4031 and image y 15 65 and amp range 0 2047
Range in image: x 0 43 , y 0 51 (with image size 43 x 51)
dzp 0.016679538 -> scaling image by 1.0154810312115594
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 75810061
Faked up EXPNUM 75810061
Faked up EXPNUM 75810061
90prime get_fwhm: self.fwhm = 3.81
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 15 122 and amp range 0 2015 and image y 3421 3584 and amp range 2048 4095
Range in image: x 0 108 , y 0 164 (with image size 108 x 164)
dzp 0.016961161 -> scaling image by 1.0157444649385616
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Stage tims : Wall: 15.88 s, CPU: 1.25 s, VmPeak: 958 MB, VmSize: 958 MB, VmRSS: 176 MB, VmData: 208 MB, maxrss: 0.180032 MB
Stage tims finished: 2020-08-05T22:14:50.896928
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage tims : 5.879 s worker CPU, 52.300 s worker Wall, Wall: 15.884 s, Cores in use: 0.45, Total efficiency (on 32 cores): 1.4 %
Stage tims finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-tims.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-tims.p
Running stage refs
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage refs at 2020-08-05T22:14:51.014001
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-01171.fits
Matched 13 large galaxies to Gaia stars.
Matched 0 large galaxies to Tycho-2 stars.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/metrics/cus/reference-custom-253044p54233.fits
Stage refs : Wall: 11.13 s, CPU: 2.37 s, VmPeak: 988 MB, VmSize: 982 MB, VmRSS: 200 MB, VmData: 232 MB, maxrss: 0.2051 MB
Stage refs finished: 2020-08-05T22:15:02.139437
Resources for stage refs : 0.000 s worker CPU, 0.001 s worker Wall, Wall: 11.128 s, Cores in use: 0.21, Total efficiency (on 32 cores): 0.7 %
Stage refs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-refs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-refs.p
Running stage outliers
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage outliers at 2020-08-05T22:15:02.175028
Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/metrics/cus/outlier-mask-custom-253044p54233.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/metrics/cus/outliers-pre-custom-253044p54233.jpg
Image ksb_170523_081418_ooi_g_ls9-CCD3 g : masking 89 positive outlier pixels and 0 negative outlier pixels
Image ksb_160531_080046_ooi_r_ls9-CCD1 r : masking 351 positive outlier pixels and 0 negative outlier pixels
Image ksb_170501_090154_ooi_r_ls9-CCD3 r : masking 257 positive outlier pixels and 0 negative outlier pixels
Image k4m_160615_064734_ooi_zd_ls9-CCD3 z : masking 1785 positive outlier pixels and 0 negative outlier pixels
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/metrics/cus/outlier-mask-custom-253044p54233.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/metrics/cus/outliers-post-custom-253044p54233.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/metrics/cus/outliers-masked-pos-custom-253044p54233.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/metrics/cus/outliers-masked-neg-custom-253044p54233.jpg
Stage outliers : Wall: 1.13 s, CPU: 0.71 s, VmPeak: 991 MB, VmSize: 991 MB, VmRSS: 217 MB, VmData: 245 MB, maxrss: 0.221948 MB
Stage outliers finished: 2020-08-05T22:15:03.303028
Resources for stage outliers : 0.970 s worker CPU, 1.533 s worker Wall, Wall: 1.130 s, Cores in use: 1.49, Total efficiency (on 32 cores): 4.7 %
Stage outliers finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-outliers.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-outliers.p
Running stage halos
Running stage halos at 2020-08-05T22:15:03.365739
Stage halos : Wall: 0.29 s, CPU: 0.28 s, VmPeak: 991 MB, VmSize: 991 MB, VmRSS: 217 MB, VmData: 245 MB, maxrss: 0.222148 MB
Stage halos finished: 2020-08-05T22:15:03.655270
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.292 s, Cores in use: 0.98, Total efficiency (on 32 cores): 3.0 %
Stage halos finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-halos.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-halos.p
Running stage fit_on_coadds
Warning: Stellar halo subtraction is only implemented for DECam
Running stage fit_on_coadds at 2020-08-05T22:15:03.756069
PSF sigmas in coadd pixels: ['2.35', '2.29', '2.46', '2.48']
Using average PSF sigma 2.395566358689102
Gaussian PSF norm 0.11775703509555277 vs pixelized 0.096465595
average tim pixel scale / coadd scale: 1.729534137354211
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-copsf-g.fits.fz
PSF sigmas in coadd pixels: ['3.31', '2.80', '1.82', '1.89']
Using average PSF sigma 2.454527133479732
Gaussian PSF norm 0.11492836560089613 vs pixelized 0.09748186
average tim pixel scale / coadd scale: 1.7294610056489392
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-copsf-r.fits.fz
PSF sigmas in coadd pixels: ['1.52', '1.72', '1.94']
Using average PSF sigma 1.7273379876689077
Gaussian PSF norm 0.16331186703916192 vs pixelized 0.13715449
average tim pixel scale / coadd scale: 0.9908615580390256
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-copsf-z.fits.fz
Stage fit_on_coadds : Wall: 0.19 s, CPU: 0.18 s, VmPeak: 991 MB, VmSize: 991 MB, VmRSS: 219 MB, VmData: 245 MB, maxrss: 0.223796 MB
Stage fit_on_coadds finished: 2020-08-05T22:15:03.947402
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage fit_on_coadds : 0.179 s worker CPU, 0.193 s worker Wall, Wall: 0.194 s, Cores in use: 1.87, Total efficiency (on 32 cores): 5.8 %
Stage fit_on_coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-fit_on_coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-fit_on_coadds.p
Running stage srcs
Warning: Stellar halo subtraction is only implemented for DECam
Running stage srcs at 2020-08-05T22:15:04.010851
SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])]
Found 13 potential peaks
Of 13 potential peaks: 5 in veto map, 2 cut by saddle test, 0 cut by aper test, 6 kept
SED z : 6 new peaks
Found 18 potential peaks
Of 18 potential peaks: 14 in veto map, 3 cut by saddle test, 0 cut by aper test, 1 kept
SED r : 1 new peaks
Found 22 potential peaks
Of 22 potential peaks: 18 in veto map, 3 cut by saddle test, 0 cut by aper test, 1 kept
SED g : 1 new peaks
Found 25 potential peaks
Of 25 potential peaks: 20 in veto map, 2 cut by saddle test, 0 cut by aper test, 3 kept
SED Flat : 3 new peaks
Found 26 potential peaks
Of 26 potential peaks: 22 in veto map, 2 cut by saddle test, 0 cut by aper test, 2 kept
SED Red : 2 new peaks
Adding 13 new sources
Median coadd sky for g : 8.3848e-05
Median coadd sky for r : -8.7174005e-05
Median coadd sky for z : -7.649238e-06
Stage srcs : Wall: 0.45 s, CPU: 0.40 s, VmPeak: 991 MB, VmSize: 991 MB, VmRSS: 219 MB, VmData: 245 MB, maxrss: 0.2243 MB
Stage srcs finished: 2020-08-05T22:15:04.456572
Resources for stage srcs : 0.108 s worker CPU, 0.108 s worker Wall, Wall: 0.448 s, Cores in use: 1.14, Total efficiency (on 32 cores): 3.6 %
Stage srcs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-srcs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-srcs.p
Running stage fitblobs
Running stage fitblobs at 2020-08-05T22:15:04.470272
Blob 1 of 15, id: 3, sources: 2, size: 92x159, npix 8941, brick X: 97,189, Y: 59,218, one pixel: 176 59
Blob 2 of 15, id: 10, sources: 3, size: 79x73, npix 3422, brick X: 202,281, Y: 203,276, one pixel: 258 203
Blob 3 of 15, id: 2, sources: 2, size: 62x52, npix 2084, brick X: 70,132, Y: 30,82, one pixel: 90 30
Blob 4 of 15, id: 12, sources: 3, size: 53x53, npix 1706, brick X: 36,89, Y: 228,281, one pixel: 62 228
Blob 5 of 15, id: 9, sources: 1, size: 32x53, npix 1224, brick X: 0,32, Y: 176,229, one pixel: 5 176
Blob 6 of 15, id: 6, sources: 1, size: 29x29, npix 554, brick X: 40,69, Y: 135,164, one pixel: 54 135
Blob 7 of 15, id: 4, sources: 1, size: 25x26, npix 425, brick X: 242,267, Y: 70,96, one pixel: 254 70
Blob 8 of 15, id: 14, sources: 1, size: 30x20, npix 424, brick X: 198,228, Y: 261,281, one pixel: 211 261
Blob 9 of 15, id: 0, sources: 1, size: 29x16, npix 345, brick X: 134,163, Y: 0,16, one pixel: 135 0
Blob 10 of 15, id: 5, sources: 1, size: 19x20, npix 227, brick X: 199,218, Y: 75,95, one pixel: 207 75
Blob 11 of 15, id: 11, sources: 1, size: 19x20, npix 227, brick X: 157,176, Y: 205,225, one pixel: 165 205
Blob 12 of 15, id: 13, sources: 1, size: 19x19, npix 199, brick X: 124,143, Y: 233,252, one pixel: 132 233
Blob 13 of 15, id: 1, sources: 1, size: 18x19, npix 198, brick X: 263,281, Y: 21,40, one pixel: 271 21
Blob 14 of 15, id: 8, sources: 1, size: 19x18, npix 189, brick X: 185,204, Y: 174,192, one pixel: 193 174
Blob 15 of 15, id: 7, sources: 1, size: 18x18, npix 171, brick X: 74,92, Y: 150,168, one pixel: 83 150
Blob 15 finished, total: Wall: 0.05 s, CPU: 0.04 s, VmPeak: 809 MB, VmSize: 731 MB, VmRSS: 146 MB, VmData: 183 MB, maxrss: 0.149904 MB
Blob 13 finished, total: Wall: 0.06 s, CPU: 0.06 s, VmPeak: 807 MB, VmSize: 731 MB, VmRSS: 145 MB, VmData: 183 MB, maxrss: 0.148096 MB
Blob 14 finished, total: Wall: 0.10 s, CPU: 0.06 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 84 MB, VmData: 138 MB, maxrss: 0.086324 MB
Blob 5 finished, total: Wall: 0.16 s, CPU: 0.16 s, VmPeak: 811 MB, VmSize: 731 MB, VmRSS: 147 MB, VmData: 183 MB, maxrss: 0.150784 MB
Blob 11 finished, total: Wall: 0.17 s, CPU: 0.17 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 91 MB, VmData: 139 MB, maxrss: 0.093216 MB
Blob 10 finished, total: Wall: 0.19 s, CPU: 0.19 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 92 MB, VmData: 139 MB, maxrss: 0.0944 MB
Blob 12 finished, total: Wall: 0.20 s, CPU: 0.19 s, VmPeak: 669 MB, VmSize: 669 MB, VmRSS: 92 MB, VmData: 140 MB, maxrss: 0.094564 MB
Blob 7 finished, total: Wall: 0.41 s, CPU: 0.41 s, VmPeak: 670 MB, VmSize: 670 MB, VmRSS: 93 MB, VmData: 141 MB, maxrss: 0.09552 MB
Blob 9 finished, total: Wall: 0.52 s, CPU: 0.52 s, VmPeak: 809 MB, VmSize: 732 MB, VmRSS: 148 MB, VmData: 183 MB, maxrss: 0.151336 MB
Blob 2 finished, total: Wall: 0.58 s, CPU: 0.58 s, VmPeak: 808 MB, VmSize: 732 MB, VmRSS: 146 MB, VmData: 183 MB, maxrss: 0.150012 MB
Blob 6 finished, total: Wall: 0.58 s, CPU: 0.58 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 93 MB, VmData: 140 MB, maxrss: 0.095108 MB
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.05403, 54.24420), pm (-4.8, -13.9), parallax 0.167 with NanoMaggies: g=16.3, r=16.5, z=15.5
Reset brightness to NanoMaggies: g=16.4, r=15.8, z=15.6
Blob 4 finished, total: Wall: 0.76 s, CPU: 0.76 s, VmPeak: 804 MB, VmSize: 731 MB, VmRSS: 147 MB, VmData: 183 MB, maxrss: 0.150808 MB
Blob 8 finished, total: Wall: 0.84 s, CPU: 0.84 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 93 MB, VmData: 139 MB, maxrss: 0.095156 MB
Blob 3 finished, total: Wall: 1.36 s, CPU: 1.31 s, VmPeak: 668 MB, VmSize: 667 MB, VmRSS: 90 MB, VmData: 138 MB, maxrss: 0.092736 MB
Known galaxy.  Initial shape: EllipseWithPriors(0.25): log r_e=1.97496, ee1=0.69525, ee2=0.399069
Measuring 5 iterative sources
Blob 1 finished, total: Wall: 7.02 s, CPU: 7.01 s, VmPeak: 684 MB, VmSize: 675 MB, VmRSS: 103 MB, VmData: 146 MB, maxrss: 0.11362 MB
Old catalog: 30
New catalog: 21
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/metrics/cus/blobs-custom-253044p54233.fits.gz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/metrics/cus/all-models-custom-253044p54233.fits
Stage fitblobs : Wall: 7.37 s, CPU: 0.36 s, VmPeak: 991 MB, VmSize: 991 MB, VmRSS: 219 MB, VmData: 245 MB, maxrss: 0.224656 MB
Stage fitblobs finished: 2020-08-05T22:15:11.841829
Resources for stage fitblobs : 13.088 s worker CPU, 13.254 s worker Wall, Wall: 7.374 s, Cores in use: 1.82, Total efficiency (on 32 cores): 5.7 %
Stage fitblobs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-fitblobs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-fitblobs.p
Running stage coadds
Running stage coadds at 2020-08-05T22:15:11.857962
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-ccds.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-image-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-invvar-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-nexp-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-depth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-galdepth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-psfsize-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-model-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-chi2-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-blobmodel-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-image-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-invvar-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-nexp-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-depth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-galdepth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-psfsize-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-model-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-chi2-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-blobmodel-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-image-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-invvar-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-nexp-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-depth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-galdepth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-psfsize-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-model-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-chi2-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-blobmodel-z.fits.fz
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-depth.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/tmp-legacysurvey-custom-253044p54233-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/tmp-legacysurvey-custom-253044p54233-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/tmp-legacysurvey-custom-253044p54233-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/tmp-legacysurvey-custom-253044p54233-resid.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-resid.jpg
Stage coadds : Wall: 14.57 s, CPU: 0.68 s, VmPeak: 991 MB, VmSize: 991 MB, VmRSS: 220 MB, VmData: 245 MB, maxrss: 0.225108 MB
Stage coadds finished: 2020-08-05T22:15:26.426009
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage coadds : 24.831 s worker CPU, 42.170 s worker Wall, Wall: 14.570 s, Cores in use: 1.75, Total efficiency (on 32 cores): 5.5 %
Stage coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-coadds.p
Running stage wise_forced
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage wise_forced at 2020-08-05T22:15:26.440053
Cut to 1 unWISE tiles
Reading WISE tile 2527p545 band 1
Reading WISE tile 2527p545 band 2
Reading WISE tile 2527p545 band 3
Reading WISE tile 2527p545 band 4
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.05 s, CPU: 0.05 s, VmPeak: 808 MB, VmSize: 804 MB, VmRSS: 146 MB, VmData: 176 MB, maxrss: 0.186764 MB
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.04 s, CPU: 0.04 s, VmPeak: 808 MB, VmSize: 798 MB, VmRSS: 140 MB, VmData: 170 MB, maxrss: 0.202256 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w2-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w2-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w2-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w2-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.05 s, CPU: 0.05 s, VmPeak: 805 MB, VmSize: 803 MB, VmRSS: 144 MB, VmData: 171 MB, maxrss: 0.20748 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w1-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w1-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w1-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w1-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.04 s, CPU: 0.04 s, VmPeak: 865 MB, VmSize: 865 MB, VmRSS: 204 MB, VmData: 237 MB, maxrss: 0.208592 MB
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-image-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-model-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-invvar-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-image-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-model-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-invvar-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-image-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-model-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-invvar-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-image-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-model-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-invvar-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/tmp-legacysurvey-custom-253044p54233-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/tmp-legacysurvey-custom-253044p54233-wisemodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-wisemodel.jpg
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Stage wise_forced : Wall: 3.25 s, CPU: 0.36 s, VmPeak: 991 MB, VmSize: 991 MB, VmRSS: 221 MB, VmData: 245 MB, maxrss: 0.225812 MB
Stage wise_forced finished: 2020-08-05T22:15:29.691085
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage wise_forced : 3.612 s worker CPU, 6.518 s worker Wall, Wall: 3.253 s, Cores in use: 1.22, Total efficiency (on 32 cores): 3.8 %
Stage wise_forced finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-wise_forced.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-wise_forced.p
Running stage writecat
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage writecat at 2020-08-05T22:15:29.733283
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/coadd/cus/custom-253044p54233/legacysurvey-custom-253044p54233-maskbits.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/tractor-i/cus/tractor-custom-253044p54233.fits
Reading SFD maps...
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/tractor/cus/tractor-custom-253044p54233.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/tractor/cus/brick-custom-253044p54233.sha256sum
Stage writecat : Wall: 0.87 s, CPU: 0.53 s, VmPeak: 1124 MB, VmSize: 996 MB, VmRSS: 222 MB, VmData: 245 MB, maxrss: 0.358676 MB
Stage writecat finished: 2020-08-05T22:15:30.602129
Resources for stage writecat : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.871 s, Cores in use: 0.61, Total efficiency (on 32 cores): 1.9 %
Stage writecat finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-writecat.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/253/PGC2463611/PGC2463611-largegalaxy-writecat.p
All done: 48.674 s worker CPU, 116.085 s worker Wall, Wall: 55.651 s, Cores in use: 1.01, Total efficiency (on 32 cores): 3.1 %
Finished galaxy PGC2463611 in 0.971 minutes.
Started working on galaxy PGC2463611 at Fri Aug  7 09:33:15 2020
Bands touching this brick, g r z
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 253.0437484055481 54.23361969915836 --width 281 --height 281 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 

runbrick.py starting at 2020-08-07T09:33:16.603708
legacypipe git version: DR9.5.11-watershed-3-gd509eaaf
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '253.0437484055481', '54.23361969915836', '--width', '281', '--height', '281', '--pixscale', '0.262', '--threads', '16', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 253.0437484055481 54.23361969915836 --width 281 --height 281 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0

Parsed RA,Dec 253.0437484055481 54.23361969915836
Starting process 8704 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 8706 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 8708 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 8710 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 8712 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059544 MB
Starting process 8714 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059492 MB
Starting process 8716 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059492 MB
Starting process 8718 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 8720 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 8722 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 8724 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 8728 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 8730 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 8726 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 8733 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 8735 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage fit_on_coadds
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-08-07T09:33:17.092133
Keeping 11 CCDs:
  k4m_160615_064734_ooi_zd_ls9-CCD3 z expnum 95405 exptime 80.105 propid 2016A-0453 seeing 0.94 MJD 57554.283 object MzLS_666483_z 
    mosaic/CP/V4.3/CP20160614/k4m_160615_064734_ooi_zd_ls9.fits.fz
  k4m_170412_112301_ooi_zd_ls9-CCD1 z expnum 121439 exptime 107.105 propid 2016A-0453 seeing 1.06 MJD 57855.474 object MzLS_667217_z 
    mosaic/CP/V4.3/CP20170411/k4m_170412_112301_ooi_zd_ls9.fits.fz
  k4m_170523_095715_ooi_zd_ls9-CCD4 z expnum 129693 exptime 84.101 propid 2016A-0453 seeing 1.19 MJD 57896.415 object MzLS_665745_z 
    mosaic/CP/V4.3/CP20170522/k4m_170523_095715_ooi_zd_ls9.fits.fz
  ksb_170520_101640_ooi_r_ls9-CCD4 r expnum 78930197 exptime 181.5 propid 2015A-0801 seeing 2.04 MJD 57893.428 object 32623 
    90prime/CP/V2.3/CP20170519/ksb_170520_101640_ooi_r_ls9.fits.fz
  ksb_170523_080943_ooi_g_ls9-CCD4 g expnum 78960126 exptime 100.18 propid 2015A-0801 seeing 1.45 MJD 57896.340 object 32623 
    90prime/CP/V2.3/CP20170522/ksb_170523_080943_ooi_g_ls9.fits.fz
  ksb_160712_052408_ooi_r_ls9-CCD2 r expnum 75810061 exptime 233.3 propid BASS seeing 1.73 MJD 57581.225 object 31542 
    90prime/CP/V2.3/CP20160711/ksb_160712_052408_ooi_r_ls9.fits.fz
  ksb_170327_113133_ooi_g_ls9-CCD2 g expnum 78390275 exptime 91.542 propid BASS seeing 1.41 MJD 57839.480 object 31542 
    90prime/CP/V2.3/CP20170326/ksb_170327_113133_ooi_g_ls9.fits.fz
  ksb_170501_090154_ooi_r_ls9-CCD3 r expnum 78740200 exptime 62.028 propid BASS seeing 1.12 MJD 57874.376 object 31553 
    90prime/CP/V2.3/CP20170430/ksb_170501_090154_ooi_r_ls9.fits.fz
  ksb_170523_081418_ooi_g_ls9-CCD3 g expnum 78960128 exptime 103.558 propid 2015A-0801 seeing 1.52 MJD 57896.343 object 31553 
    90prime/CP/V2.3/CP20170522/ksb_170523_081418_ooi_g_ls9.fits.fz
  ksb_160531_080046_ooi_r_ls9-CCD1 r expnum 75390152 exptime 65.0 propid BASS seeing 1.16 MJD 57539.334 object 31541 
    90prime/CP/V2.3/CP20160530/ksb_160531_080046_ooi_r_ls9.fits.fz
  ksb_170602_101151_ooi_g_ls9-CCD1 g expnum 79060155 exptime 124.85 propid 2015A-0801 seeing 1.53 MJD 57906.425 object 31541 
    90prime/CP/V2.3/CP20170601/ksb_170602_101151_ooi_g_ls9.fits.fz
Faked up EXPNUM 78960126
Faked up EXPNUM 78960126
Faked up EXPNUM 78960126
90prime get_fwhm: self.fwhm = 3.19
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3966 4009 and amp range 2016 4031 and image y 15 67 and amp range 0 2047
Range in image: x 0 44 , y 0 53 (with image size 44 x 53)
dzp 0.016679538 -> scaling image by 1.0154810312115594
Faked up EXPNUM 78740200
Faked up EXPNUM 78740200
Faked up EXPNUM 78740200
90prime get_fwhm: self.fwhm = 2.47
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 186 352 and amp range 0 2015 and image y 2968 3132 and amp range 2048 4095
Range in image: x 0 167 , y 0 165 (with image size 167 x 165)
dzp -0.07725406 -> scaling image by 0.9313188157188347
Faked up EXPNUM 75390152
Faked up EXPNUM 75390152
Faked up EXPNUM 75390152
90prime get_fwhm: self.fwhm = 2.57
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2121 2287 and amp range 2016 4031 and image y 1444 1610 and amp range 0 2047
Range in image: x 0 167 , y 0 167 (with image size 167 x 167)
dzp -0.0041296566 -> scaling image by 0.9962036700684744
Faked up EXPNUM 78930197
Faked up EXPNUM 78930197
Faked up EXPNUM 78930197
90prime get_fwhm: self.fwhm = 4.49
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3959 4009 and amp range 2016 4031 and image y 15 79 and amp range 0 2047
Range in image: x 0 51 , y 0 65 (with image size 51 x 65)
dzp 0.02499258 -> scaling image by 1.0232859986636875
Faked up EXPNUM 78390275
Faked up EXPNUM 78390275
Faked up EXPNUM 78390275
90prime get_fwhm: self.fwhm = 3.11
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 15 134 and amp range 0 2015 and image y 3425 3589 and amp range 2048 4095
Range in image: x 0 120 , y 0 165 (with image size 120 x 165)
dzp 0.051889952 -> scaling image by 1.0489528829333021
Faked up EXPNUM 79060155
Faked up EXPNUM 79060155
Faked up EXPNUM 79060155
90prime get_fwhm: self.fwhm = 3.37
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2135 2301 and amp range 2016 4031 and image y 1453 1618 and amp range 0 2047
Range in image: x 0 167 , y 0 166 (with image size 167 x 166)
dzp -0.011406615 -> scaling image by 0.9895491130016149
mosaic get_fwhm: self.fwhm = 3.61
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3609 3891 and amp range 2006 4053 and image y 897 1180 and amp range 0 2039
Range in image: x 0 283 , y 0 284 (with image size 283 x 284)
dzp -0.00064781494 -> scaling image by 0.9994035183523652
mosaic get_fwhm: self.fwhm = 4.09
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 841 1131 and amp range 0 2005 and image y 1016 1307 and amp range 0 2039
Range in image: x 0 291 , y 0 292 (with image size 291 x 292)
dzp 0.007397243 -> scaling image by 1.006836374469654
mosaic get_fwhm: self.fwhm = 4.59
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2054 2345 and amp range 2048 4095 and image y 3243 3531 and amp range 2022 4060
Range in image: x 0 292 , y 0 289 (with image size 292 x 289)
dzp -0.02499645 -> scaling image by 0.9772404160512723
Faked up EXPNUM 75810061
Faked up EXPNUM 75810061
Faked up EXPNUM 75810061
90prime get_fwhm: self.fwhm = 3.81
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 15 121 and amp range 0 2015 and image y 3423 3587 and amp range 2048 4095
Range in image: x 0 107 , y 0 165 (with image size 107 x 165)
dzp 0.016961161 -> scaling image by 1.0157444649385616
Faked up EXPNUM 78960128
Faked up EXPNUM 78960128
Faked up EXPNUM 78960128
90prime get_fwhm: self.fwhm = 3.35
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 186 352 and amp range 0 2015 and image y 2963 3127 and amp range 2048 4095
Range in image: x 0 167 , y 0 165 (with image size 167 x 165)
dzp -0.045992933 -> scaling image by 0.958523636456738
Stage tims : Wall: 140.82 s, CPU: 0.96 s, VmPeak: 951 MB, VmSize: 951 MB, VmRSS: 174 MB, VmData: 207 MB, maxrss: 0.177928 MB
Stage tims finished: 2020-08-07T09:35:37.911333
Resources for stage tims : 5.713 s worker CPU, 845.251 s worker Wall, Wall: 140.821 s, Cores in use: 0.05, Total efficiency (on 16 cores): 0.3 %
Stage tims finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-tims.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-tims.p
Running stage refs
Running stage refs at 2020-08-07T09:35:37.977382
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-01171.fits
Matched 13 large galaxies to Gaia stars.
Matched 0 large galaxies to Tycho-2 stars.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/reference-custom-253043p54233.fits
Stage refs : Wall: 3.98 s, CPU: 2.33 s, VmPeak: 982 MB, VmSize: 975 MB, VmRSS: 199 MB, VmData: 231 MB, maxrss: 0.203392 MB
Stage refs finished: 2020-08-07T09:35:41.955394
Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 3.980 s, Cores in use: 0.59, Total efficiency (on 16 cores): 3.7 %
Stage refs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-refs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-refs.p
Running stage outliers
Running stage outliers at 2020-08-07T09:35:41.993467
Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/outlier-mask-custom-253043p54233.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/outliers-pre-custom-253043p54233.jpg
Image ksb_170523_081418_ooi_g_ls9-CCD3 g : masking 89 positive outlier pixels and 0 negative outlier pixels
Image ksb_170501_090154_ooi_r_ls9-CCD3 r : masking 125 positive outlier pixels and 0 negative outlier pixels
Image ksb_160531_080046_ooi_r_ls9-CCD1 r : masking 373 positive outlier pixels and 0 negative outlier pixels
Image k4m_160615_064734_ooi_zd_ls9-CCD3 z : masking 1775 positive outlier pixels and 0 negative outlier pixels
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/outlier-mask-custom-253043p54233.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/outliers-post-custom-253043p54233.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/outliers-masked-pos-custom-253043p54233.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/outliers-masked-neg-custom-253043p54233.jpg
Stage outliers : Wall: 1.04 s, CPU: 0.72 s, VmPeak: 985 MB, VmSize: 985 MB, VmRSS: 215 MB, VmData: 244 MB, maxrss: 0.2197 MB
Stage outliers finished: 2020-08-07T09:35:43.033313
Resources for stage outliers : 0.763 s worker CPU, 0.834 s worker Wall, Wall: 1.041 s, Cores in use: 1.42, Total efficiency (on 16 cores): 8.9 %
Stage outliers finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-outliers.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-outliers.p
Running stage halos
Running stage halos at 2020-08-07T09:35:43.084480
Stage halos : Wall: 0.29 s, CPU: 0.28 s, VmPeak: 985 MB, VmSize: 985 MB, VmRSS: 215 MB, VmData: 244 MB, maxrss: 0.220116 MB
Stage halos finished: 2020-08-07T09:35:43.369907
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.287 s, Cores in use: 0.97, Total efficiency (on 16 cores): 6.1 %
Stage halos finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-halos.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-halos.p
Running stage fit_on_coadds
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Running stage fit_on_coadds at 2020-08-07T09:35:43.445731
PSF sigmas in coadd pixels: ['2.35', '2.29', '2.46', '2.48']
Using average PSF sigma 2.395566358689102
Gaussian PSF norm 0.11775703509555277 vs pixelized 0.09646213
average tim pixel scale / coadd scale: 1.729534137354211
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-copsf-g.fits.fz
PSF sigmas in coadd pixels: ['3.31', '2.80', '1.82', '1.89']
Using average PSF sigma 2.454527133479732
Gaussian PSF norm 0.11492836560089613 vs pixelized 0.09748004
average tim pixel scale / coadd scale: 1.7294610056489392
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-copsf-r.fits.fz
PSF sigmas in coadd pixels: ['1.52', '1.72', '1.94']
Using average PSF sigma 1.7273379876689077
Gaussian PSF norm 0.16331186703916192 vs pixelized 0.13717054
average tim pixel scale / coadd scale: 0.9908615580390256
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-copsf-z.fits.fz
Stage fit_on_coadds : Wall: 0.24 s, CPU: 0.17 s, VmPeak: 985 MB, VmSize: 985 MB, VmRSS: 217 MB, VmData: 244 MB, maxrss: 0.221792 MB
Stage fit_on_coadds finished: 2020-08-07T09:35:43.681063
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage fit_on_coadds : 0.182 s worker CPU, 0.182 s worker Wall, Wall: 0.237 s, Cores in use: 1.48, Total efficiency (on 16 cores): 9.2 %
Stage fit_on_coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-fit_on_coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-fit_on_coadds.p
Running stage srcs
Running stage srcs at 2020-08-07T09:35:43.712122
SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])]
Found 13 potential peaks
Of 13 potential peaks: 5 in veto map, 1 cut by saddle test, 0 cut by aper test, 7 kept
SED z : 7 new peaks
Found 18 potential peaks
Of 18 potential peaks: 14 in veto map, 3 cut by saddle test, 0 cut by aper test, 1 kept
SED r : 1 new peaks
Found 21 potential peaks
Of 21 potential peaks: 17 in veto map, 3 cut by saddle test, 0 cut by aper test, 1 kept
SED g : 1 new peaks
Found 25 potential peaks
Of 25 potential peaks: 20 in veto map, 2 cut by saddle test, 0 cut by aper test, 3 kept
SED Flat : 3 new peaks
Found 25 potential peaks
Of 25 potential peaks: 21 in veto map, 2 cut by saddle test, 0 cut by aper test, 2 kept
SED Red : 2 new peaks
Adding 14 new sources
Median coadd sky for g : 7.9724465e-05
Median coadd sky for r : -8.704088e-05
Median coadd sky for z : 3.242027e-05
Stage srcs : Wall: 0.45 s, CPU: 0.41 s, VmPeak: 985 MB, VmSize: 985 MB, VmRSS: 217 MB, VmData: 244 MB, maxrss: 0.2225 MB
Stage srcs finished: 2020-08-07T09:35:44.165972
Resources for stage srcs : 0.086 s worker CPU, 0.086 s worker Wall, Wall: 0.455 s, Cores in use: 1.10, Total efficiency (on 16 cores): 6.9 %
Stage srcs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-srcs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-srcs.p
Running stage fitblobs
Running stage fitblobs at 2020-08-07T09:35:44.190710
Blob 1 of 15, id: 3, sources: 1, size: 92x158, npix 8954, brick X: 92,184, Y: 58,216, one pixel: 172 58
Blob 2 of 15, id: 10, sources: 3, size: 83x73, npix 3449, brick X: 197,280, Y: 202,275, one pixel: 253 202
Blob 3 of 15, id: 2, sources: 2, size: 62x53, npix 2089, brick X: 65,127, Y: 28,81, one pixel: 88 28
Blob 4 of 15, id: 12, sources: 3, size: 55x54, npix 1786, brick X: 30,85, Y: 227,281, one pixel: 57 227
Blob 5 of 15, id: 9, sources: 2, size: 28x49, npix 958, brick X: 0,28, Y: 177,226, one pixel: 5 177
Blob 6 of 15, id: 6, sources: 1, size: 28x28, npix 549, brick X: 36,64, Y: 134,162, one pixel: 49 134
Blob 7 of 15, id: 14, sources: 1, size: 30x21, npix 452, brick X: 193,223, Y: 260,281, one pixel: 205 260
Blob 8 of 15, id: 4, sources: 1, size: 25x26, npix 423, brick X: 237,262, Y: 69,95, one pixel: 249 69
Blob 9 of 15, id: 0, sources: 2, size: 27x15, npix 289, brick X: 130,157, Y: 0,15, one pixel: 131 0
Blob 10 of 15, id: 5, sources: 1, size: 19x20, npix 236, brick X: 194,213, Y: 74,94, one pixel: 202 74
Blob 11 of 15, id: 11, sources: 1, size: 20x20, npix 219, brick X: 152,172, Y: 204,224, one pixel: 161 204
Blob 12 of 15, id: 1, sources: 1, size: 19x19, npix 217, brick X: 258,277, Y: 20,39, one pixel: 266 20
Blob 13 of 15, id: 8, sources: 1, size: 19x19, npix 208, brick X: 180,199, Y: 172,191, one pixel: 188 172
Blob 14 of 15, id: 13, sources: 1, size: 19x19, npix 199, brick X: 119,138, Y: 232,251, one pixel: 127 232
Blob 15 of 15, id: 7, sources: 1, size: 17x18, npix 162, brick X: 70,87, Y: 149,167, one pixel: 78 149
Blob 15 finished, total: Wall: 0.07 s, CPU: 0.07 s, VmPeak: 804 MB, VmSize: 724 MB, VmRSS: 142 MB, VmData: 182 MB, maxrss: 0.14538 MB
Blob 13 finished, total: Wall: 0.12 s, CPU: 0.08 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 87 MB, VmData: 139 MB, maxrss: 0.088912 MB
Blob 12 finished, total: Wall: 0.13 s, CPU: 0.08 s, VmPeak: 808 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.147296 MB
Blob 14 finished, total: Wall: 0.21 s, CPU: 0.21 s, VmPeak: 809 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.147392 MB
Blob 10 finished, total: Wall: 0.28 s, CPU: 0.28 s, VmPeak: 811 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.147088 MB
Blob 11 finished, total: Wall: 0.33 s, CPU: 0.33 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 93 MB, VmData: 139 MB, maxrss: 0.095632 MB
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.04343, 54.22332), pm (-5.4, -5.8), parallax 1.390 with NanoMaggies: g=21.9, r=20.2, z=19
Reset brightness to NanoMaggies: g=21.8, r=20.4, z=19.3
Blob 9 finished, total: Wall: 0.34 s, CPU: 0.34 s, VmPeak: 808 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.14708 MB
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.06134, 54.23807), pm (26.3, 0.7), parallax 1.862 with NanoMaggies: g=17.6, r=16.7, z=15.6
Reset brightness to NanoMaggies: g=17.5, r=16.3, z=15.6
Blob 5 finished, total: Wall: 0.49 s, CPU: 0.47 s, VmPeak: 804 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.146412 MB
Blob 8 finished, total: Wall: 0.50 s, CPU: 0.50 s, VmPeak: 808 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.146368 MB
Measuring 1 iterative sources
Blob 6 finished, total: Wall: 0.73 s, CPU: 0.73 s, VmPeak: 809 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.146628 MB
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.05403, 54.24420), pm (-4.8, -13.9), parallax 0.167 with NanoMaggies: g=16.2, r=16.1, z=15.3
Reset brightness to NanoMaggies: g=16.4, r=15.8, z=15.6
Blob 4 finished, total: Wall: 0.82 s, CPU: 0.82 s, VmPeak: 804 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.147232 MB
Blob 7 finished, total: Wall: 0.96 s, CPU: 0.96 s, VmPeak: 670 MB, VmSize: 670 MB, VmRSS: 95 MB, VmData: 141 MB, maxrss: 0.097712 MB
kdtree_internal.c:2152:kdtree_build_2_ddd: Data, N, or D is zero
Exception in one_blob: brick custom-253043p54233, iblob 3
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1637, in _bounce_one_blob
    result = one_blob(X)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob
    B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 341, in run
    ras[not_galaxy], decs[not_galaxy], 1./3600.)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec
    X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match
    (kd1,kd2) = _buildtrees(x1, x2)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees
    kd2 = spherematch_c.KdTree(fx2)
SystemError: <class 'spherematch.KdTree'> returned NULL without setting an error
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python/astrometry/util/timingpool.py", line 269, in __call__
    R = self.func(*args, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1637, in _bounce_one_blob
    result = one_blob(X)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob
    B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 341, in run
    ras[not_galaxy], decs[not_galaxy], 1./3600.)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec
    X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match
    (kd1,kd2) = _buildtrees(x1, x2)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees
    kd2 = spherematch_c.KdTree(fx2)
SystemError: <class 'spherematch.KdTree'> returned NULL without setting an error
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3771, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3739, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3295, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3284, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1131, in stage_fitblobs
    r = Riter.next(timeout)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 748, in next
    raise value
SystemError: <class 'spherematch.KdTree'> returned NULL without setting an error
Something went wrong; please check the logfile.
ERROR: galaxy PGC2463611; please check the logfile.
Finished galaxy PGC2463611 in 2.512 minutes.
Started working on galaxy PGC2463611 at Fri Aug  7 12:07:54 2020
Bands touching this brick, g r z
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 253.0437484055481 54.23361969915836 --width 281 --height 281 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 

runbrick.py starting at 2020-08-07T12:07:54.853231
legacypipe git version: DR9.5.11-watershed-4-gcc619972
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '253.0437484055481', '54.23361969915836', '--width', '281', '--height', '281', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 253.0437484055481 54.23361969915836 --width 281 --height 281 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0

Parsed RA,Dec 253.0437484055481 54.23361969915836
Starting process 16850 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB
Starting process 16851 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 16852 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 16853 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 16854 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05954 MB
Starting process 16855 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059552 MB
Starting process 16856 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059552 MB
Starting process 16857 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 16858 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 16859 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 16860 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 16861 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 16862 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 16863 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 16864 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 16865 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 16866 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 16867 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059276 MB
Starting process 16868 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB
Starting process 16869 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB
Starting process 16870 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB
Starting process 16871 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB
Starting process 16872 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB
Starting process 16873 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB
Starting process 16874 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB
Starting process 16875 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Starting process 16876 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Starting process 16877 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059308 MB
Starting process 16878 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB
Starting process 16879 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB
Starting process 16880 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB
Starting process 16881 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059316 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-srcs.p and forcing stage srcs
Runstage fit_on_coadds
Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-fit_on_coadds.p and forcing stage fit_on_coadds
Runstage halos
Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-halos.p and forcing stage halos
Runstage outliers
Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-outliers.p and forcing stage outliers
Runstage refs
Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-refs.p and forcing stage refs
Runstage tims
Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-tims.p and forcing stage tims
Running stage tims
Running stage tims at 2020-08-07T12:07:55.311345
Keeping 11 CCDs:
  k4m_160615_064734_ooi_zd_ls9-CCD3 z expnum 95405 exptime 80.105 propid 2016A-0453 seeing 0.94 MJD 57554.283 object MzLS_666483_z 
    mosaic/CP/V4.3/CP20160614/k4m_160615_064734_ooi_zd_ls9.fits.fz
  k4m_170412_112301_ooi_zd_ls9-CCD1 z expnum 121439 exptime 107.105 propid 2016A-0453 seeing 1.06 MJD 57855.474 object MzLS_667217_z 
    mosaic/CP/V4.3/CP20170411/k4m_170412_112301_ooi_zd_ls9.fits.fz
  k4m_170523_095715_ooi_zd_ls9-CCD4 z expnum 129693 exptime 84.101 propid 2016A-0453 seeing 1.19 MJD 57896.415 object MzLS_665745_z 
    mosaic/CP/V4.3/CP20170522/k4m_170523_095715_ooi_zd_ls9.fits.fz
  ksb_170520_101640_ooi_r_ls9-CCD4 r expnum 78930197 exptime 181.5 propid 2015A-0801 seeing 2.04 MJD 57893.428 object 32623 
    90prime/CP/V2.3/CP20170519/ksb_170520_101640_ooi_r_ls9.fits.fz
  ksb_170523_080943_ooi_g_ls9-CCD4 g expnum 78960126 exptime 100.18 propid 2015A-0801 seeing 1.45 MJD 57896.340 object 32623 
    90prime/CP/V2.3/CP20170522/ksb_170523_080943_ooi_g_ls9.fits.fz
  ksb_160712_052408_ooi_r_ls9-CCD2 r expnum 75810061 exptime 233.3 propid BASS seeing 1.73 MJD 57581.225 object 31542 
    90prime/CP/V2.3/CP20160711/ksb_160712_052408_ooi_r_ls9.fits.fz
  ksb_170327_113133_ooi_g_ls9-CCD2 g expnum 78390275 exptime 91.542 propid BASS seeing 1.41 MJD 57839.480 object 31542 
    90prime/CP/V2.3/CP20170326/ksb_170327_113133_ooi_g_ls9.fits.fz
  ksb_170501_090154_ooi_r_ls9-CCD3 r expnum 78740200 exptime 62.028 propid BASS seeing 1.12 MJD 57874.376 object 31553 
    90prime/CP/V2.3/CP20170430/ksb_170501_090154_ooi_r_ls9.fits.fz
  ksb_170523_081418_ooi_g_ls9-CCD3 g expnum 78960128 exptime 103.558 propid 2015A-0801 seeing 1.52 MJD 57896.343 object 31553 
    90prime/CP/V2.3/CP20170522/ksb_170523_081418_ooi_g_ls9.fits.fz
  ksb_160531_080046_ooi_r_ls9-CCD1 r expnum 75390152 exptime 65.0 propid BASS seeing 1.16 MJD 57539.334 object 31541 
    90prime/CP/V2.3/CP20160530/ksb_160531_080046_ooi_r_ls9.fits.fz
  ksb_170602_101151_ooi_g_ls9-CCD1 g expnum 79060155 exptime 124.85 propid 2015A-0801 seeing 1.53 MJD 57906.425 object 31541 
    90prime/CP/V2.3/CP20170601/ksb_170602_101151_ooi_g_ls9.fits.fz
Faked up EXPNUM 78930197
Faked up EXPNUM 78930197
Faked up EXPNUM 78930197
90prime get_fwhm: self.fwhm = 4.49
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3959 4009 and amp range 2016 4031 and image y 15 79 and amp range 0 2047
Range in image: x 0 51 , y 0 65 (with image size 51 x 65)
dzp 0.02499258 -> scaling image by 1.0232859986636875
mosaic get_fwhm: self.fwhm = 3.61
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3609 3891 and amp range 2006 4053 and image y 897 1180 and amp range 0 2039
Range in image: x 0 283 , y 0 284 (with image size 283 x 284)
dzp -0.00064781494 -> scaling image by 0.9994035183523652
Faked up EXPNUM 78960128
Faked up EXPNUM 78960128
Faked up EXPNUM 78960128
90prime get_fwhm: self.fwhm = 3.35
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 186 352 and amp range 0 2015 and image y 2963 3127 and amp range 2048 4095
Range in image: x 0 167 , y 0 165 (with image size 167 x 165)
dzp -0.045992933 -> scaling image by 0.958523636456738
Faked up EXPNUM 78960126
Faked up EXPNUM 78960126
Faked up EXPNUM 78960126
90prime get_fwhm: self.fwhm = 3.19
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3966 4009 and amp range 2016 4031 and image y 15 67 and amp range 0 2047
Range in image: x 0 44 , y 0 53 (with image size 44 x 53)
dzp 0.016679538 -> scaling image by 1.0154810312115594
Faked up EXPNUM 75390152
Faked up EXPNUM 75390152
Faked up EXPNUM 75390152
90prime get_fwhm: self.fwhm = 2.57
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2121 2287 and amp range 2016 4031 and image y 1444 1610 and amp range 0 2047
Range in image: x 0 167 , y 0 167 (with image size 167 x 167)
dzp -0.0041296566 -> scaling image by 0.9962036700684744
Faked up EXPNUM 78740200
Faked up EXPNUM 78740200
Faked up EXPNUM 78740200
90prime get_fwhm: self.fwhm = 2.47
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 186 352 and amp range 0 2015 and image y 2968 3132 and amp range 2048 4095
Range in image: x 0 167 , y 0 165 (with image size 167 x 165)
dzp -0.07725406 -> scaling image by 0.9313188157188347
Faked up EXPNUM 79060155
Faked up EXPNUM 79060155
Faked up EXPNUM 79060155
90prime get_fwhm: self.fwhm = 3.37
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2135 2301 and amp range 2016 4031 and image y 1453 1618 and amp range 0 2047
Range in image: x 0 167 , y 0 166 (with image size 167 x 166)
dzp -0.011406615 -> scaling image by 0.9895491130016149
mosaic get_fwhm: self.fwhm = 4.09
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 841 1131 and amp range 0 2005 and image y 1016 1307 and amp range 0 2039
Range in image: x 0 291 , y 0 292 (with image size 291 x 292)
dzp 0.007397243 -> scaling image by 1.006836374469654
mosaic get_fwhm: self.fwhm = 4.59
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2054 2345 and amp range 2048 4095 and image y 3243 3531 and amp range 2022 4060
Range in image: x 0 292 , y 0 289 (with image size 292 x 289)
dzp -0.02499645 -> scaling image by 0.9772404160512723
Faked up EXPNUM 78390275
Faked up EXPNUM 78390275
Faked up EXPNUM 78390275
90prime get_fwhm: self.fwhm = 3.11
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 15 134 and amp range 0 2015 and image y 3425 3589 and amp range 2048 4095
Range in image: x 0 120 , y 0 165 (with image size 120 x 165)
dzp 0.051889952 -> scaling image by 1.0489528829333021
Faked up EXPNUM 75810061
Faked up EXPNUM 75810061
Faked up EXPNUM 75810061
90prime get_fwhm: self.fwhm = 3.81
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 15 121 and amp range 0 2015 and image y 3423 3587 and amp range 2048 4095
Range in image: x 0 107 , y 0 165 (with image size 107 x 165)
dzp 0.016961161 -> scaling image by 1.0157444649385616
Stage tims : Wall: 2.43 s, CPU: 0.86 s, VmPeak: 951 MB, VmSize: 951 MB, VmRSS: 174 MB, VmData: 208 MB, maxrss: 0.177988 MB
Stage tims finished: 2020-08-07T12:07:57.738606
Resources for stage tims : 5.710 s worker CPU, 13.245 s worker Wall, Wall: 2.430 s, Cores in use: 2.70, Total efficiency (on 32 cores): 8.5 %
Stage tims finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-tims.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-tims.p
Running stage refs
Running stage refs at 2020-08-07T12:07:57.778744
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-01171.fits
Matched 13 large galaxies to Gaia stars.
Matched 0 large galaxies to Tycho-2 stars.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/reference-custom-253043p54233.fits
Stage refs : Wall: 2.36 s, CPU: 2.14 s, VmPeak: 982 MB, VmSize: 975 MB, VmRSS: 199 MB, VmData: 231 MB, maxrss: 0.203336 MB
Stage refs finished: 2020-08-07T12:08:00.141847
Resources for stage refs : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 2.365 s, Cores in use: 0.91, Total efficiency (on 32 cores): 2.8 %
Stage refs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-refs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-refs.p
Running stage outliers
Running stage outliers at 2020-08-07T12:08:00.177961
Stage outliers : Wall: 0.29 s, CPU: 0.26 s, VmPeak: 982 MB, VmSize: 975 MB, VmRSS: 200 MB, VmData: 232 MB, maxrss: 0.204316 MB
Stage outliers finished: 2020-08-07T12:08:00.471556
Resources for stage outliers : 0.000 s worker CPU, 0.001 s worker Wall, Wall: 0.296 s, Cores in use: 0.90, Total efficiency (on 32 cores): 2.8 %
Stage outliers finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-outliers.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-outliers.p
Running stage halos
Running stage halos at 2020-08-07T12:08:00.496252
Stage halos : Wall: 0.21 s, CPU: 0.20 s, VmPeak: 982 MB, VmSize: 975 MB, VmRSS: 203 MB, VmData: 236 MB, maxrss: 0.207672 MB
Stage halos finished: 2020-08-07T12:08:00.705713
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.211 s, Cores in use: 0.95, Total efficiency (on 32 cores): 3.0 %
Stage halos finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-halos.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-halos.p
Running stage fit_on_coadds
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Running stage fit_on_coadds at 2020-08-07T12:08:00.740328
PSF sigmas in coadd pixels: ['2.35', '2.29', '2.46', '2.48']
Using average PSF sigma 2.395566358689102
Gaussian PSF norm 0.11775703509555277 vs pixelized 0.09646213
average tim pixel scale / coadd scale: 1.729534137354211
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-copsf-g.fits.fz
PSF sigmas in coadd pixels: ['3.31', '2.80', '1.82', '1.89']
Using average PSF sigma 2.454527133479732
Gaussian PSF norm 0.11492836560089613 vs pixelized 0.09748004
average tim pixel scale / coadd scale: 1.7294610056489392
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-copsf-r.fits.fz
PSF sigmas in coadd pixels: ['1.52', '1.72', '1.94']
Using average PSF sigma 1.7273379876689077
Gaussian PSF norm 0.16331186703916192 vs pixelized 0.13717054
average tim pixel scale / coadd scale: 0.9908615580390256
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-copsf-z.fits.fz
Stage fit_on_coadds : Wall: 0.25 s, CPU: 0.16 s, VmPeak: 982 MB, VmSize: 980 MB, VmRSS: 216 MB, VmData: 245 MB, maxrss: 0.22076 MB
Stage fit_on_coadds finished: 2020-08-07T12:08:00.988252
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage fit_on_coadds : 0.206 s worker CPU, 0.300 s worker Wall, Wall: 0.250 s, Cores in use: 1.49, Total efficiency (on 32 cores): 4.7 %
Stage fit_on_coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-fit_on_coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-fit_on_coadds.p
Running stage srcs
Warning: Stellar halo subtraction is only implemented for DECam
Running stage srcs at 2020-08-07T12:08:01.001235
SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])]
Found 13 potential peaks
Of 13 potential peaks: 5 in veto map, 1 cut by saddle test, 0 cut by aper test, 7 kept
SED z : 7 new peaks
Found 18 potential peaks
Of 18 potential peaks: 14 in veto map, 3 cut by saddle test, 0 cut by aper test, 1 kept
SED r : 1 new peaks
Found 21 potential peaks
Of 21 potential peaks: 17 in veto map, 3 cut by saddle test, 0 cut by aper test, 1 kept
SED g : 1 new peaks
Found 25 potential peaks
Of 25 potential peaks: 20 in veto map, 2 cut by saddle test, 0 cut by aper test, 3 kept
SED Flat : 3 new peaks
Found 25 potential peaks
Of 25 potential peaks: 21 in veto map, 2 cut by saddle test, 0 cut by aper test, 2 kept
SED Red : 2 new peaks
Adding 14 new sources
Median coadd sky for g : 7.9724465e-05
Median coadd sky for r : -8.704093e-05
Median coadd sky for z : 3.242027e-05
Stage srcs : Wall: 0.46 s, CPU: 0.40 s, VmPeak: 982 MB, VmSize: 980 MB, VmRSS: 216 MB, VmData: 245 MB, maxrss: 0.22144 MB
Stage srcs finished: 2020-08-07T12:08:01.457774
Resources for stage srcs : 0.102 s worker CPU, 0.107 s worker Wall, Wall: 0.459 s, Cores in use: 1.09, Total efficiency (on 32 cores): 3.4 %
Stage srcs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-srcs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-srcs.p
Running stage fitblobs
Running stage fitblobs at 2020-08-07T12:08:01.471442
Blob 1 of 15, id: 3, sources: 1, size: 92x158, npix 8954, brick X: 92,184, Y: 58,216, one pixel: 172 58
Blob 2 of 15, id: 10, sources: 3, size: 83x73, npix 3449, brick X: 197,280, Y: 202,275, one pixel: 253 202
Blob 3 of 15, id: 2, sources: 2, size: 62x53, npix 2089, brick X: 65,127, Y: 28,81, one pixel: 88 28
Blob 4 of 15, id: 12, sources: 3, size: 55x54, npix 1786, brick X: 30,85, Y: 227,281, one pixel: 57 227
Blob 5 of 15, id: 9, sources: 2, size: 28x49, npix 958, brick X: 0,28, Y: 177,226, one pixel: 5 177
Blob 6 of 15, id: 6, sources: 1, size: 28x28, npix 549, brick X: 36,64, Y: 134,162, one pixel: 49 134
Blob 7 of 15, id: 14, sources: 1, size: 30x21, npix 452, brick X: 193,223, Y: 260,281, one pixel: 205 260
Blob 8 of 15, id: 4, sources: 1, size: 25x26, npix 423, brick X: 237,262, Y: 69,95, one pixel: 249 69
Blob 9 of 15, id: 0, sources: 2, size: 27x15, npix 289, brick X: 130,157, Y: 0,15, one pixel: 131 0
Blob 10 of 15, id: 5, sources: 1, size: 19x20, npix 236, brick X: 194,213, Y: 74,94, one pixel: 202 74
Blob 11 of 15, id: 11, sources: 1, size: 20x20, npix 219, brick X: 152,172, Y: 204,224, one pixel: 161 204
Blob 12 of 15, id: 1, sources: 1, size: 19x19, npix 217, brick X: 258,277, Y: 20,39, one pixel: 266 20
Blob 13 of 15, id: 8, sources: 1, size: 19x19, npix 208, brick X: 180,199, Y: 172,191, one pixel: 188 172
Blob 14 of 15, id: 13, sources: 1, size: 19x19, npix 199, brick X: 119,138, Y: 232,251, one pixel: 127 232
Blob 15 of 15, id: 7, sources: 1, size: 17x18, npix 162, brick X: 70,87, Y: 149,167, one pixel: 78 149
Blob 13 finished, total: Wall: 0.08 s, CPU: 0.06 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 82 MB, VmData: 137 MB, maxrss: 0.08406 MB
Blob 15 finished, total: Wall: 0.09 s, CPU: 0.07 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 83 MB, VmData: 137 MB, maxrss: 0.084824 MB
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.04343, 54.22332), pm (-5.4, -5.8), parallax 1.390 with NanoMaggies: g=24.4, r=24.4, z=20.5
Reset brightness to NanoMaggies: g=21.8, r=20.4, z=19.3
Blob 12 finished, total: Wall: 0.11 s, CPU: 0.09 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 82 MB, VmData: 137 MB, maxrss: 0.084088 MB
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.06134, 54.23807), pm (26.3, 0.7), parallax 1.862 with NanoMaggies: g=18.1, r=17.3, z=15.7
Reset brightness to NanoMaggies: g=17.5, r=16.3, z=15.6
Blob 9 finished, total: Wall: 0.14 s, CPU: 0.12 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 83 MB, VmData: 137 MB, maxrss: 0.084684 MB
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.05403, 54.24420), pm (-4.8, -13.9), parallax 0.167 with NanoMaggies: g=18.5, r=(flux -14.1), z=17.7
Reset brightness to NanoMaggies: g=16.4, r=15.8, z=15.6
Blob 10 finished, total: Wall: 0.17 s, CPU: 0.17 s, VmPeak: 808 MB, VmSize: 725 MB, VmRSS: 142 MB, VmData: 183 MB, maxrss: 0.145088 MB
Blob 11 finished, total: Wall: 0.18 s, CPU: 0.16 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.089684 MB
Blob 14 finished, total: Wall: 0.27 s, CPU: 0.25 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.08962 MB
Blob 8 finished, total: Wall: 0.42 s, CPU: 0.39 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.089868 MB
Measuring 1 iterative sources
Blob 4 finished, total: Wall: 0.57 s, CPU: 0.54 s, VmPeak: 667 MB, VmSize: 666 MB, VmRSS: 89 MB, VmData: 137 MB, maxrss: 0.091708 MB
Blob 6 finished, total: Wall: 0.65 s, CPU: 0.62 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.08998 MB
kdtree_internal.c:2152:kdtree_build_2_ddd: Data, N, or D is zero
Blob 7 finished, total: Wall: 0.86 s, CPU: 0.86 s, VmPeak: 807 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.145952 MB
Exception in one_blob: brick custom-253043p54233, iblob 3
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1641, in _bounce_one_blob
    result = one_blob(X)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob
    B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 345, in run
    ras[not_galaxy], decs[not_galaxy], 1./3600.)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec
    X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match
    (kd1,kd2) = _buildtrees(x1, x2)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees
    kd2 = spherematch_c.KdTree(fx2)
SystemError: <class 'spherematch.KdTree'> returned NULL without setting an error
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python/astrometry/util/timingpool.py", line 269, in __call__
    R = self.func(*args, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1641, in _bounce_one_blob
    result = one_blob(X)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob
    B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 345, in run
    ras[not_galaxy], decs[not_galaxy], 1./3600.)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec
    X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match
    (kd1,kd2) = _buildtrees(x1, x2)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees
    kd2 = spherematch_c.KdTree(fx2)
SystemError: <class 'spherematch.KdTree'> returned NULL without setting an error
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3775, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3743, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3299, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3288, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1135, in stage_fitblobs
    r = Riter.next(timeout)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 748, in next
    raise value
SystemError: <class 'spherematch.KdTree'> returned NULL without setting an error
Something went wrong; please check the logfile.
ERROR: galaxy PGC2463611; please check the logfile.
Finished galaxy PGC2463611 in 0.158 minutes.
Started working on galaxy PGC2463611 at Fri Aug  7 12:32:32 2020
Bands touching this brick, g r z
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 253.0437484055481 54.23361969915836 --width 281 --height 281 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 

runbrick.py starting at 2020-08-07T12:32:32.927277
legacypipe git version: DR9.5.11-watershed-4-gcc619972
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '253.0437484055481', '54.23361969915836', '--width', '281', '--height', '281', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 253.0437484055481 54.23361969915836 --width 281 --height 281 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0

Parsed RA,Dec 253.0437484055481 54.23361969915836
Starting process 16342 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB
Starting process 16343 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 16344 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 16345 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 16346 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05954 MB
Starting process 16347 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059552 MB
Starting process 16348 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059552 MB
Starting process 16349 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 16350 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 16351 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 16352 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 16353 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 16354 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 16355 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 16356 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 16357 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 16358 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 16359 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059276 MB
Starting process 16360 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB
Starting process 16361 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB
Starting process 16362 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB
Starting process 16363 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB
Starting process 16364 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB
Starting process 16365 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB
Starting process 16366 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Starting process 16367 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Starting process 16369 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059308 MB
Starting process 16370 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB
Starting process 16368 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Starting process 16371 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB
Starting process 16372 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB
Starting process 16373 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059316 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage fit_on_coadds
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-08-07T12:32:33.342288
Keeping 11 CCDs:
  k4m_160615_064734_ooi_zd_ls9-CCD3 z expnum 95405 exptime 80.105 propid 2016A-0453 seeing 0.94 MJD 57554.283 object MzLS_666483_z 
    mosaic/CP/V4.3/CP20160614/k4m_160615_064734_ooi_zd_ls9.fits.fz
  k4m_170412_112301_ooi_zd_ls9-CCD1 z expnum 121439 exptime 107.105 propid 2016A-0453 seeing 1.06 MJD 57855.474 object MzLS_667217_z 
    mosaic/CP/V4.3/CP20170411/k4m_170412_112301_ooi_zd_ls9.fits.fz
  k4m_170523_095715_ooi_zd_ls9-CCD4 z expnum 129693 exptime 84.101 propid 2016A-0453 seeing 1.19 MJD 57896.415 object MzLS_665745_z 
    mosaic/CP/V4.3/CP20170522/k4m_170523_095715_ooi_zd_ls9.fits.fz
  ksb_170520_101640_ooi_r_ls9-CCD4 r expnum 78930197 exptime 181.5 propid 2015A-0801 seeing 2.04 MJD 57893.428 object 32623 
    90prime/CP/V2.3/CP20170519/ksb_170520_101640_ooi_r_ls9.fits.fz
  ksb_170523_080943_ooi_g_ls9-CCD4 g expnum 78960126 exptime 100.18 propid 2015A-0801 seeing 1.45 MJD 57896.340 object 32623 
    90prime/CP/V2.3/CP20170522/ksb_170523_080943_ooi_g_ls9.fits.fz
  ksb_160712_052408_ooi_r_ls9-CCD2 r expnum 75810061 exptime 233.3 propid BASS seeing 1.73 MJD 57581.225 object 31542 
    90prime/CP/V2.3/CP20160711/ksb_160712_052408_ooi_r_ls9.fits.fz
  ksb_170327_113133_ooi_g_ls9-CCD2 g expnum 78390275 exptime 91.542 propid BASS seeing 1.41 MJD 57839.480 object 31542 
    90prime/CP/V2.3/CP20170326/ksb_170327_113133_ooi_g_ls9.fits.fz
  ksb_170501_090154_ooi_r_ls9-CCD3 r expnum 78740200 exptime 62.028 propid BASS seeing 1.12 MJD 57874.376 object 31553 
    90prime/CP/V2.3/CP20170430/ksb_170501_090154_ooi_r_ls9.fits.fz
  ksb_170523_081418_ooi_g_ls9-CCD3 g expnum 78960128 exptime 103.558 propid 2015A-0801 seeing 1.52 MJD 57896.343 object 31553 
    90prime/CP/V2.3/CP20170522/ksb_170523_081418_ooi_g_ls9.fits.fz
  ksb_160531_080046_ooi_r_ls9-CCD1 r expnum 75390152 exptime 65.0 propid BASS seeing 1.16 MJD 57539.334 object 31541 
    90prime/CP/V2.3/CP20160530/ksb_160531_080046_ooi_r_ls9.fits.fz
  ksb_170602_101151_ooi_g_ls9-CCD1 g expnum 79060155 exptime 124.85 propid 2015A-0801 seeing 1.53 MJD 57906.425 object 31541 
    90prime/CP/V2.3/CP20170601/ksb_170602_101151_ooi_g_ls9.fits.fz
Faked up EXPNUM 75390152
Faked up EXPNUM 75390152
Faked up EXPNUM 75390152
90prime get_fwhm: self.fwhm = 2.57
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2121 2287 and amp range 2016 4031 and image y 1444 1610 and amp range 0 2047
Range in image: x 0 167 , y 0 167 (with image size 167 x 167)
dzp -0.0041296566 -> scaling image by 0.9962036700684744
mosaic get_fwhm: self.fwhm = 3.61
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3609 3891 and amp range 2006 4053 and image y 897 1180 and amp range 0 2039
Range in image: x 0 283 , y 0 284 (with image size 283 x 284)
dzp -0.00064781494 -> scaling image by 0.9994035183523652
Faked up EXPNUM 78390275
Faked up EXPNUM 78390275
Faked up EXPNUM 78390275
90prime get_fwhm: self.fwhm = 3.11
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 15 134 and amp range 0 2015 and image y 3425 3589 and amp range 2048 4095
Range in image: x 0 120 , y 0 165 (with image size 120 x 165)
dzp 0.051889952 -> scaling image by 1.0489528829333021
Faked up EXPNUM 78960126
Faked up EXPNUM 78960126
Faked up EXPNUM 78960126
90prime get_fwhm: self.fwhm = 3.19
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3966 4009 and amp range 2016 4031 and image y 15 67 and amp range 0 2047
Faked up EXPNUM 78740200
Faked up EXPNUM 78740200
Faked up EXPNUM 78740200
90prime get_fwhm: self.fwhm = 2.47
Found 4 relevant rows in amp-corrections file.
Range in image: x 0 44 , y 0 53 (with image size 44 x 53)
dzp 0.016679538 -> scaling image by 1.0154810312115594
Found overlap: image x 186 352 and amp range 0 2015 and image y 2968 3132 and amp range 2048 4095
Range in image: x 0 167 , y 0 165 (with image size 167 x 165)
dzp -0.07725406 -> scaling image by 0.9313188157188347
Faked up EXPNUM 79060155
Faked up EXPNUM 79060155
Faked up EXPNUM 79060155
90prime get_fwhm: self.fwhm = 3.37
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2135 2301 and amp range 2016 4031 and image y 1453 1618 and amp range 0 2047
Range in image: x 0 167 , y 0 166 (with image size 167 x 166)
dzp -0.011406615 -> scaling image by 0.9895491130016149
mosaic get_fwhm: self.fwhm = 4.09
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 841 1131 and amp range 0 2005 and image y 1016 1307 and amp range 0 2039
Range in image: x 0 291 , y 0 292 (with image size 291 x 292)
dzp 0.007397243 -> scaling image by 1.006836374469654
Faked up EXPNUM 75810061
Faked up EXPNUM 75810061
Faked up EXPNUM 75810061
90prime get_fwhm: self.fwhm = 3.81
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 15 121 and amp range 0 2015 and image y 3423 3587 and amp range 2048 4095
Range in image: x 0 107 , y 0 165 (with image size 107 x 165)
dzp 0.016961161 -> scaling image by 1.0157444649385616
Faked up EXPNUM 78960128
Faked up EXPNUM 78960128
Faked up EXPNUM 78960128
90prime get_fwhm: self.fwhm = 3.35
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 186 352 and amp range 0 2015 and image y 2963 3127 and amp range 2048 4095
Range in image: x 0 167 , y 0 165 (with image size 167 x 165)
dzp -0.045992933 -> scaling image by 0.958523636456738
mosaic get_fwhm: self.fwhm = 4.59
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2054 2345 and amp range 2048 4095 and image y 3243 3531 and amp range 2022 4060
Range in image: x 0 292 , y 0 289 (with image size 292 x 289)
dzp -0.02499645 -> scaling image by 0.9772404160512723
Faked up EXPNUM 78930197
Faked up EXPNUM 78930197
Faked up EXPNUM 78930197
90prime get_fwhm: self.fwhm = 4.49
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3959 4009 and amp range 2016 4031 and image y 15 79 and amp range 0 2047
Range in image: x 0 51 , y 0 65 (with image size 51 x 65)
dzp 0.02499258 -> scaling image by 1.0232859986636875
Stage tims : Wall: 2.30 s, CPU: 0.80 s, VmPeak: 951 MB, VmSize: 951 MB, VmRSS: 174 MB, VmData: 208 MB, maxrss: 0.178388 MB
Stage tims finished: 2020-08-07T12:32:35.638171
Resources for stage tims : 5.700 s worker CPU, 13.062 s worker Wall, Wall: 2.298 s, Cores in use: 2.83, Total efficiency (on 32 cores): 8.8 %
Stage tims finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-tims.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-tims.p
Running stage refs
Running stage refs at 2020-08-07T12:32:35.660413
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-01171.fits
Matched 13 large galaxies to Gaia stars.
Matched 0 large galaxies to Tycho-2 stars.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/reference-custom-253043p54233.fits
Stage refs : Wall: 2.32 s, CPU: 2.12 s, VmPeak: 982 MB, VmSize: 975 MB, VmRSS: 199 MB, VmData: 232 MB, maxrss: 0.203932 MB
Stage refs finished: 2020-08-07T12:32:37.982605
Resources for stage refs : 0.000 s worker CPU, 0.001 s worker Wall, Wall: 2.324 s, Cores in use: 0.91, Total efficiency (on 32 cores): 2.9 %
Stage refs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-refs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-refs.p
Running stage outliers
Running stage outliers at 2020-08-07T12:32:38.008223
Stage outliers : Wall: 0.26 s, CPU: 0.23 s, VmPeak: 982 MB, VmSize: 975 MB, VmRSS: 200 MB, VmData: 232 MB, maxrss: 0.20452 MB
Stage outliers finished: 2020-08-07T12:32:38.264322
Resources for stage outliers : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 0.258 s, Cores in use: 0.91, Total efficiency (on 32 cores): 2.8 %
Stage outliers finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-outliers.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-outliers.p
Running stage halos
Running stage halos at 2020-08-07T12:32:38.296560
Stage halos : Wall: 0.20 s, CPU: 0.19 s, VmPeak: 982 MB, VmSize: 975 MB, VmRSS: 203 MB, VmData: 236 MB, maxrss: 0.207808 MB
Stage halos finished: 2020-08-07T12:32:38.494966
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.200 s, Cores in use: 0.95, Total efficiency (on 32 cores): 3.0 %
Stage halos finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-halos.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-halos.p
Running stage fit_on_coadds
Warning: Stellar halo subtraction is only implemented for DECam
Running stage fit_on_coadds at 2020-08-07T12:32:38.564518
PSF sigmas in coadd pixels: ['2.35', '2.29', '2.46', '2.48']
Using average PSF sigma 2.395566358689102
Gaussian PSF norm 0.11775703509555277 vs pixelized 0.09646213
average tim pixel scale / coadd scale: 1.729534137354211
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-copsf-g.fits.fz
PSF sigmas in coadd pixels: ['3.31', '2.80', '1.82', '1.89']
Using average PSF sigma 2.454527133479732
Gaussian PSF norm 0.11492836560089613 vs pixelized 0.09748004
average tim pixel scale / coadd scale: 1.7294610056489392
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-copsf-r.fits.fz
PSF sigmas in coadd pixels: ['1.52', '1.72', '1.94']
Using average PSF sigma 1.7273379876689077
Gaussian PSF norm 0.16331186703916192 vs pixelized 0.13717054
average tim pixel scale / coadd scale: 0.9908615580390256
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-copsf-z.fits.fz
Stage fit_on_coadds : Wall: 0.20 s, CPU: 0.17 s, VmPeak: 982 MB, VmSize: 980 MB, VmRSS: 215 MB, VmData: 244 MB, maxrss: 0.219744 MB
Stage fit_on_coadds finished: 2020-08-07T12:32:38.766808
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage fit_on_coadds : 0.189 s worker CPU, 0.235 s worker Wall, Wall: 0.204 s, Cores in use: 1.77, Total efficiency (on 32 cores): 5.5 %
Stage fit_on_coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-fit_on_coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-fit_on_coadds.p
Running stage srcs
Running stage srcs at 2020-08-07T12:32:38.778726
SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])]
Found 13 potential peaks
Of 13 potential peaks: 5 in veto map, 1 cut by saddle test, 0 cut by aper test, 7 kept
SED z : 7 new peaks
Found 18 potential peaks
Of 18 potential peaks: 14 in veto map, 3 cut by saddle test, 0 cut by aper test, 1 kept
SED r : 1 new peaks
Found 21 potential peaks
Of 21 potential peaks: 17 in veto map, 3 cut by saddle test, 0 cut by aper test, 1 kept
SED g : 1 new peaks
Found 25 potential peaks
Of 25 potential peaks: 20 in veto map, 2 cut by saddle test, 0 cut by aper test, 3 kept
SED Flat : 3 new peaks
Found 25 potential peaks
Of 25 potential peaks: 21 in veto map, 2 cut by saddle test, 0 cut by aper test, 2 kept
SED Red : 2 new peaks
Adding 14 new sources
Median coadd sky for g : 7.9724465e-05
Median coadd sky for r : -8.704088e-05
Median coadd sky for z : 3.2420263e-05
Stage srcs : Wall: 0.58 s, CPU: 0.52 s, VmPeak: 982 MB, VmSize: 980 MB, VmRSS: 215 MB, VmData: 244 MB, maxrss: 0.220668 MB
Stage srcs finished: 2020-08-07T12:32:39.357112
Resources for stage srcs : 0.098 s worker CPU, 0.107 s worker Wall, Wall: 0.581 s, Cores in use: 1.07, Total efficiency (on 32 cores): 3.4 %
Stage srcs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-srcs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-srcs.p
Running stage fitblobs
Running stage fitblobs at 2020-08-07T12:32:39.370779
Blob 1 of 15, id: 3, sources: 1, size: 92x158, npix 8954, brick X: 92,184, Y: 58,216, one pixel: 172 58
Blob 2 of 15, id: 10, sources: 3, size: 83x73, npix 3449, brick X: 197,280, Y: 202,275, one pixel: 253 202
Blob 3 of 15, id: 2, sources: 2, size: 62x53, npix 2089, brick X: 65,127, Y: 28,81, one pixel: 88 28
Blob 4 of 15, id: 12, sources: 3, size: 55x54, npix 1786, brick X: 30,85, Y: 227,281, one pixel: 57 227
Blob 5 of 15, id: 9, sources: 2, size: 28x49, npix 958, brick X: 0,28, Y: 177,226, one pixel: 5 177
Blob 6 of 15, id: 6, sources: 1, size: 28x28, npix 549, brick X: 36,64, Y: 134,162, one pixel: 49 134
Blob 7 of 15, id: 14, sources: 1, size: 30x21, npix 452, brick X: 193,223, Y: 260,281, one pixel: 205 260
Blob 8 of 15, id: 4, sources: 1, size: 25x26, npix 423, brick X: 237,262, Y: 69,95, one pixel: 249 69
Blob 9 of 15, id: 0, sources: 2, size: 27x15, npix 289, brick X: 130,157, Y: 0,15, one pixel: 131 0
Blob 10 of 15, id: 5, sources: 1, size: 19x20, npix 236, brick X: 194,213, Y: 74,94, one pixel: 202 74
Blob 11 of 15, id: 11, sources: 1, size: 20x20, npix 219, brick X: 152,172, Y: 204,224, one pixel: 161 204
Blob 12 of 15, id: 1, sources: 1, size: 19x19, npix 217, brick X: 258,277, Y: 20,39, one pixel: 266 20
Blob 13 of 15, id: 8, sources: 1, size: 19x19, npix 208, brick X: 180,199, Y: 172,191, one pixel: 188 172
Blob 14 of 15, id: 13, sources: 1, size: 19x19, npix 199, brick X: 119,138, Y: 232,251, one pixel: 127 232
Blob 15 of 15, id: 7, sources: 1, size: 17x18, npix 162, brick X: 70,87, Y: 149,167, one pixel: 78 149
Blob 15 finished, total: Wall: 0.06 s, CPU: 0.05 s, VmPeak: 804 MB, VmSize: 724 MB, VmRSS: 141 MB, VmData: 182 MB, maxrss: 0.14478 MB
Blob 12 finished, total: Wall: 0.11 s, CPU: 0.08 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 83 MB, VmData: 137 MB, maxrss: 0.084524 MB
Blob 13 finished, total: Wall: 0.11 s, CPU: 0.09 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 82 MB, VmData: 137 MB, maxrss: 0.084044 MB
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.04343, 54.22332), pm (-5.4, -5.8), parallax 1.390 with NanoMaggies: g=24.4, r=24.4, z=20.5
Reset brightness to NanoMaggies: g=21.8, r=20.4, z=19.3
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.05403, 54.24420), pm (-4.8, -13.9), parallax 0.167 with NanoMaggies: g=18.5, r=(flux -14.1), z=17.7
Reset brightness to NanoMaggies: g=16.4, r=15.8, z=15.6
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.06134, 54.23807), pm (26.3, 0.7), parallax 1.862 with NanoMaggies: g=18.1, r=17.3, z=15.7
Reset brightness to NanoMaggies: g=17.5, r=16.3, z=15.6
Blob 9 finished, total: Wall: 0.15 s, CPU: 0.14 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 82 MB, VmData: 136 MB, maxrss: 0.08448 MB
Blob 11 finished, total: Wall: 0.18 s, CPU: 0.17 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.08962 MB
Blob 10 finished, total: Wall: 0.20 s, CPU: 0.18 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 87 MB, VmData: 137 MB, maxrss: 0.089524 MB
Blob 14 finished, total: Wall: 0.23 s, CPU: 0.21 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.0906 MB
Blob 8 finished, total: Wall: 0.44 s, CPU: 0.42 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.0898 MB
Blob 4 finished, total: Wall: 0.55 s, CPU: 0.55 s, VmPeak: 809 MB, VmSize: 725 MB, VmRSS: 142 MB, VmData: 183 MB, maxrss: 0.1454 MB
Measuring 1 iterative sources
Blob 6 finished, total: Wall: 0.63 s, CPU: 0.60 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.089932 MB
kdtree_internal.c:2152:kdtree_build_2_ddd: Data, N, or D is zero
Blob 7 finished, total: Wall: 0.89 s, CPU: 0.88 s, VmPeak: 807 MB, VmSize: 725 MB, VmRSS: 142 MB, VmData: 183 MB, maxrss: 0.145648 MB
Exception in one_blob: brick custom-253043p54233, iblob 3
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1641, in _bounce_one_blob
    result = one_blob(X)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob
    B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 345, in run
    ras[not_galaxy], decs[not_galaxy], 1./3600.)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec
    X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match
    (kd1,kd2) = _buildtrees(x1, x2)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees
    kd2 = spherematch_c.KdTree(fx2)
SystemError: <class 'spherematch.KdTree'> returned NULL without setting an error
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python/astrometry/util/timingpool.py", line 269, in __call__
    R = self.func(*args, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1641, in _bounce_one_blob
    result = one_blob(X)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob
    B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 345, in run
    ras[not_galaxy], decs[not_galaxy], 1./3600.)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec
    X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match
    (kd1,kd2) = _buildtrees(x1, x2)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees
    kd2 = spherematch_c.KdTree(fx2)
SystemError: <class 'spherematch.KdTree'> returned NULL without setting an error
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3775, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3743, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3299, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3288, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1135, in stage_fitblobs
    r = Riter.next(timeout)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 748, in next
    raise value
SystemError: <class 'spherematch.KdTree'> returned NULL without setting an error
Something went wrong; please check the logfile.
ERROR: galaxy PGC2463611; please check the logfile.
Finished galaxy PGC2463611 in 0.144 minutes.
Started working on galaxy PGC2463611 at Fri Aug  7 13:33:28 2020
Bands touching this brick, g r z
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 253.0437484055481 54.23361969915836 --width 281 --height 281 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 

runbrick.py starting at 2020-08-07T13:33:29.094569
legacypipe git version: DR9.5.11-watershed-5-g77ea3463
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '253.0437484055481', '54.23361969915836', '--width', '281', '--height', '281', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 253.0437484055481 54.23361969915836 --width 281 --height 281 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0

Parsed RA,Dec 253.0437484055481 54.23361969915836
Starting process 20132 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB
Starting process 20133 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 20134 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 20135 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 20136 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05954 MB
Starting process 20137 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059488 MB
Starting process 20138 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059488 MB
Starting process 20139 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 20140 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 20141 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 20142 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 20143 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 20144 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 20145 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 20146 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 20147 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 20148 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 20149 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059276 MB
Starting process 20150 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB
Starting process 20151 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB
Starting process 20152 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB
Starting process 20153 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB
Starting process 20154 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB
Starting process 20155 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB
Starting process 20156 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB
Starting process 20157 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB
Starting process 20158 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Starting process 20159 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059308 MB
Starting process 20160 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB
Starting process 20161 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB
Starting process 20162 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB
Starting process 20163 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059316 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage fit_on_coadds
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-08-07T13:33:29.925691
Keeping 11 CCDs:
  k4m_160615_064734_ooi_zd_ls9-CCD3 z expnum 95405 exptime 80.105 propid 2016A-0453 seeing 0.94 MJD 57554.283 object MzLS_666483_z 
    mosaic/CP/V4.3/CP20160614/k4m_160615_064734_ooi_zd_ls9.fits.fz
  k4m_170412_112301_ooi_zd_ls9-CCD1 z expnum 121439 exptime 107.105 propid 2016A-0453 seeing 1.06 MJD 57855.474 object MzLS_667217_z 
    mosaic/CP/V4.3/CP20170411/k4m_170412_112301_ooi_zd_ls9.fits.fz
  k4m_170523_095715_ooi_zd_ls9-CCD4 z expnum 129693 exptime 84.101 propid 2016A-0453 seeing 1.19 MJD 57896.415 object MzLS_665745_z 
    mosaic/CP/V4.3/CP20170522/k4m_170523_095715_ooi_zd_ls9.fits.fz
  ksb_170520_101640_ooi_r_ls9-CCD4 r expnum 78930197 exptime 181.5 propid 2015A-0801 seeing 2.04 MJD 57893.428 object 32623 
    90prime/CP/V2.3/CP20170519/ksb_170520_101640_ooi_r_ls9.fits.fz
  ksb_170523_080943_ooi_g_ls9-CCD4 g expnum 78960126 exptime 100.18 propid 2015A-0801 seeing 1.45 MJD 57896.340 object 32623 
    90prime/CP/V2.3/CP20170522/ksb_170523_080943_ooi_g_ls9.fits.fz
  ksb_160712_052408_ooi_r_ls9-CCD2 r expnum 75810061 exptime 233.3 propid BASS seeing 1.73 MJD 57581.225 object 31542 
    90prime/CP/V2.3/CP20160711/ksb_160712_052408_ooi_r_ls9.fits.fz
  ksb_170327_113133_ooi_g_ls9-CCD2 g expnum 78390275 exptime 91.542 propid BASS seeing 1.41 MJD 57839.480 object 31542 
    90prime/CP/V2.3/CP20170326/ksb_170327_113133_ooi_g_ls9.fits.fz
  ksb_170501_090154_ooi_r_ls9-CCD3 r expnum 78740200 exptime 62.028 propid BASS seeing 1.12 MJD 57874.376 object 31553 
    90prime/CP/V2.3/CP20170430/ksb_170501_090154_ooi_r_ls9.fits.fz
  ksb_170523_081418_ooi_g_ls9-CCD3 g expnum 78960128 exptime 103.558 propid 2015A-0801 seeing 1.52 MJD 57896.343 object 31553 
    90prime/CP/V2.3/CP20170522/ksb_170523_081418_ooi_g_ls9.fits.fz
  ksb_160531_080046_ooi_r_ls9-CCD1 r expnum 75390152 exptime 65.0 propid BASS seeing 1.16 MJD 57539.334 object 31541 
    90prime/CP/V2.3/CP20160530/ksb_160531_080046_ooi_r_ls9.fits.fz
  ksb_170602_101151_ooi_g_ls9-CCD1 g expnum 79060155 exptime 124.85 propid 2015A-0801 seeing 1.53 MJD 57906.425 object 31541 
    90prime/CP/V2.3/CP20170601/ksb_170602_101151_ooi_g_ls9.fits.fz
mosaic get_fwhm: self.fwhm = 4.09
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 841 1131 and amp range 0 2005 and image y 1016 1307 and amp range 0 2039
Range in image: x 0 291 , y 0 292 (with image size 291 x 292)
dzp 0.007397243 -> scaling image by 1.006836374469654
Faked up EXPNUM 78740200
Faked up EXPNUM 78740200
Faked up EXPNUM 78740200
90prime get_fwhm: self.fwhm = 2.47
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 186 352 and amp range 0 2015 and image y 2968 3132 and amp range 2048 4095
Range in image: x 0 167 , y 0 165 (with image size 167 x 165)
dzp -0.07725406 -> scaling image by 0.9313188157188347
mosaic get_fwhm: self.fwhm = 3.61
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3609 3891 and amp range 2006 4053 and image y 897 1180 and amp range 0 2039
Range in image: x 0 283 , y 0 284 (with image size 283 x 284)
dzp -0.00064781494 -> scaling image by 0.9994035183523652
Faked up EXPNUM 78960128
Faked up EXPNUM 78960128
Faked up EXPNUM 78960128
90prime get_fwhm: self.fwhm = 3.35
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 186 352 and amp range 0 2015 and image y 2963 3127 and amp range 2048 4095
Range in image: x 0 167 , y 0 165 (with image size 167 x 165)
dzp -0.045992933 -> scaling image by 0.958523636456738
Faked up EXPNUM 78960126
Faked up EXPNUM 78960126
Faked up EXPNUM 78960126
90prime get_fwhm: self.fwhm = 3.19
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3966 4009 and amp range 2016 4031 and image y 15 67 and amp range 0 2047
Range in image: x 0 44 , y 0 53 (with image size 44 x 53)
dzp 0.016679538 -> scaling image by 1.0154810312115594
Faked up EXPNUM 79060155
Faked up EXPNUM 79060155
Faked up EXPNUM 79060155
90prime get_fwhm: self.fwhm = 3.37
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2135 2301 and amp range 2016 4031 and image y 1453 1618 and amp range 0 2047
Range in image: x 0 167 , y 0 166 (with image size 167 x 166)
dzp -0.011406615 -> scaling image by 0.9895491130016149
mosaic get_fwhm: self.fwhm = 4.59
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2054 2345 and amp range 2048 4095 and image y 3243 3531 and amp range 2022 4060
Range in image: x 0 292 , y 0 289 (with image size 292 x 289)
dzp -0.02499645 -> scaling image by 0.9772404160512723
Faked up EXPNUM 78930197
Faked up EXPNUM 78930197
Faked up EXPNUM 78930197
90prime get_fwhm: self.fwhm = 4.49
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3959 4009 and amp range 2016 4031 and image y 15 79 and amp range 0 2047
Range in image: x 0 51 , y 0 65 (with image size 51 x 65)
dzp 0.02499258 -> scaling image by 1.0232859986636875
Faked up EXPNUM 75810061
Faked up EXPNUM 75810061
Faked up EXPNUM 75810061
90prime get_fwhm: self.fwhm = 3.81
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 15 121 and amp range 0 2015 and image y 3423 3587 and amp range 2048 4095
Range in image: x 0 107 , y 0 165 (with image size 107 x 165)
dzp 0.016961161 -> scaling image by 1.0157444649385616
Faked up EXPNUM 75390152
Faked up EXPNUM 75390152
Faked up EXPNUM 75390152
90prime get_fwhm: self.fwhm = 2.57
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2121 2287 and amp range 2016 4031 and image y 1444 1610 and amp range 0 2047
Range in image: x 0 167 , y 0 167 (with image size 167 x 167)
dzp -0.0041296566 -> scaling image by 0.9962036700684744
Faked up EXPNUM 78390275
Faked up EXPNUM 78390275
Faked up EXPNUM 78390275
90prime get_fwhm: self.fwhm = 3.11
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 15 134 and amp range 0 2015 and image y 3425 3589 and amp range 2048 4095
Range in image: x 0 120 , y 0 165 (with image size 120 x 165)
dzp 0.051889952 -> scaling image by 1.0489528829333021
Stage tims : Wall: 3.90 s, CPU: 0.83 s, VmPeak: 951 MB, VmSize: 951 MB, VmRSS: 174 MB, VmData: 207 MB, maxrss: 0.177848 MB
Stage tims finished: 2020-08-07T13:33:33.825209
Resources for stage tims : 5.426 s worker CPU, 15.969 s worker Wall, Wall: 3.902 s, Cores in use: 1.60, Total efficiency (on 32 cores): 5.0 %
Stage tims finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-tims.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-tims.p
Running stage refs
Running stage refs at 2020-08-07T13:33:33.849423
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-01171.fits
Matched 13 large galaxies to Gaia stars.
Matched 0 large galaxies to Tycho-2 stars.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/reference-custom-253043p54233.fits
Stage refs : Wall: 2.45 s, CPU: 2.22 s, VmPeak: 982 MB, VmSize: 975 MB, VmRSS: 198 MB, VmData: 231 MB, maxrss: 0.202932 MB
Stage refs finished: 2020-08-07T13:33:36.298395
Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 2.451 s, Cores in use: 0.90, Total efficiency (on 32 cores): 2.8 %
Stage refs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-refs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-refs.p
Running stage outliers
Running stage outliers at 2020-08-07T13:33:36.322440
Stage outliers : Wall: 0.32 s, CPU: 0.29 s, VmPeak: 982 MB, VmSize: 975 MB, VmRSS: 199 MB, VmData: 231 MB, maxrss: 0.204128 MB
Stage outliers finished: 2020-08-07T13:33:36.638582
Resources for stage outliers : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 0.318 s, Cores in use: 0.93, Total efficiency (on 32 cores): 2.9 %
Stage outliers finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-outliers.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-outliers.p
Running stage halos
Running stage halos at 2020-08-07T13:33:36.663789
Stage halos : Wall: 0.32 s, CPU: 0.31 s, VmPeak: 982 MB, VmSize: 975 MB, VmRSS: 202 MB, VmData: 235 MB, maxrss: 0.207296 MB
Stage halos finished: 2020-08-07T13:33:36.984671
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.323 s, Cores in use: 0.97, Total efficiency (on 32 cores): 3.0 %
Stage halos finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-halos.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-halos.p
Running stage fit_on_coadds
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Running stage fit_on_coadds at 2020-08-07T13:33:37.020574
PSF sigmas in coadd pixels: ['2.35', '2.29', '2.46', '2.48']
Using average PSF sigma 2.395566358689102
Gaussian PSF norm 0.11775703509555277 vs pixelized 0.09646213
average tim pixel scale / coadd scale: 1.729534137354211
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-copsf-g.fits.fz
PSF sigmas in coadd pixels: ['3.31', '2.80', '1.82', '1.89']
Using average PSF sigma 2.454527133479732
Gaussian PSF norm 0.11492836560089613 vs pixelized 0.09748002
average tim pixel scale / coadd scale: 1.7294610056489392
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-copsf-r.fits.fz
PSF sigmas in coadd pixels: ['1.52', '1.72', '1.94']
Using average PSF sigma 1.7273379876689077
Gaussian PSF norm 0.16331186703916192 vs pixelized 0.13717054
average tim pixel scale / coadd scale: 0.9908615580390256
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-copsf-z.fits.fz
Stage fit_on_coadds : Wall: 0.21 s, CPU: 0.17 s, VmPeak: 982 MB, VmSize: 980 MB, VmRSS: 214 MB, VmData: 244 MB, maxrss: 0.219628 MB
Stage fit_on_coadds finished: 2020-08-07T13:33:37.226183
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage fit_on_coadds : 0.222 s worker CPU, 0.281 s worker Wall, Wall: 0.208 s, Cores in use: 1.90, Total efficiency (on 32 cores): 5.9 %
Stage fit_on_coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-fit_on_coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-fit_on_coadds.p
Running stage srcs
Running stage srcs at 2020-08-07T13:33:37.238095
SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])]
Found 13 potential peaks
Of 13 potential peaks: 5 in veto map, 1 cut by saddle test, 0 cut by aper test, 7 kept
SED z : 7 new peaks
Found 18 potential peaks
Of 18 potential peaks: 14 in veto map, 3 cut by saddle test, 0 cut by aper test, 1 kept
SED r : 1 new peaks
Found 21 potential peaks
Of 21 potential peaks: 17 in veto map, 3 cut by saddle test, 0 cut by aper test, 1 kept
SED g : 1 new peaks
Found 25 potential peaks
Of 25 potential peaks: 20 in veto map, 2 cut by saddle test, 0 cut by aper test, 3 kept
SED Flat : 3 new peaks
Found 25 potential peaks
Of 25 potential peaks: 21 in veto map, 2 cut by saddle test, 0 cut by aper test, 2 kept
SED Red : 2 new peaks
Adding 14 new sources
Median coadd sky for g : 7.9724465e-05
Median coadd sky for r : -8.704088e-05
Median coadd sky for z : 3.242027e-05
Stage srcs : Wall: 0.69 s, CPU: 0.52 s, VmPeak: 982 MB, VmSize: 980 MB, VmRSS: 215 MB, VmData: 244 MB, maxrss: 0.220488 MB
Stage srcs finished: 2020-08-07T13:33:37.925235
Resources for stage srcs : 0.126 s worker CPU, 0.136 s worker Wall, Wall: 0.689 s, Cores in use: 0.94, Total efficiency (on 32 cores): 2.9 %
Stage srcs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-srcs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-srcs.p
Running stage fitblobs
Running stage fitblobs at 2020-08-07T13:33:37.938983
Blob 1 of 15, id: 3, sources: 1, size: 92x158, npix 8954, brick X: 92,184, Y: 58,216, one pixel: 172 58
Blob 2 of 15, id: 10, sources: 3, size: 83x73, npix 3449, brick X: 197,280, Y: 202,275, one pixel: 253 202
Blob 3 of 15, id: 2, sources: 2, size: 62x53, npix 2089, brick X: 65,127, Y: 28,81, one pixel: 88 28
Blob 4 of 15, id: 12, sources: 3, size: 55x54, npix 1786, brick X: 30,85, Y: 227,281, one pixel: 57 227
Blob 5 of 15, id: 9, sources: 2, size: 28x49, npix 958, brick X: 0,28, Y: 177,226, one pixel: 5 177
Blob 6 of 15, id: 6, sources: 1, size: 28x28, npix 549, brick X: 36,64, Y: 134,162, one pixel: 49 134
Blob 7 of 15, id: 14, sources: 1, size: 30x21, npix 452, brick X: 193,223, Y: 260,281, one pixel: 205 260
Blob 8 of 15, id: 4, sources: 1, size: 25x26, npix 423, brick X: 237,262, Y: 69,95, one pixel: 249 69
Blob 9 of 15, id: 0, sources: 2, size: 27x15, npix 289, brick X: 130,157, Y: 0,15, one pixel: 131 0
Blob 10 of 15, id: 5, sources: 1, size: 19x20, npix 236, brick X: 194,213, Y: 74,94, one pixel: 202 74
Blob 11 of 15, id: 11, sources: 1, size: 20x20, npix 219, brick X: 152,172, Y: 204,224, one pixel: 161 204
Blob 12 of 15, id: 1, sources: 1, size: 19x19, npix 217, brick X: 258,277, Y: 20,39, one pixel: 266 20
Blob 13 of 15, id: 8, sources: 1, size: 19x19, npix 208, brick X: 180,199, Y: 172,191, one pixel: 188 172
Blob 14 of 15, id: 13, sources: 1, size: 19x19, npix 199, brick X: 119,138, Y: 232,251, one pixel: 127 232
Blob 15 of 15, id: 7, sources: 1, size: 17x18, npix 162, brick X: 70,87, Y: 149,167, one pixel: 78 149
Blob 13 finished, total: Wall: 0.11 s, CPU: 0.06 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 82 MB, VmData: 137 MB, maxrss: 0.084068 MB
Blob 12 finished, total: Wall: 0.11 s, CPU: 0.06 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 82 MB, VmData: 137 MB, maxrss: 0.08426 MB
Blob 15 finished, total: Wall: 0.13 s, CPU: 0.07 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 84 MB, VmData: 137 MB, maxrss: 0.0864 MB
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.06134, 54.23807), pm (26.3, 0.7), parallax 1.862 with NanoMaggies: g=18.1, r=17.3, z=15.7
Reset brightness to NanoMaggies: g=17.5, r=16.3, z=15.6
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.04343, 54.22332), pm (-5.4, -5.8), parallax 1.390 with NanoMaggies: g=24.4, r=24.4, z=20.5
Reset brightness to NanoMaggies: g=21.8, r=20.4, z=19.3
Blob 9 finished, total: Wall: 0.22 s, CPU: 0.12 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 84 MB, VmData: 137 MB, maxrss: 0.085604 MB
Blob 10 finished, total: Wall: 0.23 s, CPU: 0.17 s, VmPeak: 804 MB, VmSize: 724 MB, VmRSS: 143 MB, VmData: 182 MB, maxrss: 0.146524 MB
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.05403, 54.24420), pm (-4.8, -13.9), parallax 0.167 with NanoMaggies: g=18.5, r=(flux -14.1), z=17.7
Reset brightness to NanoMaggies: g=16.4, r=15.8, z=15.6
Blob 11 finished, total: Wall: 0.24 s, CPU: 0.17 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.089652 MB
Blob 14 finished, total: Wall: 0.26 s, CPU: 0.23 s, VmPeak: 804 MB, VmSize: 724 MB, VmRSS: 143 MB, VmData: 182 MB, maxrss: 0.146564 MB
Blob 8 finished, total: Wall: 0.44 s, CPU: 0.41 s, VmPeak: 804 MB, VmSize: 724 MB, VmRSS: 143 MB, VmData: 182 MB, maxrss: 0.146024 MB
Measuring 1 iterative sources
Blob 4 finished, total: Wall: 0.65 s, CPU: 0.55 s, VmPeak: 667 MB, VmSize: 666 MB, VmRSS: 89 MB, VmData: 137 MB, maxrss: 0.091676 MB
Blob 6 finished, total: Wall: 0.70 s, CPU: 0.60 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.089976 MB
kdtree_internal.c:2152:kdtree_build_2_ddd: Data, N, or D is zero
Blob 7 finished, total: Wall: 0.92 s, CPU: 0.88 s, VmPeak: 807 MB, VmSize: 725 MB, VmRSS: 142 MB, VmData: 183 MB, maxrss: 0.145892 MB
Exception in one_blob: brick custom-253043p54233, iblob 3
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1641, in _bounce_one_blob
    result = one_blob(X)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob
    B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 345, in run
    ras[not_galaxy], decs[not_galaxy], 1./3600.)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec
    X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match
    (kd1,kd2) = _buildtrees(x1, x2)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees
    kd2 = spherematch_c.KdTree(fx2)
SystemError: <class 'spherematch.KdTree'> returned NULL without setting an error
multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 121, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python/astrometry/util/timingpool.py", line 269, in __call__
    R = self.func(*args, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1641, in _bounce_one_blob
    result = one_blob(X)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob
    B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 345, in run
    ras[not_galaxy], decs[not_galaxy], 1./3600.)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec
    X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match
    (kd1,kd2) = _buildtrees(x1, x2)
  File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees
    kd2 = spherematch_c.KdTree(fx2)
SystemError: <class 'spherematch.KdTree'> returned NULL without setting an error
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3775, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3743, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3299, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3288, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1135, in stage_fitblobs
    r = Riter.next(timeout)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 748, in next
    raise value
SystemError: <class 'spherematch.KdTree'> returned NULL without setting an error
Something went wrong; please check the logfile.
ERROR: galaxy PGC2463611; please check the logfile.
Finished galaxy PGC2463611 in 0.187 minutes.
Started working on galaxy PGC2463611 at Fri Aug  7 14:38:57 2020
Bands touching this brick, g r z
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 253.0437484055481 54.23361969915836 --width 281 --height 281 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 

runbrick.py starting at 2020-08-07T14:38:57.859379
legacypipe git version: DR9.5.11-watershed-6-g6a2880fc
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '253.0437484055481', '54.23361969915836', '--width', '281', '--height', '281', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 253.0437484055481 54.23361969915836 --width 281 --height 281 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0

Parsed RA,Dec 253.0437484055481 54.23361969915836
Starting process 48712 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059232 MB
Starting process 48713 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059236 MB
Starting process 48714 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059236 MB
Starting process 48715 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB
Starting process 48716 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05948 MB
Starting process 48717 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059484 MB
Starting process 48718 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059484 MB
Starting process 48719 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB
Starting process 48720 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB
Starting process 48721 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 48723 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 48724 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 48725 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 48726 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 48727 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 48728 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 48729 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 48730 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 48731 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059276 MB
Starting process 48732 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059276 MB
Starting process 48733 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059276 MB
Starting process 48734 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05928 MB
Starting process 48735 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB
Starting process 48736 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB
Starting process 48737 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB
Starting process 48738 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB
Starting process 48739 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059292 MB
Starting process 48740 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059292 MB
Starting process 48741 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB
Starting process 48742 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB
Starting process 48743 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB
Starting process 48744 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Reading pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-srcs.p
Running stage fitblobs
Running stage fitblobs at 2020-08-07T14:38:58.347521
Blob 1 of 15, id: 3, sources: 1, size: 92x158, npix 8954, brick X: 92,184, Y: 58,216, one pixel: 172 58
Blob 2 of 15, id: 10, sources: 3, size: 83x73, npix 3449, brick X: 197,280, Y: 202,275, one pixel: 253 202
Blob 3 of 15, id: 2, sources: 2, size: 62x53, npix 2089, brick X: 65,127, Y: 28,81, one pixel: 88 28
Blob 4 of 15, id: 12, sources: 3, size: 55x54, npix 1786, brick X: 30,85, Y: 227,281, one pixel: 57 227
Blob 5 of 15, id: 9, sources: 2, size: 28x49, npix 958, brick X: 0,28, Y: 177,226, one pixel: 5 177
Blob 6 of 15, id: 6, sources: 1, size: 28x28, npix 549, brick X: 36,64, Y: 134,162, one pixel: 49 134
Blob 7 of 15, id: 14, sources: 1, size: 30x21, npix 452, brick X: 193,223, Y: 260,281, one pixel: 205 260
Blob 8 of 15, id: 4, sources: 1, size: 25x26, npix 423, brick X: 237,262, Y: 69,95, one pixel: 249 69
Blob 9 of 15, id: 0, sources: 2, size: 27x15, npix 289, brick X: 130,157, Y: 0,15, one pixel: 131 0
Blob 10 of 15, id: 5, sources: 1, size: 19x20, npix 236, brick X: 194,213, Y: 74,94, one pixel: 202 74
Blob 11 of 15, id: 11, sources: 1, size: 20x20, npix 219, brick X: 152,172, Y: 204,224, one pixel: 161 204
Blob 12 of 15, id: 1, sources: 1, size: 19x19, npix 217, brick X: 258,277, Y: 20,39, one pixel: 266 20
Blob 13 of 15, id: 8, sources: 1, size: 19x19, npix 208, brick X: 180,199, Y: 172,191, one pixel: 188 172
Blob 14 of 15, id: 13, sources: 1, size: 19x19, npix 199, brick X: 119,138, Y: 232,251, one pixel: 127 232
Blob 15 of 15, id: 7, sources: 1, size: 17x18, npix 162, brick X: 70,87, Y: 149,167, one pixel: 78 149
Blob 12 finished, total: Wall: 0.09 s, CPU: 0.06 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 82 MB, VmData: 136 MB, maxrss: 0.083972 MB
Blob 15 finished, total: Wall: 0.10 s, CPU: 0.06 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 82 MB, VmData: 136 MB, maxrss: 0.083768 MB
Blob 13 finished, total: Wall: 0.11 s, CPU: 0.06 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 82 MB, VmData: 136 MB, maxrss: 0.08378 MB
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.04343, 54.22332), pm (-5.4, -5.8), parallax 1.390 with NanoMaggies: g=24.4, r=24.4, z=20.5
Reset brightness to NanoMaggies: g=21.8, r=20.4, z=19.3
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.06134, 54.23807), pm (26.3, 0.7), parallax 1.862 with NanoMaggies: g=18.1, r=17.3, z=15.7
Reset brightness to NanoMaggies: g=17.5, r=16.3, z=15.6
Blob 9 finished, total: Wall: 0.15 s, CPU: 0.12 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 82 MB, VmData: 136 MB, maxrss: 0.08446 MB
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.05403, 54.24420), pm (-4.8, -13.9), parallax 0.167 with NanoMaggies: g=18.5, r=(flux -14.1), z=17.7
Reset brightness to NanoMaggies: g=16.4, r=15.8, z=15.6
Blob 11 finished, total: Wall: 0.19 s, CPU: 0.17 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 87 MB, VmData: 137 MB, maxrss: 0.089584 MB
Blob 10 finished, total: Wall: 0.21 s, CPU: 0.18 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 87 MB, VmData: 137 MB, maxrss: 0.089472 MB
Blob 14 finished, total: Wall: 0.26 s, CPU: 0.21 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 87 MB, VmData: 137 MB, maxrss: 0.089404 MB
Blob 8 finished, total: Wall: 0.41 s, CPU: 0.38 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.089832 MB
Measuring 1 iterative sources
Blob 6 finished, total: Wall: 0.54 s, CPU: 0.53 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.08984 MB
Blob 4 finished, total: Wall: 0.56 s, CPU: 0.53 s, VmPeak: 667 MB, VmSize: 666 MB, VmRSS: 89 MB, VmData: 137 MB, maxrss: 0.09166 MB
Blob 7 finished, total: Wall: 0.86 s, CPU: 0.83 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 89 MB, VmData: 137 MB, maxrss: 0.091188 MB
Blob 5 finished, total: Wall: 1.13 s, CPU: 1.11 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 90 MB, VmData: 138 MB, maxrss: 0.092268 MB
Blob 3 finished, total: Wall: 1.17 s, CPU: 1.15 s, VmPeak: 668 MB, VmSize: 667 MB, VmRSS: 89 MB, VmData: 138 MB, maxrss: 0.092484 MB
Blob 2 finished, total: Wall: 1.51 s, CPU: 1.50 s, VmPeak: 671 MB, VmSize: 669 MB, VmRSS: 92 MB, VmData: 140 MB, maxrss: 0.09702 MB
Known galaxy.  Initial shape: EllipseWithPriors(0.25): log r_e=1.78286, ee1=0.654341, ee2=0.360766
Blob 1 finished, total: Wall: 4.53 s, CPU: 4.51 s, VmPeak: 683 MB, VmSize: 672 MB, VmRSS: 99 MB, VmData: 144 MB, maxrss: 0.111516 MB
Old catalog: 31
New catalog: 18
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/blobs-custom-253043p54233.fits.gz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/all-models-custom-253043p54233.fits
Stage fitblobs : Wall: 4.83 s, CPU: 0.28 s, VmPeak: 858 MB, VmSize: 858 MB, VmRSS: 114 MB, VmData: 153 MB, maxrss: 0.200356 MB
Stage fitblobs finished: 2020-08-07T14:39:03.182290
Resources for stage fitblobs : 11.613 s worker CPU, 12.083 s worker Wall, Wall: 4.837 s, Cores in use: 2.46, Total efficiency (on 32 cores): 7.7 %
Stage fitblobs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-fitblobs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-fitblobs.p
Running stage coadds
Running stage coadds at 2020-08-07T14:39:03.197702
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-ccds.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-image-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-invvar-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-nexp-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-depth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-galdepth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-psfsize-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-model-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-chi2-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-blobmodel-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-image-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-invvar-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-nexp-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-depth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-galdepth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-psfsize-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-model-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-chi2-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-blobmodel-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-image-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-invvar-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-nexp-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-depth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-galdepth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-psfsize-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-model-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-chi2-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-blobmodel-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-depth.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/tmp-legacysurvey-custom-253043p54233-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/tmp-legacysurvey-custom-253043p54233-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/tmp-legacysurvey-custom-253043p54233-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/tmp-legacysurvey-custom-253043p54233-resid.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-resid.jpg
Stage coadds : Wall: 2.85 s, CPU: 0.87 s, VmPeak: 901 MB, VmSize: 892 MB, VmRSS: 140 MB, VmData: 181 MB, maxrss: 0.200356 MB
Stage coadds finished: 2020-08-07T14:39:06.045016
Resources for stage coadds : 37.679 s worker CPU, 42.841 s worker Wall, Wall: 2.850 s, Cores in use: 13.53, Total efficiency (on 32 cores): 42.3 %
Stage coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-coadds.p
Running stage wise_forced
Running stage wise_forced at 2020-08-07T14:39:06.060258
Cut to 1 unWISE tiles
Reading WISE tile 2527p545 band 1
Reading WISE tile 2527p545 band 3
Reading WISE tile 2527p545 band 2
Reading WISE tile 2527p545 band 4
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.04 s, CPU: 0.04 s, VmPeak: 795 MB, VmSize: 795 MB, VmRSS: 141 MB, VmData: 175 MB, maxrss: 0.176344 MB
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.03 s, CPU: 0.03 s, VmPeak: 798 MB, VmSize: 794 MB, VmRSS: 137 MB, VmData: 170 MB, maxrss: 0.197132 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w1-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w1-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w1-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w1-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.03 s, CPU: 0.03 s, VmPeak: 792 MB, VmSize: 792 MB, VmRSS: 137 MB, VmData: 172 MB, maxrss: 0.195204 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w2-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w2-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w2-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w2-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.04 s, CPU: 0.04 s, VmPeak: 790 MB, VmSize: 789 MB, VmRSS: 134 MB, VmData: 170 MB, maxrss: 0.192116 MB
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-image-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-model-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-invvar-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-image-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-model-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-invvar-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-image-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-model-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-invvar-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-image-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-model-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-invvar-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/tmp-legacysurvey-custom-253043p54233-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/tmp-legacysurvey-custom-253043p54233-wisemodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-wisemodel.jpg
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Stage wise_forced : Wall: 1.91 s, CPU: 0.59 s, VmPeak: 927 MB, VmSize: 927 MB, VmRSS: 156 MB, VmData: 193 MB, maxrss: 0.200356 MB
Stage wise_forced finished: 2020-08-07T14:39:07.967377
Resources for stage wise_forced : 3.136 s worker CPU, 4.222 s worker Wall, Wall: 1.910 s, Cores in use: 1.95, Total efficiency (on 32 cores): 6.1 %
Stage wise_forced finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-wise_forced.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-wise_forced.p
Running stage writecat
Running stage writecat at 2020-08-07T14:39:08.069521
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-maskbits.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/tractor-i/cus/tractor-custom-253043p54233.fits
Reading SFD maps...
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/tractor/cus/tractor-custom-253043p54233.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/tractor/cus/brick-custom-253043p54233.sha256sum
Stage writecat : Wall: 0.59 s, CPU: 0.43 s, VmPeak: 1072 MB, VmSize: 944 MB, VmRSS: 172 MB, VmData: 206 MB, maxrss: 0.306688 MB
Stage writecat finished: 2020-08-07T14:39:08.658337
Resources for stage writecat : 0.000 s worker CPU, 0.001 s worker Wall, Wall: 0.591 s, Cores in use: 0.74, Total efficiency (on 32 cores): 2.3 %
Stage writecat finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-writecat.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-writecat.p
All done: 52.430 s worker CPU, 59.149 s worker Wall, Wall: 10.378 s, Cores in use: 5.27, Total efficiency (on 32 cores): 16.5 %
Finished galaxy PGC2463611 in 0.202 minutes.
Started working on galaxy PGC2463611 at Fri Aug  7 14:45:04 2020
Bands touching this brick, g r z
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 253.0437484055481 54.23361969915836 --width 281 --height 281 --pixscale 0.262 --threads 1 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 

runbrick.py starting at 2020-08-07T14:45:04.989810
legacypipe git version: DR9.5.11-watershed-6-g6a2880fc
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '253.0437484055481', '54.23361969915836', '--width', '281', '--height', '281', '--pixscale', '0.262', '--threads', '1', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 253.0437484055481 54.23361969915836 --width 281 --height 281 --pixscale 0.262 --threads 1 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0

Parsed RA,Dec 253.0437484055481 54.23361969915836
Starting process 48844 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 641 MB, VmSize: 641 MB, VmRSS: 99 MB, VmData: 128 MB, maxrss: 0.10766 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage fit_on_coadds
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-08-07T14:45:05.522718
Keeping 11 CCDs:
  k4m_160615_064734_ooi_zd_ls9-CCD3 z expnum 95405 exptime 80.105 propid 2016A-0453 seeing 0.94 MJD 57554.283 object MzLS_666483_z 
    mosaic/CP/V4.3/CP20160614/k4m_160615_064734_ooi_zd_ls9.fits.fz
  k4m_170412_112301_ooi_zd_ls9-CCD1 z expnum 121439 exptime 107.105 propid 2016A-0453 seeing 1.06 MJD 57855.474 object MzLS_667217_z 
    mosaic/CP/V4.3/CP20170411/k4m_170412_112301_ooi_zd_ls9.fits.fz
  k4m_170523_095715_ooi_zd_ls9-CCD4 z expnum 129693 exptime 84.101 propid 2016A-0453 seeing 1.19 MJD 57896.415 object MzLS_665745_z 
    mosaic/CP/V4.3/CP20170522/k4m_170523_095715_ooi_zd_ls9.fits.fz
  ksb_170520_101640_ooi_r_ls9-CCD4 r expnum 78930197 exptime 181.5 propid 2015A-0801 seeing 2.04 MJD 57893.428 object 32623 
    90prime/CP/V2.3/CP20170519/ksb_170520_101640_ooi_r_ls9.fits.fz
  ksb_170523_080943_ooi_g_ls9-CCD4 g expnum 78960126 exptime 100.18 propid 2015A-0801 seeing 1.45 MJD 57896.340 object 32623 
    90prime/CP/V2.3/CP20170522/ksb_170523_080943_ooi_g_ls9.fits.fz
  ksb_160712_052408_ooi_r_ls9-CCD2 r expnum 75810061 exptime 233.3 propid BASS seeing 1.73 MJD 57581.225 object 31542 
    90prime/CP/V2.3/CP20160711/ksb_160712_052408_ooi_r_ls9.fits.fz
  ksb_170327_113133_ooi_g_ls9-CCD2 g expnum 78390275 exptime 91.542 propid BASS seeing 1.41 MJD 57839.480 object 31542 
    90prime/CP/V2.3/CP20170326/ksb_170327_113133_ooi_g_ls9.fits.fz
  ksb_170501_090154_ooi_r_ls9-CCD3 r expnum 78740200 exptime 62.028 propid BASS seeing 1.12 MJD 57874.376 object 31553 
    90prime/CP/V2.3/CP20170430/ksb_170501_090154_ooi_r_ls9.fits.fz
  ksb_170523_081418_ooi_g_ls9-CCD3 g expnum 78960128 exptime 103.558 propid 2015A-0801 seeing 1.52 MJD 57896.343 object 31553 
    90prime/CP/V2.3/CP20170522/ksb_170523_081418_ooi_g_ls9.fits.fz
  ksb_160531_080046_ooi_r_ls9-CCD1 r expnum 75390152 exptime 65.0 propid BASS seeing 1.16 MJD 57539.334 object 31541 
    90prime/CP/V2.3/CP20160530/ksb_160531_080046_ooi_r_ls9.fits.fz
  ksb_170602_101151_ooi_g_ls9-CCD1 g expnum 79060155 exptime 124.85 propid 2015A-0801 seeing 1.53 MJD 57906.425 object 31541 
    90prime/CP/V2.3/CP20170601/ksb_170602_101151_ooi_g_ls9.fits.fz
mosaic get_fwhm: self.fwhm = 3.61
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3609 3891 and amp range 2006 4053 and image y 897 1180 and amp range 0 2039
Range in image: x 0 283 , y 0 284 (with image size 283 x 284)
dzp -0.00064781494 -> scaling image by 0.9994035183523652
mosaic get_fwhm: self.fwhm = 4.09
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 841 1131 and amp range 0 2005 and image y 1016 1307 and amp range 0 2039
Range in image: x 0 291 , y 0 292 (with image size 291 x 292)
dzp 0.007397243 -> scaling image by 1.006836374469654
mosaic get_fwhm: self.fwhm = 4.59
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2054 2345 and amp range 2048 4095 and image y 3243 3531 and amp range 2022 4060
Range in image: x 0 292 , y 0 289 (with image size 292 x 289)
dzp -0.02499645 -> scaling image by 0.9772404160512723
Faked up EXPNUM 78930197
Faked up EXPNUM 78930197
Faked up EXPNUM 78930197
90prime get_fwhm: self.fwhm = 4.49
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3959 4009 and amp range 2016 4031 and image y 15 79 and amp range 0 2047
Range in image: x 0 51 , y 0 65 (with image size 51 x 65)
dzp 0.02499258 -> scaling image by 1.0232859986636875
Faked up EXPNUM 78960126
Faked up EXPNUM 78960126
Faked up EXPNUM 78960126
90prime get_fwhm: self.fwhm = 3.19
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3966 4009 and amp range 2016 4031 and image y 15 67 and amp range 0 2047
Range in image: x 0 44 , y 0 53 (with image size 44 x 53)
dzp 0.016679538 -> scaling image by 1.0154810312115594
Faked up EXPNUM 75810061
Faked up EXPNUM 75810061
Faked up EXPNUM 75810061
90prime get_fwhm: self.fwhm = 3.81
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 15 121 and amp range 0 2015 and image y 3423 3587 and amp range 2048 4095
Range in image: x 0 107 , y 0 165 (with image size 107 x 165)
dzp 0.016961161 -> scaling image by 1.0157444649385616
Faked up EXPNUM 78390275
Faked up EXPNUM 78390275
Faked up EXPNUM 78390275
90prime get_fwhm: self.fwhm = 3.11
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 15 134 and amp range 0 2015 and image y 3425 3589 and amp range 2048 4095
Range in image: x 0 120 , y 0 165 (with image size 120 x 165)
dzp 0.051889952 -> scaling image by 1.0489528829333021
Faked up EXPNUM 78740200
Faked up EXPNUM 78740200
Faked up EXPNUM 78740200
90prime get_fwhm: self.fwhm = 2.47
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 186 352 and amp range 0 2015 and image y 2968 3132 and amp range 2048 4095
Range in image: x 0 167 , y 0 165 (with image size 167 x 165)
dzp -0.07725406 -> scaling image by 0.9313188157188347
Faked up EXPNUM 78960128
Faked up EXPNUM 78960128
Faked up EXPNUM 78960128
90prime get_fwhm: self.fwhm = 3.35
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 186 352 and amp range 0 2015 and image y 2963 3127 and amp range 2048 4095
Range in image: x 0 167 , y 0 165 (with image size 167 x 165)
dzp -0.045992933 -> scaling image by 0.958523636456738
Faked up EXPNUM 75390152
Faked up EXPNUM 75390152
Faked up EXPNUM 75390152
90prime get_fwhm: self.fwhm = 2.57
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2121 2287 and amp range 2016 4031 and image y 1444 1610 and amp range 0 2047
Range in image: x 0 167 , y 0 167 (with image size 167 x 167)
dzp -0.0041296566 -> scaling image by 0.9962036700684744
Faked up EXPNUM 79060155
Faked up EXPNUM 79060155
Faked up EXPNUM 79060155
90prime get_fwhm: self.fwhm = 3.37
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2135 2301 and amp range 2016 4031 and image y 1453 1618 and amp range 0 2047
Range in image: x 0 167 , y 0 166 (with image size 167 x 166)
dzp -0.011406615 -> scaling image by 0.9895491130016149
Stage tims : Wall: 8.97 s, CPU: 2.38 s, VmPeak: 860 MB, VmSize: 758 MB, VmRSS: 176 MB, VmData: 195 MB, maxrss: 0.179752 MB
Stage tims finished: 2020-08-07T14:45:14.494756
Resources for stage tims : Wall: 8.97 s, CPU: 2.38 s
Stage tims finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-tims.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-tims.p
Running stage refs
Running stage refs at 2020-08-07T14:45:14.513072
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-01171.fits
Matched 13 large galaxies to Gaia stars.
Matched 0 large galaxies to Tycho-2 stars.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/reference-custom-253043p54233.fits
Stage refs : Wall: 2.39 s, CPU: 2.15 s, VmPeak: 860 MB, VmSize: 782 MB, VmRSS: 200 MB, VmData: 219 MB, maxrss: 0.204356 MB
Stage refs finished: 2020-08-07T14:45:16.899827
Resources for stage refs : Wall: 2.39 s, CPU: 2.15 s
Stage refs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-refs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-refs.p
Running stage outliers
Running stage outliers at 2020-08-07T14:45:16.921486
Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/outlier-mask-custom-253043p54233.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/outliers-pre-custom-253043p54233.jpg
Image ksb_170523_081418_ooi_g_ls9-CCD3 g : masking 89 positive outlier pixels and 0 negative outlier pixels
Image ksb_170501_090154_ooi_r_ls9-CCD3 r : masking 125 positive outlier pixels and 0 negative outlier pixels
Image ksb_160531_080046_ooi_r_ls9-CCD1 r : masking 373 positive outlier pixels and 0 negative outlier pixels
Image k4m_160615_064734_ooi_zd_ls9-CCD3 z : masking 1775 positive outlier pixels and 0 negative outlier pixels
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/outlier-mask-custom-253043p54233.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/outliers-post-custom-253043p54233.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/outliers-masked-pos-custom-253043p54233.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/outliers-masked-neg-custom-253043p54233.jpg
Stage outliers : Wall: 1.09 s, CPU: 1.06 s, VmPeak: 860 MB, VmSize: 786 MB, VmRSS: 202 MB, VmData: 219 MB, maxrss: 0.206612 MB
Stage outliers finished: 2020-08-07T14:45:18.007250
Resources for stage outliers : Wall: 1.09 s, CPU: 1.06 s
Stage outliers finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-outliers.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-outliers.p
Running stage halos
Running stage halos at 2020-08-07T14:45:18.026440
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Stage halos : Wall: 0.26 s, CPU: 0.25 s, VmPeak: 860 MB, VmSize: 786 MB, VmRSS: 202 MB, VmData: 219 MB, maxrss: 0.206672 MB
Stage halos finished: 2020-08-07T14:45:18.284080
Resources for stage halos : Wall: 0.26 s, CPU: 0.25 s
Stage halos finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-halos.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-halos.p
Running stage fit_on_coadds
Running stage fit_on_coadds at 2020-08-07T14:45:18.307089
PSF sigmas in coadd pixels: ['2.35', '2.29', '2.46', '2.48']
Using average PSF sigma 2.395566358689102
Gaussian PSF norm 0.11775703509555277 vs pixelized 0.09646213
average tim pixel scale / coadd scale: 1.729534137354211
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-copsf-g.fits.fz
PSF sigmas in coadd pixels: ['3.31', '2.80', '1.82', '1.89']
Using average PSF sigma 2.454527133479732
Gaussian PSF norm 0.11492836560089613 vs pixelized 0.09748004
average tim pixel scale / coadd scale: 1.7294610056489392
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-copsf-r.fits.fz
PSF sigmas in coadd pixels: ['1.52', '1.72', '1.94']
Using average PSF sigma 1.7273379876689077
Gaussian PSF norm 0.16331186703916192 vs pixelized 0.13717055
average tim pixel scale / coadd scale: 0.9908615580390256
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-copsf-z.fits.fz
Stage fit_on_coadds : Wall: 0.26 s, CPU: 0.24 s, VmPeak: 860 MB, VmSize: 786 MB, VmRSS: 202 MB, VmData: 219 MB, maxrss: 0.206712 MB
Stage fit_on_coadds finished: 2020-08-07T14:45:18.562436
Resources for stage fit_on_coadds : Wall: 0.26 s, CPU: 0.24 s
Stage fit_on_coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-fit_on_coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-fit_on_coadds.p
Running stage srcs
Running stage srcs at 2020-08-07T14:45:18.571711
SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])]
Found 13 potential peaks
Of 13 potential peaks: 5 in veto map, 1 cut by saddle test, 0 cut by aper test, 7 kept
SED z : 7 new peaks
Found 18 potential peaks
Of 18 potential peaks: 14 in veto map, 3 cut by saddle test, 0 cut by aper test, 1 kept
SED r : 1 new peaks
Found 21 potential peaks
Of 21 potential peaks: 17 in veto map, 3 cut by saddle test, 0 cut by aper test, 1 kept
SED g : 1 new peaks
Found 25 potential peaks
Of 25 potential peaks: 20 in veto map, 2 cut by saddle test, 0 cut by aper test, 3 kept
SED Flat : 3 new peaks
Found 25 potential peaks
Of 25 potential peaks: 21 in veto map, 2 cut by saddle test, 0 cut by aper test, 2 kept
SED Red : 2 new peaks
Adding 14 new sources
Median coadd sky for g : 7.972446e-05
Median coadd sky for r : -8.704088e-05
Median coadd sky for z : 3.2420267e-05
Stage srcs : Wall: 0.47 s, CPU: 0.46 s, VmPeak: 860 MB, VmSize: 786 MB, VmRSS: 202 MB, VmData: 219 MB, maxrss: 0.207224 MB
Stage srcs finished: 2020-08-07T14:45:19.045118
Resources for stage srcs : Wall: 0.47 s, CPU: 0.46 s
Stage srcs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-srcs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-srcs.p
Running stage fitblobs
Running stage fitblobs at 2020-08-07T14:45:19.055086
Blob 1 of 15, id: 3, sources: 1, size: 92x158, npix 8954, brick X: 92,184, Y: 58,216, one pixel: 172 58
Known galaxy.  Initial shape: EllipseWithPriors(0.25): log r_e=1.78286, ee1=0.654341, ee2=0.360766
Blob 1 finished, total: Wall: 4.32 s, CPU: 4.28 s, VmPeak: 860 MB, VmSize: 795 MB, VmRSS: 207 MB, VmData: 219 MB, maxrss: 0.21202 MB
Blob 2 of 15, id: 10, sources: 3, size: 83x73, npix 3449, brick X: 197,280, Y: 202,275, one pixel: 253 202
Measuring 1 iterative sources
Blob 2 finished, total: Wall: 1.33 s, CPU: 1.33 s, VmPeak: 860 MB, VmSize: 795 MB, VmRSS: 207 MB, VmData: 219 MB, maxrss: 0.212272 MB
Blob 3 of 15, id: 2, sources: 2, size: 62x53, npix 2089, brick X: 65,127, Y: 28,81, one pixel: 88 28
Blob 3 finished, total: Wall: 1.07 s, CPU: 1.07 s, VmPeak: 860 MB, VmSize: 795 MB, VmRSS: 207 MB, VmData: 219 MB, maxrss: 0.212272 MB
Blob 4 of 15, id: 12, sources: 3, size: 55x54, npix 1786, brick X: 30,85, Y: 227,281, one pixel: 57 227
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.05403, 54.24420), pm (-4.8, -13.9), parallax 0.167 with NanoMaggies: g=18.5, r=(flux -14.1), z=17.7
Reset brightness to NanoMaggies: g=16.4, r=15.8, z=15.6
Blob 4 finished, total: Wall: 0.43 s, CPU: 0.43 s, VmPeak: 860 MB, VmSize: 795 MB, VmRSS: 207 MB, VmData: 219 MB, maxrss: 0.212272 MB
Blob 5 of 15, id: 9, sources: 2, size: 28x49, npix 958, brick X: 0,28, Y: 177,226, one pixel: 5 177
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.06134, 54.23807), pm (26.3, 0.7), parallax 1.862 with NanoMaggies: g=18.1, r=17.3, z=15.7
Reset brightness to NanoMaggies: g=17.5, r=16.3, z=15.6
Blob 5 finished, total: Wall: 1.00 s, CPU: 1.00 s, VmPeak: 860 MB, VmSize: 795 MB, VmRSS: 207 MB, VmData: 219 MB, maxrss: 0.212272 MB
Blob 6 of 15, id: 6, sources: 1, size: 28x28, npix 549, brick X: 36,64, Y: 134,162, one pixel: 49 134
Blob 6 finished, total: Wall: 0.48 s, CPU: 0.48 s, VmPeak: 860 MB, VmSize: 795 MB, VmRSS: 207 MB, VmData: 219 MB, maxrss: 0.212272 MB
Blob 7 of 15, id: 14, sources: 1, size: 30x21, npix 452, brick X: 193,223, Y: 260,281, one pixel: 205 260
Blob 7 finished, total: Wall: 0.72 s, CPU: 0.72 s, VmPeak: 860 MB, VmSize: 795 MB, VmRSS: 207 MB, VmData: 219 MB, maxrss: 0.212272 MB
Blob 8 of 15, id: 4, sources: 1, size: 25x26, npix 423, brick X: 237,262, Y: 69,95, one pixel: 249 69
Blob 8 finished, total: Wall: 0.31 s, CPU: 0.31 s, VmPeak: 860 MB, VmSize: 795 MB, VmRSS: 207 MB, VmData: 219 MB, maxrss: 0.212272 MB
Blob 9 of 15, id: 0, sources: 2, size: 27x15, npix 289, brick X: 130,157, Y: 0,15, one pixel: 131 0
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (253.04343, 54.22332), pm (-5.4, -5.8), parallax 1.390 with NanoMaggies: g=24.4, r=24.4, z=20.5
Reset brightness to NanoMaggies: g=21.8, r=20.4, z=19.3
Blob 9 finished, total: Wall: 0.08 s, CPU: 0.08 s, VmPeak: 860 MB, VmSize: 795 MB, VmRSS: 207 MB, VmData: 219 MB, maxrss: 0.212272 MB
Blob 10 of 15, id: 5, sources: 1, size: 19x20, npix 236, brick X: 194,213, Y: 74,94, one pixel: 202 74
Blob 10 finished, total: Wall: 0.13 s, CPU: 0.13 s, VmPeak: 860 MB, VmSize: 795 MB, VmRSS: 207 MB, VmData: 219 MB, maxrss: 0.212272 MB
Blob 11 of 15, id: 11, sources: 1, size: 20x20, npix 219, brick X: 152,172, Y: 204,224, one pixel: 161 204
Blob 11 finished, total: Wall: 0.12 s, CPU: 0.12 s, VmPeak: 860 MB, VmSize: 795 MB, VmRSS: 207 MB, VmData: 219 MB, maxrss: 0.212272 MB
Blob 12 of 15, id: 1, sources: 1, size: 19x19, npix 217, brick X: 258,277, Y: 20,39, one pixel: 266 20
Blob 12 finished, total: Wall: 0.04 s, CPU: 0.04 s, VmPeak: 860 MB, VmSize: 795 MB, VmRSS: 207 MB, VmData: 219 MB, maxrss: 0.212272 MB
Blob 13 of 15, id: 8, sources: 1, size: 19x19, npix 208, brick X: 180,199, Y: 172,191, one pixel: 188 172
Blob 13 finished, total: Wall: 0.04 s, CPU: 0.04 s, VmPeak: 860 MB, VmSize: 795 MB, VmRSS: 207 MB, VmData: 219 MB, maxrss: 0.212272 MB
Blob 14 of 15, id: 13, sources: 1, size: 19x19, npix 199, brick X: 119,138, Y: 232,251, one pixel: 127 232
Blob 14 finished, total: Wall: 0.16 s, CPU: 0.16 s, VmPeak: 860 MB, VmSize: 795 MB, VmRSS: 207 MB, VmData: 219 MB, maxrss: 0.212272 MB
Blob 15 of 15, id: 7, sources: 1, size: 17x18, npix 162, brick X: 70,87, Y: 149,167, one pixel: 78 149
Blob 15 finished, total: Wall: 0.04 s, CPU: 0.04 s, VmPeak: 860 MB, VmSize: 795 MB, VmRSS: 207 MB, VmData: 219 MB, maxrss: 0.212272 MB
Old catalog: 31
New catalog: 18
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/blobs-custom-253043p54233.fits.gz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/metrics/cus/all-models-custom-253043p54233.fits
Stage fitblobs : Wall: 12.56 s, CPU: 10.66 s, VmPeak: 860 MB, VmSize: 795 MB, VmRSS: 207 MB, VmData: 219 MB, maxrss: 0.212272 MB
Stage fitblobs finished: 2020-08-07T14:45:31.615207
Resources for stage fitblobs : Wall: 12.56 s, CPU: 10.66 s
Stage fitblobs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-fitblobs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-fitblobs.p
Running stage coadds
Running stage coadds at 2020-08-07T14:45:31.627007
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-ccds.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-image-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-invvar-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-nexp-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-depth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-galdepth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-psfsize-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-model-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-chi2-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-blobmodel-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-image-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-invvar-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-nexp-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-depth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-galdepth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-psfsize-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-model-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-chi2-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-blobmodel-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-image-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-invvar-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-nexp-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-depth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-galdepth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-psfsize-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-model-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-chi2-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-blobmodel-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-depth.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/tmp-legacysurvey-custom-253043p54233-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/tmp-legacysurvey-custom-253043p54233-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/tmp-legacysurvey-custom-253043p54233-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/tmp-legacysurvey-custom-253043p54233-resid.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-resid.jpg
Stage coadds : Wall: 3.12 s, CPU: 0.86 s, VmPeak: 860 MB, VmSize: 795 MB, VmRSS: 208 MB, VmData: 219 MB, maxrss: 0.212588 MB
Stage coadds finished: 2020-08-07T14:45:34.745954
Resources for stage coadds : Wall: 3.12 s, CPU: 0.86 s
Stage coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-coadds.p
Running stage wise_forced
Running stage wise_forced at 2020-08-07T14:45:34.756588
Cut to 1 unWISE tiles
Reading WISE tile 2527p545 band 1
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w1-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w1-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w1-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w1-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.16 s, CPU: 0.08 s, VmPeak: 860 MB, VmSize: 847 MB, VmRSS: 203 MB, VmData: 210 MB, maxrss: 0.26962 MB
Reading WISE tile 2527p545 band 2
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w2-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w2-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w2-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w2-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.02 s, CPU: 0.02 s, VmPeak: 905 MB, VmSize: 847 MB, VmRSS: 204 MB, VmData: 210 MB, maxrss: 0.26962 MB
Reading WISE tile 2527p545 band 3
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w3-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.02 s, CPU: 0.02 s, VmPeak: 905 MB, VmSize: 847 MB, VmRSS: 204 MB, VmData: 210 MB, maxrss: 0.26962 MB
Reading WISE tile 2527p545 band 4
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/252/2527p545/unwise-2527p545-w4-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.02 s, CPU: 0.02 s, VmPeak: 905 MB, VmSize: 892 MB, VmRSS: 248 MB, VmData: 255 MB, maxrss: 0.26962 MB
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-image-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-model-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-invvar-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-image-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-model-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-invvar-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-image-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-model-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-invvar-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-image-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-model-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-invvar-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/tmp-legacysurvey-custom-253043p54233-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/tmp-legacysurvey-custom-253043p54233-wisemodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-wisemodel.jpg
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Stage wise_forced : Wall: 4.07 s, CPU: 3.25 s, VmPeak: 905 MB, VmSize: 892 MB, VmRSS: 249 MB, VmData: 255 MB, maxrss: 0.26962 MB
Stage wise_forced finished: 2020-08-07T14:45:38.826296
Resources for stage wise_forced : Wall: 4.07 s, CPU: 3.25 s
Stage wise_forced finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-wise_forced.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-wise_forced.p
Running stage writecat
Running stage writecat at 2020-08-07T14:45:38.844268
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/coadd/cus/custom-253043p54233/legacysurvey-custom-253043p54233-maskbits.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/tractor-i/cus/tractor-custom-253043p54233.fits
Reading SFD maps...
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/tractor/cus/tractor-custom-253043p54233.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/tractor/cus/brick-custom-253043p54233.sha256sum
Stage writecat : Wall: 0.71 s, CPU: 0.47 s, VmPeak: 1024 MB, VmSize: 896 MB, VmRSS: 250 MB, VmData: 255 MB, maxrss: 0.387424 MB
Stage writecat finished: 2020-08-07T14:45:39.557078
Resources for stage writecat : Wall: 0.71 s, CPU: 0.47 s
Stage writecat finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-writecat.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/253/PGC2463611/PGC2463611-largegalaxy-writecat.p
All done: Wall: 34.05 s, CPU: 21.89 s
Finished galaxy PGC2463611 in 0.598 minutes.