Started working on galaxy PGC2502231 at Sun Jun  7 18:35:38 2020
Bands touching this brick, g r z
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 166.94296049999997 55.4426106 --width 273 --height 273 --pixscale 0.262 --threads 4 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/PGC2502231-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/PGC2502231-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 

runbrick.py starting at 2020-06-07T18:35:39.445046
legacypipe git version: DR9.5.2
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '166.94296049999997', '55.4426106', '--width', '273', '--height', '273', '--pixscale', '0.262', '--threads', '4', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/PGC2502231-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/PGC2502231-largegalaxy-%%(stage)s.p', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 166.94296049999997 55.4426106 --width 273 --height 273 --pixscale 0.262 --threads 4 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/PGC2502231-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/PGC2502231-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0

Parsed RA,Dec 166.94296049999997 55.4426106
Starting process 32207 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 632 MB, VmSize: 632 MB, VmRSS: 56 MB, VmData: 127 MB, maxrss: 0.057308 MB
Starting process 32208 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 632 MB, VmSize: 632 MB, VmRSS: 56 MB, VmData: 127 MB, maxrss: 0.057372 MB
Starting process 32209 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 632 MB, VmSize: 632 MB, VmRSS: 56 MB, VmData: 127 MB, maxrss: 0.05738 MB
Starting process 32210 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 632 MB, VmSize: 632 MB, VmRSS: 56 MB, VmData: 127 MB, maxrss: 0.057376 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage fit_on_coadds
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-06-07T18:35:40.089375
Keeping 9 CCDs:
   k4m_170411_080824_ooi_zd_ls9.fits.fz k4m_170411_080824_ooi_zd_ls9-CCD4 z exptime 106.105 propid 2016A-0453 seeing 0.80 MJD 57854.339 object MzLS_670746_z
   k4m_160520_035838_ooi_zd_ls9.fits.fz k4m_160520_035838_ooi_zd_ls9-CCD1 z exptime 113.103 propid 2016A-0453 seeing 1.29 MJD 57528.166 object MzLS_671459_z
   k4m_170528_042331_ooi_zd_ls9.fits.fz k4m_170528_042331_ooi_zd_ls9-CCD4 z exptime 83.103 propid 2016A-0453 seeing 0.96 MJD 57901.183 object MzLS_670030_z
   ksb_170521_052821_ooi_g_ls9.fits.fz ksb_170521_052821_ooi_g_ls9-CCD2 g exptime 119.387 propid 2015A-0801 seeing 1.83 MJD 57894.228 object 33203
   ksb_170325_072138_ooi_r_ls9.fits.fz ksb_170325_072138_ooi_r_ls9-CCD1 r exptime 91.723 propid BASS seeing 2.20 MJD 57837.307 object 32122
   ksb_170131_101922_ooi_g_ls9.fits.fz ksb_170131_101922_ooi_g_ls9-CCD1 g exptime 79.3 propid BASS seeing 1.82 MJD 57784.430 object 32122
   ksb_170403_061551_ooi_r_ls9.fits.fz ksb_170403_061551_ooi_r_ls9-CCD3 r exptime 137.459 propid BASS seeing 1.56 MJD 57846.261 object 32131
   ksb_160502_042113_ooi_g_ls9.fits.fz ksb_160502_042113_ooi_g_ls9-CCD3 g exptime 83.4 propid 2015A-0801 seeing 1.43 MJD 57510.181 object 32131
   ksb_170430_043946_ooi_r_ls9.fits.fz ksb_170430_043946_ooi_r_ls9-CCD2 r exptime 86.089 propid BASS seeing 1.53 MJD 57873.194 object 33203
mosaic get_fwhm: self.fwhm = 4.95
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 1392 1668 and amp range 0 2005 and image y 3007 3284 and amp range 2040 4078
Range in image: x 0 277 , y 0 278 (with image size 277 x 278)
dzp -0.005990375 -> scaling image by 0.994497853444703
mosaic get_fwhm: self.fwhm = 3.07
Swapping in SKY_JOHN values for 1 splinesky cells; 96.31473 -> 96.27921
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 7 88 and amp range 0 2047 and image y 1025 1300 and amp range 0 2021
Range in image: x 0 82 , y 0 276 (with image size 82 x 276)
dzp -0.018417627 -> scaling image by 0.9831798050134541
Faked up EXPNUM 78940070
Faked up EXPNUM 78940070
Faked up EXPNUM 78940070
90prime get_fwhm: self.fwhm = 4.03
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2266 2425 and amp range 2016 4031 and image y 2326 2486 and amp range 2048 4095
Range in image: x 0 160 , y 0 161 (with image size 160 x 161)
dzp 0.010008041 -> scaling image by 1.0092603604948323
mosaic get_fwhm: self.fwhm = 3.71
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2552 2836 and amp range 2048 4095 and image y 3318 3600 and amp range 2022 4060
Range in image: x 0 285 , y 0 283 (with image size 285 x 283)
dzp -0.02499645 -> scaling image by 0.9772404160512723
Faked up EXPNUM 78370123
Faked up EXPNUM 78370123
Faked up EXPNUM 78370123
90prime get_fwhm: self.fwhm = 4.85
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2465 2625 and amp range 2016 4031 and image y 1286 1447 and amp range 0 2047
Range in image: x 0 161 , y 0 162 (with image size 161 x 162)
dzp -0.006547878 -> scaling image by 0.9939873303458991
Faked up EXPNUM 77840191
Faked up EXPNUM 77840191
Faked up EXPNUM 77840191
90prime get_fwhm: self.fwhm = 4.01
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2468 2629 and amp range 2016 4031 and image y 1273 1433 and amp range 0 2047
Range in image: x 0 162 , y 0 161 (with image size 162 x 161)
dzp -0.011406615 -> scaling image by 0.9895491130016149
Faked up EXPNUM 78460096
Faked up EXPNUM 78460096
Faked up EXPNUM 78460096
90prime get_fwhm: self.fwhm = 3.45
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 512 673 and amp range 0 2015 and image y 2769 2929 and amp range 2048 4095
Range in image: x 0 162 , y 0 161 (with image size 162 x 161)
dzp -0.07725406 -> scaling image by 0.9313188157188347
Faked up EXPNUM 78730064
Faked up EXPNUM 78730064
Faked up EXPNUM 78730064
90prime get_fwhm: self.fwhm = 3.37
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2267 2427 and amp range 2016 4031 and image y 2346 2506 and amp range 2048 4095
Range in image: x 0 161 , y 0 161 (with image size 161 x 161)
dzp 0.021366077 -> scaling image by 1.0198737893800265
Faked up EXPNUM 75100053
Faked up EXPNUM 75100053
Faked up EXPNUM 75100053
90prime get_fwhm: self.fwhm = 3.15
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 506 667 and amp range 0 2015 and image y 2742 2901 and amp range 2048 4095
Range in image: x 0 162 , y 0 160 (with image size 162 x 160)
dzp -0.0045482004 -> scaling image by 0.9958197144381092
Stage tims : Wall: 6.63 s, CPU: 0.97 s, VmPeak: 941 MB, VmSize: 941 MB, VmRSS: 171 MB, VmData: 204 MB, maxrss: 0.568268 MB
Stage tims finished: 2020-06-07T18:35:46.723362
Resources for stage tims : 3.565 s worker CPU, 14.703 s worker Wall, Wall: 6.635 s, Cores in use: 0.68, Total efficiency (on 4 cores): 17.1 %
Stage tims finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/PGC2502231-largegalaxy-tims.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/PGC2502231-largegalaxy-tims.p
Running stage refs
Running stage refs at 2020-06-07T18:35:46.743636
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-01148.fits
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-01054.fits
Matched 15 large galaxies to Gaia stars.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/metrics/cus/reference-custom-166942p55442.fits
Stage refs : Wall: 2.13 s, CPU: 1.45 s, VmPeak: 972 MB, VmSize: 954 MB, VmRSS: 184 MB, VmData: 216 MB, maxrss: 0.568268 MB
Stage refs finished: 2020-06-07T18:35:48.877439
Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 2.135 s, Cores in use: 0.68, Total efficiency (on 4 cores): 16.9 %
Stage refs finished
Running stage outliers
Running stage outliers at 2020-06-07T18:35:48.879113
Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/metrics/cus/outlier-mask-custom-166942p55442.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/metrics/cus/outliers-pre-custom-166942p55442.jpg
Image k4m_170411_080824_ooi_zd_ls9-CCD4 z : masking 0 positive outlier pixels and 1664 negative outlier pixels
Image k4m_160520_035838_ooi_zd_ls9-CCD1 z : masking 143 positive outlier pixels and 0 negative outlier pixels
Image k4m_170528_042331_ooi_zd_ls9-CCD4 z : masking 1703 positive outlier pixels and 0 negative outlier pixels
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/metrics/cus/outlier-mask-custom-166942p55442.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/metrics/cus/outliers-post-custom-166942p55442.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/metrics/cus/outliers-masked-pos-custom-166942p55442.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/metrics/cus/outliers-masked-neg-custom-166942p55442.jpg
Stage outliers : Wall: 1.02 s, CPU: 0.71 s, VmPeak: 972 MB, VmSize: 963 MB, VmRSS: 199 MB, VmData: 229 MB, maxrss: 0.568268 MB
Stage outliers finished: 2020-06-07T18:35:49.897638
Resources for stage outliers : 0.963 s worker CPU, 1.012 s worker Wall, Wall: 1.020 s, Cores in use: 1.64, Total efficiency (on 4 cores): 41.0 %
Stage outliers finished
Running stage halos
Running stage halos at 2020-06-07T18:35:49.899605
Stage halos : Wall: 0.24 s, CPU: 0.23 s, VmPeak: 972 MB, VmSize: 963 MB, VmRSS: 200 MB, VmData: 229 MB, maxrss: 0.568268 MB
Stage halos finished: 2020-06-07T18:35:50.136547
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage halos : 0.000 s worker CPU, 0.001 s worker Wall, Wall: 0.238 s, Cores in use: 0.97, Total efficiency (on 4 cores): 24.3 %
Stage halos finished
Running stage fit_on_coadds
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Running stage fit_on_coadds at 2020-06-07T18:35:50.138591
PSF sigmas in coadd pixels: ['2.97', '2.95', '2.32']
Using average PSF sigma 2.7454065999014596
Gaussian PSF norm 0.1027515530063938 vs pixelized 0.08057035
average tim pixel scale / coadd scale: 1.7296966300381502
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-copsf-g.fits.fz
PSF sigmas in coadd pixels: ['3.57', '2.54', '2.48']
Using average PSF sigma 2.8631351728065795
Gaussian PSF norm 0.09852653638331564 vs pixelized 0.09974354
average tim pixel scale / coadd scale: 1.7297048279873402
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-copsf-r.fits.fz
PSF sigmas in coadd pixels: ['1.29', '2.09', '1.56']
Using average PSF sigma 1.6486302723487762
Gaussian PSF norm 0.171108584201831 vs pixelized 0.14038187
average tim pixel scale / coadd scale: 0.9908657236636139
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-copsf-z.fits.fz
Stage fit_on_coadds : Wall: 0.25 s, CPU: 0.20 s, VmPeak: 972 MB, VmSize: 963 MB, VmRSS: 201 MB, VmData: 230 MB, maxrss: 0.568268 MB
Stage fit_on_coadds finished: 2020-06-07T18:35:50.389279
Resources for stage fit_on_coadds : 0.240 s worker CPU, 0.244 s worker Wall, Wall: 0.252 s, Cores in use: 1.74, Total efficiency (on 4 cores): 43.5 %
Stage fit_on_coadds finished
Running stage srcs
Running stage srcs at 2020-06-07T18:35:50.392216
SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])]
Found 26 potential peaks
Of 26 potential peaks: 12 in veto map, 2 cut by saddle test, 0 cut by aper test, 12 kept
SED z : 12 new peaks
Found 24 potential peaks
Of 24 potential peaks: 16 in veto map, 3 cut by saddle test, 0 cut by aper test, 5 kept
SED r : 5 new peaks
Found 21 potential peaks
Of 21 potential peaks: 19 in veto map, 2 cut by saddle test, 0 cut by aper test, 0 kept
SED g : 0 new peaks
Found 35 potential peaks
Of 35 potential peaks: 31 in veto map, 4 cut by saddle test, 0 cut by aper test, 0 kept
SED Flat : 0 new peaks
Found 36 potential peaks
Of 36 potential peaks: 32 in veto map, 3 cut by saddle test, 0 cut by aper test, 1 kept
SED Red : 1 new peaks
Adding 18 new sources
Median coadd sky for g : 0.00018928244
Median coadd sky for r : 0.00032533327
Median coadd sky for z : -0.00032674178
Stage srcs : Wall: 0.58 s, CPU: 0.52 s, VmPeak: 972 MB, VmSize: 963 MB, VmRSS: 202 MB, VmData: 230 MB, maxrss: 0.568268 MB
Stage srcs finished: 2020-06-07T18:35:50.968798
Resources for stage srcs : 0.135 s worker CPU, 0.149 s worker Wall, Wall: 0.578 s, Cores in use: 1.14, Total efficiency (on 4 cores): 28.4 %
Stage srcs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/PGC2502231-largegalaxy-srcs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/PGC2502231-largegalaxy-srcs.p
Running stage fitblobs
Running stage fitblobs at 2020-06-07T18:35:50.981395
Blob 1 of 13, id: 3, sources: 4, size: 163x184, npix 14837, brick X: 66,229, Y: 10,194, one pixel: 162 10
Blob 2 of 13, id: 10, sources: 3, size: 79x53, npix 3069, brick X: 21,100, Y: 169,222, one pixel: 37 169
Blob 3 of 13, id: 1, sources: 2, size: 48x39, npix 1144, brick X: 85,133, Y: 0,39, one pixel: 115 0
Blob 4 of 13, id: 5, sources: 2, size: 39x36, npix 900, brick X: 207,246, Y: 27,63, one pixel: 218 27
Blob 5 of 13, id: 7, sources: 1, size: 36x35, npix 864, brick X: 2,38, Y: 53,88, one pixel: 16 53
Blob 4 finished, total: Wall: 1.15 s, CPU: 1.15 s, VmPeak: 805 MB, VmSize: 713 MB, VmRSS: 139 MB, VmData: 178 MB, maxrss: 0.1427 MB
Blob 6 of 13, id: 8, sources: 1, size: 35x34, npix 808, brick X: 228,263, Y: 90,124, one pixel: 241 90
Blob 3 finished, total: Wall: 1.31 s, CPU: 1.31 s, VmPeak: 803 MB, VmSize: 713 MB, VmRSS: 138 MB, VmData: 178 MB, maxrss: 0.140896 MB
Blob 7 of 13, id: 2, sources: 1, size: 42x23, npix 733, brick X: 225,267, Y: 0,23, one pixel: 229 0
Blob 5 finished, total: Wall: 0.59 s, CPU: 0.59 s, VmPeak: 805 MB, VmSize: 713 MB, VmRSS: 139 MB, VmData: 178 MB, maxrss: 0.1427 MB
Blob 8 of 13, id: 6, sources: 2, size: 37x34, npix 672, brick X: 182,219, Y: 42,76, one pixel: 192 42
Blob 2 finished, total: Wall: 1.86 s, CPU: 1.86 s, VmPeak: 804 MB, VmSize: 713 MB, VmRSS: 139 MB, VmData: 178 MB, maxrss: 0.142328 MB
Blob 9 of 13, id: 12, sources: 1, size: 30x27, npix 500, brick X: 185,215, Y: 225,252, one pixel: 196 225
Blob 6 finished, total: Wall: 0.60 s, CPU: 0.60 s, VmPeak: 803 MB, VmSize: 713 MB, VmRSS: 138 MB, VmData: 178 MB, maxrss: 0.140896 MB
Blob 10 of 13, id: 11, sources: 1, size: 24x25, npix 377, brick X: 249,273, Y: 202,227, one pixel: 261 202
Blob 9 finished, total: Wall: 0.54 s, CPU: 0.54 s, VmPeak: 803 MB, VmSize: 713 MB, VmRSS: 138 MB, VmData: 178 MB, maxrss: 0.140896 MB
Blob 11 of 13, id: 4, sources: 1, size: 22x22, npix 301, brick X: 51,73, Y: 25,47, one pixel: 59 25
Blob 12 of 13, id: 0, sources: 1, size: 20x19, npix 227, brick X: 49,69, Y: 0,19, one pixel: 57 0
Blob 7 finished, total: Wall: 0.86 s, CPU: 0.86 s, VmPeak: 805 MB, VmSize: 713 MB, VmRSS: 139 MB, VmData: 178 MB, maxrss: 0.1427 MB
Blob 13 of 13, id: 9, sources: 1, size: 14x19, npix 165, brick X: 259,273, Y: 165,184, one pixel: 267 165
Blob 10 finished, total: Wall: 0.25 s, CPU: 0.25 s, VmPeak: 803 MB, VmSize: 713 MB, VmRSS: 138 MB, VmData: 178 MB, maxrss: 0.140896 MB
Blob 8 finished, total: Wall: 1.32 s, CPU: 1.32 s, VmPeak: 804 MB, VmSize: 713 MB, VmRSS: 139 MB, VmData: 178 MB, maxrss: 0.142476 MB
Blob 11 finished, total: Wall: 0.56 s, CPU: 0.56 s, VmPeak: 805 MB, VmSize: 713 MB, VmRSS: 140 MB, VmData: 178 MB, maxrss: 0.143148 MB
Blob 12 finished, total: Wall: 0.50 s, CPU: 0.50 s, VmPeak: 803 MB, VmSize: 713 MB, VmRSS: 138 MB, VmData: 178 MB, maxrss: 0.140896 MB
Blob 13 finished, total: Wall: 0.52 s, CPU: 0.52 s, VmPeak: 804 MB, VmSize: 713 MB, VmRSS: 139 MB, VmData: 178 MB, maxrss: 0.142476 MB
Known galaxy.  Initial shape: EllipseWithPriors(0.25): log r_e=1.01403, ee1=-0.152411, ee2=0.0323287
Measuring 3 iterative sources
Blob 1 finished, total: Wall: 10.55 s, CPU: 10.55 s, VmPeak: 803 MB, VmSize: 713 MB, VmRSS: 142 MB, VmData: 179 MB, maxrss: 0.145532 MB
Old catalog: 23
New catalog: 22
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/metrics/cus/blobs-custom-166942p55442.fits.gz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/metrics/cus/all-models-custom-166942p55442.fits
Stage fitblobs : Wall: 10.89 s, CPU: 0.34 s, VmPeak: 972 MB, VmSize: 963 MB, VmRSS: 202 MB, VmData: 230 MB, maxrss: 0.568268 MB
Stage fitblobs finished: 2020-06-07T18:36:01.873024
Resources for stage fitblobs : 20.759 s worker CPU, 20.791 s worker Wall, Wall: 10.893 s, Cores in use: 1.94, Total efficiency (on 4 cores): 48.4 %
Stage fitblobs finished
Running stage coadds
Running stage coadds at 2020-06-07T18:36:01.874955
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-ccds.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-image-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-invvar-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-nexp-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-depth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-galdepth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-psfsize-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-model-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-chi2-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-blobmodel-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-image-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-invvar-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-nexp-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-depth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-galdepth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-psfsize-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-model-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-chi2-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-blobmodel-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-image-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-invvar-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-nexp-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-depth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-galdepth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-psfsize-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-model-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-chi2-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-blobmodel-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-depth.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/tmp-legacysurvey-custom-166942p55442-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/tmp-legacysurvey-custom-166942p55442-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/tmp-legacysurvey-custom-166942p55442-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/tmp-legacysurvey-custom-166942p55442-resid.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-resid.jpg
Stage coadds : Wall: 10.32 s, CPU: 0.59 s, VmPeak: 972 MB, VmSize: 963 MB, VmRSS: 203 MB, VmData: 230 MB, maxrss: 0.568268 MB
Stage coadds finished: 2020-06-07T18:36:12.190071
Resources for stage coadds : 1.940 s worker CPU, 2.226 s worker Wall, Wall: 10.316 s, Cores in use: 0.25, Total efficiency (on 4 cores): 6.1 %
Stage coadds finished
Running stage wise_forced
Running stage wise_forced at 2020-06-07T18:36:12.191887
Cut to 1 unWISE tiles
Reading WISE tile 1679p560 band 1
Reading WISE tile 1679p560 band 2
Reading WISE tile 1679p560 band 4
Reading WISE tile 1679p560 band 3
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/167/1679p560/unwise-1679p560-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/167/1679p560/unwise-1679p560-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/167/1679p560/unwise-1679p560-w4-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/167/1679p560/unwise-1679p560-w4-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/167/1679p560/unwise-1679p560-w4-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.07 s, CPU: 0.07 s, VmPeak: 804 MB, VmSize: 794 MB, VmRSS: 146 MB, VmData: 176 MB, maxrss: 0.188948 MB
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/167/1679p560/unwise-1679p560-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/167/1679p560/unwise-1679p560-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/167/1679p560/unwise-1679p560-w3-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/167/1679p560/unwise-1679p560-w3-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/167/1679p560/unwise-1679p560-w3-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.06 s, CPU: 0.06 s, VmPeak: 805 MB, VmSize: 792 MB, VmRSS: 144 MB, VmData: 173 MB, maxrss: 0.208864 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/167/1679p560/unwise-1679p560-w2-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/167/1679p560/unwise-1679p560-w2-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/167/1679p560/unwise-1679p560-w2-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/167/1679p560/unwise-1679p560-w2-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.04 s, CPU: 0.04 s, VmPeak: 803 MB, VmSize: 788 MB, VmRSS: 140 MB, VmData: 170 MB, maxrss: 0.204564 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/167/1679p560/unwise-1679p560-w1-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/167/1679p560/unwise-1679p560-w1-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/167/1679p560/unwise-1679p560-w1-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/167/1679p560/unwise-1679p560-w1-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.04 s, CPU: 0.04 s, VmPeak: 803 MB, VmSize: 792 MB, VmRSS: 145 MB, VmData: 173 MB, maxrss: 0.209256 MB
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-image-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-model-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-invvar-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-image-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-model-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-invvar-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-image-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-model-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-invvar-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-image-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-model-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-invvar-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/tmp-legacysurvey-custom-166942p55442-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/tmp-legacysurvey-custom-166942p55442-wisemodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-wisemodel.jpg
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Stage wise_forced : Wall: 2.11 s, CPU: 0.45 s, VmPeak: 972 MB, VmSize: 963 MB, VmRSS: 203 MB, VmData: 230 MB, maxrss: 0.568268 MB
Stage wise_forced finished: 2020-06-07T18:36:14.298385
Resources for stage wise_forced : 4.586 s worker CPU, 5.280 s worker Wall, Wall: 2.108 s, Cores in use: 2.39, Total efficiency (on 4 cores): 59.8 %
Stage wise_forced finished
Running stage writecat
Running stage writecat at 2020-06-07T18:36:14.300195
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/coadd/cus/custom-166942p55442/legacysurvey-custom-166942p55442-maskbits.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/tractor-i/cus/tractor-custom-166942p55442.fits
Reading SFD maps...
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/tractor/cus/tractor-custom-166942p55442.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/166/PGC2502231/tractor/cus/brick-custom-166942p55442.sha256sum
Stage writecat : Wall: 0.74 s, CPU: 0.48 s, VmPeak: 1095 MB, VmSize: 967 MB, VmRSS: 205 MB, VmData: 230 MB, maxrss: 0.568268 MB
Stage writecat finished: 2020-06-07T18:36:15.038118
Resources for stage writecat : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 0.739 s, Cores in use: 0.65, Total efficiency (on 4 cores): 16.2 %
Stage writecat finished
All done: 32.190 s worker CPU, 44.408 s worker Wall, Wall: 34.954 s, Cores in use: 1.09, Total efficiency (on 4 cores): 27.3 %
Finished galaxy PGC2502231 in 0.617 minutes.