Started working on galaxy PGC1401624 at Sat Jun 6 15:21:18 2020 Bands touching this brick, g r z python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 167.710683 11.968089599999999 --width 233 --height 233 --pixscale 0.262 --threads 4 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run south --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/PGC1401624-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/PGC1401624-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 runbrick.py starting at 2020-06-06T15:21:19.551799 legacypipe git version: DR9.5.2 Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '167.710683', '11.968089599999999', '--width', '233', '--height', '233', '--pixscale', '0.262', '--threads', '4', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'south', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/PGC1401624-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/PGC1401624-largegalaxy-%%(stage)s.p', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 167.710683 11.968089599999999 --width 233 --height 233 --pixscale 0.262 --threads 4 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run south --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/PGC1401624-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/PGC1401624-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 Parsed RA,Dec 167.710683 11.968089599999999 Starting process 25573 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 632 MB, VmSize: 632 MB, VmRSS: 56 MB, VmData: 127 MB, maxrss: 0.057304 MB Starting process 25574 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 632 MB, VmSize: 632 MB, VmRSS: 56 MB, VmData: 127 MB, maxrss: 0.057304 MB Starting process 25575 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 632 MB, VmSize: 632 MB, VmRSS: 56 MB, VmData: 127 MB, maxrss: 0.057304 MB Starting process 25576 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 632 MB, VmSize: 632 MB, VmRSS: 56 MB, VmData: 127 MB, maxrss: 0.057308 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage fit_on_coadds Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-06-06T15:21:19.912391 Keeping 10 CCDs: c4d_160117_081507_ooi_g_ls9.fits.fz c4d_160117_081507_ooi_g_ls9-S16 g exptime 65.0 propid 2014B-0404 seeing 1.30 MJD 57404.342 object DECaLS_3374_g c4d_160117_081329_ooi_r_ls9.fits.fz c4d_160117_081329_ooi_r_ls9-S16 r exptime 40.0 propid 2014B-0404 seeing 1.09 MJD 57404.342 object DECaLS_3374_r c4d_190306_051857_ooi_g_ls9.fits.fz c4d_190306_051857_ooi_g_ls9-S16 g exptime 81.0 propid 2014B-0404 seeing 1.20 MJD 58548.221 object DECaLS_3374_g c4d_160110_083003_ooi_z_ls9.fits.fz c4d_160110_083003_ooi_z_ls9-S16 z exptime 87.0 propid 2014B-0404 seeing 1.18 MJD 57397.353 object DECaLS_3374_z c4d_170111_062352_ooi_z_ls9.fits.fz c4d_170111_062352_ooi_z_ls9-S15 z exptime 87.0 propid 2014B-0404 seeing 1.16 MJD 57764.265 object DECaLS_19246_z c4d_170422_003227_ooi_g_ls9.fits.fz c4d_170422_003227_ooi_g_ls9-S15 g exptime 163.0 propid 2014B-0404 seeing 1.18 MJD 57865.020 object DECaLS_19246_g c4d_170422_000843_ooi_r_ls9.fits.fz c4d_170422_000843_ooi_r_ls9-S15 r exptime 78.0 propid 2014B-0404 seeing 1.14 MJD 57865.005 object DECaLS_19246_r c4d_170514_004925_ooi_z_ls9.fits.fz c4d_170514_004925_ooi_z_ls9-S20 z exptime 250.0 propid 2014B-0404 seeing 1.49 MJD 57887.031 object DECaLS_35118_z c4d_180318_043628_ooi_r_ls9.fits.fz c4d_180318_043628_ooi_r_ls9-S20 r exptime 51.0 propid 2014B-0404 seeing 1.13 MJD 58195.192 object DECaLS_35118_r c4d_180318_043343_ooi_g_ls9.fits.fz c4d_180318_043343_ooi_g_ls9-S20 g exptime 135.0 propid 2014B-0404 seeing 1.32 MJD 58195.190 object DECaLS_35118_g The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_9553d83-z-20160110T082744_oi[S16,noinherit][2046,4094][real]: DECaLS_ The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_9571eaa-r-20160117T081200_oi[S16,noinherit][2046,4094][real]: DECaLS_ The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_953a3c6-g-20190306T051857_oi[S16,noinherit][2046,4094][real]: DECaLS_ The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_956e86d-g-20160117T081308_oi[S16,noinherit][2046,4094][real]: DECaLS_ The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95176d0-r-20170422T000637_oi[S15,noinherit][2046,4094][real]: DECaLS_ The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_9517d11-g-20170422T002854_oi[S15,noinherit][2046,4094][real]: DECaLS_ The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_953c314-z-20170111T062137_oi[S15,noinherit][2046,4094][real]: DECaLS_ The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_952df04-r-20180318T043628_oi[S20,noinherit][2046,4094][real]: DECaLS_ The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95164f7-z-20170514T004423_oi[S20,noinherit][2046,4094][real]: DECaLS_ The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_952e886-g-20180318T043343_oi[S20,noinherit][2046,4094][real]: DECaLS_ Stage tims : Wall: 114.59 s, CPU: 0.82 s, VmPeak: 995 MB, VmSize: 995 MB, VmRSS: 172 MB, VmData: 206 MB, maxrss: 0.45462 MB Stage tims finished: 2020-06-06T15:23:14.503854 WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_9553d83-z-20160110T082744_oi[S16,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_953c314-z-20170111T062137_oi[S15,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_9571eaa-r-20160117T081200_oi[S16,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95176d0-r-20170422T000637_oi[S15,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_952df04-r-20180318T043628_oi[S20,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_956e86d-g-20160117T081308_oi[S16,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95164f7-z-20170514T004423_oi[S20,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] Resources for stage tims : 3.678 s worker CPU, 387.053 s worker Wall, Wall: 114.593 s, Cores in use: 0.04, Total efficiency (on 4 cores): 1.0 % Stage tims finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/PGC1401624-largegalaxy-tims.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/PGC1401624-largegalaxy-tims.p Running stage refs WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_953a3c6-g-20190306T051857_oi[S16,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_9517d11-g-20170422T002854_oi[S15,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_952e886-g-20180318T043343_oi[S20,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] Running stage refs at 2020-06-06T15:23:14.523431 Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-04859.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-04988.fits Matched 13 large galaxies to Gaia stars. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/metrics/cus/reference-custom-167710p11968.fits Stage refs : Wall: 9.64 s, CPU: 1.09 s, VmPeak: 1026 MB, VmSize: 1007 MB, VmRSS: 184 MB, VmData: 217 MB, maxrss: 0.45462 MB Stage refs finished: 2020-06-06T15:23:24.165727 Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 9.643 s, Cores in use: 0.11, Total efficiency (on 4 cores): 2.8 % Stage refs finished Running stage outliers Running stage outliers at 2020-06-06T15:23:24.167187 Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/metrics/cus/outlier-mask-custom-167710p11968.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/metrics/cus/outliers-pre-custom-167710p11968.jpg Image c4d_180318_043343_ooi_g_ls9-S20 g : masking 0 positive outlier pixels and 53 negative outlier pixels Image c4d_170514_004925_ooi_z_ls9-S20 z : masking 261 positive outlier pixels and 0 negative outlier pixels Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/metrics/cus/outlier-mask-custom-167710p11968.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/metrics/cus/outliers-post-custom-167710p11968.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/metrics/cus/outliers-masked-pos-custom-167710p11968.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/metrics/cus/outliers-masked-neg-custom-167710p11968.jpg Stage outliers : Wall: 0.56 s, CPU: 0.39 s, VmPeak: 1026 MB, VmSize: 1016 MB, VmRSS: 194 MB, VmData: 225 MB, maxrss: 0.45462 MB Stage outliers finished: 2020-06-06T15:23:24.725857 Resources for stage outliers : 0.543 s worker CPU, 0.553 s worker Wall, Wall: 0.560 s, Cores in use: 1.67, Total efficiency (on 4 cores): 41.6 % Stage outliers finished Running stage halos Running stage halos at 2020-06-06T15:23:24.727443 Stage halos : Wall: 0.10 s, CPU: 0.09 s, VmPeak: 1026 MB, VmSize: 1016 MB, VmRSS: 194 MB, VmData: 225 MB, maxrss: 0.45462 MB Stage halos finished: 2020-06-06T15:23:24.822670 Resources for stage halos : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 0.096 s, Cores in use: 0.97, Total efficiency (on 4 cores): 24.3 % Stage halos finished Running stage fit_on_coadds Running stage fit_on_coadds at 2020-06-06T15:23:24.823985 PSF sigmas in coadd pixels: ['2.12', '1.95', '1.91', '2.15'] Using average PSF sigma 2.032005117581845 Gaussian PSF norm 0.13882582742192132 vs pixelized 0.12461262 average tim pixel scale / coadd scale: 1.0011377040108396 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-copsf-g.fits.fz PSF sigmas in coadd pixels: ['1.77', '1.85', '1.84'] Using average PSF sigma 1.8183003701701341 Gaussian PSF norm 0.15514201965843696 vs pixelized 0.1346105 average tim pixel scale / coadd scale: 1.0011606482710718 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-copsf-r.fits.fz PSF sigmas in coadd pixels: ['1.92', '1.89', '2.42'] Using average PSF sigma 2.0735985581302643 Gaussian PSF norm 0.13604117859159742 vs pixelized 0.11711492 average tim pixel scale / coadd scale: 1.0011329885238056 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-copsf-z.fits.fz Stage fit_on_coadds : Wall: 0.15 s, CPU: 0.12 s, VmPeak: 1026 MB, VmSize: 1016 MB, VmRSS: 197 MB, VmData: 226 MB, maxrss: 0.45462 MB Stage fit_on_coadds finished: 2020-06-06T15:23:24.969426 Resources for stage fit_on_coadds : 0.129 s worker CPU, 0.133 s worker Wall, Wall: 0.146 s, Cores in use: 1.71, Total efficiency (on 4 cores): 42.8 % Stage fit_on_coadds finished Running stage srcs Running stage srcs at 2020-06-06T15:23:24.971755 SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])] Found 7 potential peaks Of 7 potential peaks: 1 in veto map, 0 cut by saddle test, 0 cut by aper test, 6 kept SED z : 6 new peaks Found 16 potential peaks Of 16 potential peaks: 8 in veto map, 3 cut by saddle test, 0 cut by aper test, 5 kept SED r : 5 new peaks Found 19 potential peaks Of 19 potential peaks: 14 in veto map, 1 cut by saddle test, 0 cut by aper test, 4 kept SED g : 4 new peaks Found 22 potential peaks Of 22 potential peaks: 19 in veto map, 2 cut by saddle test, 0 cut by aper test, 1 kept SED Flat : 1 new peaks Found 22 potential peaks Of 22 potential peaks: 18 in veto map, 2 cut by saddle test, 0 cut by aper test, 2 kept SED Red : 2 new peaks Adding 18 new sources Median coadd sky for g : 4.9824506e-05 Median coadd sky for r : 0.0001009391 Median coadd sky for z : 0.00056912936 Stage srcs : Wall: 0.32 s, CPU: 0.28 s, VmPeak: 1026 MB, VmSize: 1016 MB, VmRSS: 198 MB, VmData: 226 MB, maxrss: 0.45462 MB Stage srcs finished: 2020-06-06T15:23:25.295354 Resources for stage srcs : 0.085 s worker CPU, 0.085 s worker Wall, Wall: 0.324 s, Cores in use: 1.14, Total efficiency (on 4 cores): 28.5 % Stage srcs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/PGC1401624-largegalaxy-srcs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/PGC1401624-largegalaxy-srcs.p Running stage fitblobs Running stage fitblobs at 2020-06-06T15:23:25.315400 Blob 1 of 14, id: 3, sources: 4, size: 114x113, npix 6427, brick X: 54,168, Y: 48,161, one pixel: 127 48 Blob 2 of 14, id: 0, sources: 2, size: 43x29, npix 797, brick X: 60,103, Y: 0,29, one pixel: 61 0 Blob 3 of 14, id: 10, sources: 2, size: 42x28, npix 728, brick X: 15,57, Y: 177,205, one pixel: 43 177 Blob 4 of 14, id: 4, sources: 1, size: 31x35, npix 711, brick X: 18,49, Y: 82,117, one pixel: 32 82 Blob 5 of 14, id: 7, sources: 1, size: 30x34, npix 599, brick X: 183,213, Y: 109,143, one pixel: 192 109 Blob 4 finished, total: Wall: 0.74 s, CPU: 0.74 s, VmPeak: 957 MB, VmSize: 719 MB, VmRSS: 141 MB, VmData: 182 MB, maxrss: 0.144168 MB Blob 6 of 14, id: 9, sources: 1, size: 29x31, npix 593, brick X: 0,29, Y: 152,183, one pixel: 13 152 Measuring 1 iterative sources Blob 3 finished, total: Wall: 1.67 s, CPU: 1.67 s, VmPeak: 961 MB, VmSize: 719 MB, VmRSS: 141 MB, VmData: 182 MB, maxrss: 0.144464 MB Blob 7 of 14, id: 12, sources: 1, size: 27x25, npix 469, brick X: 49,76, Y: 208,233, one pixel: 58 208 Blob 5 finished, total: Wall: 1.02 s, CPU: 1.02 s, VmPeak: 957 MB, VmSize: 719 MB, VmRSS: 141 MB, VmData: 182 MB, maxrss: 0.144168 MB Blob 8 of 14, id: 5, sources: 1, size: 26x27, npix 428, brick X: 58,84, Y: 88,115, one pixel: 71 88 Blob 2 finished, total: Wall: 1.83 s, CPU: 1.83 s, VmPeak: 962 MB, VmSize: 714 MB, VmRSS: 137 MB, VmData: 177 MB, maxrss: 0.140676 MB Blob 9 of 14, id: 13, sources: 1, size: 26x21, npix 398, brick X: 153,179, Y: 212,233, one pixel: 163 212 Blob 8 finished, total: Wall: 0.67 s, CPU: 0.67 s, VmPeak: 962 MB, VmSize: 714 MB, VmRSS: 138 MB, VmData: 177 MB, maxrss: 0.141496 MB Blob 10 of 14, id: 1, sources: 1, size: 20x19, npix 218, brick X: 186,206, Y: 1,20, one pixel: 195 1 Blob 7 finished, total: Wall: 0.96 s, CPU: 0.96 s, VmPeak: 957 MB, VmSize: 719 MB, VmRSS: 143 MB, VmData: 182 MB, maxrss: 0.146192 MB Blob 11 of 14, id: 2, sources: 1, size: 19x19, npix 208, brick X: 149,168, Y: 18,37, one pixel: 157 18 Blob 10 finished, total: Wall: 0.35 s, CPU: 0.35 s, VmPeak: 957 MB, VmSize: 719 MB, VmRSS: 143 MB, VmData: 182 MB, maxrss: 0.146192 MB Blob 12 of 14, id: 6, sources: 1, size: 19x19, npix 190, brick X: 204,223, Y: 93,112, one pixel: 213 93 Blob 9 finished, total: Wall: 0.76 s, CPU: 0.76 s, VmPeak: 962 MB, VmSize: 714 MB, VmRSS: 138 MB, VmData: 177 MB, maxrss: 0.141496 MB Blob 13 of 14, id: 8, sources: 1, size: 14x18, npix 150, brick X: 219,233, Y: 128,146, one pixel: 227 128 Blob 14 of 14, id: 11, sources: 1, size: 17x17, npix 145, brick X: 180,197, Y: 198,215, one pixel: 188 198 Blob 11 finished, total: Wall: 0.31 s, CPU: 0.31 s, VmPeak: 957 MB, VmSize: 719 MB, VmRSS: 143 MB, VmData: 182 MB, maxrss: 0.146192 MB Blob 6 finished, total: Wall: 1.79 s, CPU: 1.79 s, VmPeak: 961 MB, VmSize: 719 MB, VmRSS: 143 MB, VmData: 182 MB, maxrss: 0.146476 MB Blob 12 finished, total: Wall: 0.32 s, CPU: 0.32 s, VmPeak: 962 MB, VmSize: 714 MB, VmRSS: 138 MB, VmData: 177 MB, maxrss: 0.141496 MB Blob 13 finished, total: Wall: 0.83 s, CPU: 0.83 s, VmPeak: 957 MB, VmSize: 719 MB, VmRSS: 143 MB, VmData: 182 MB, maxrss: 0.146192 MB Blob 14 finished, total: Wall: 0.88 s, CPU: 0.88 s, VmPeak: 961 MB, VmSize: 719 MB, VmRSS: 143 MB, VmData: 182 MB, maxrss: 0.146476 MB Known galaxy. Initial shape: EllipseWithPriors(0.25): log r_e=0.751255, ee1=-0.0349789, ee2=0.640309 Blob 1 finished, total: Wall: 5.04 s, CPU: 5.04 s, VmPeak: 961 MB, VmSize: 719 MB, VmRSS: 144 MB, VmData: 182 MB, maxrss: 0.147464 MB Old catalog: 19 New catalog: 18 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/metrics/cus/blobs-custom-167710p11968.fits.gz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/metrics/cus/all-models-custom-167710p11968.fits Stage fitblobs : Wall: 5.27 s, CPU: 0.21 s, VmPeak: 1026 MB, VmSize: 1016 MB, VmRSS: 198 MB, VmData: 226 MB, maxrss: 0.45462 MB Stage fitblobs finished: 2020-06-06T15:23:30.586766 Resources for stage fitblobs : 17.266 s worker CPU, 17.291 s worker Wall, Wall: 5.272 s, Cores in use: 3.31, Total efficiency (on 4 cores): 82.9 % Stage fitblobs finished Running stage coadds Running stage coadds at 2020-06-06T15:23:30.588276 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-ccds.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-image-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-invvar-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-nexp-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-depth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-galdepth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-psfsize-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-model-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-chi2-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-blobmodel-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-image-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-invvar-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-nexp-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-depth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-galdepth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-psfsize-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-model-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-chi2-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-blobmodel-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-image-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-invvar-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-nexp-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-depth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-galdepth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-psfsize-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-model-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-chi2-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-blobmodel-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-depth.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/tmp-legacysurvey-custom-167710p11968-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/tmp-legacysurvey-custom-167710p11968-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/tmp-legacysurvey-custom-167710p11968-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/tmp-legacysurvey-custom-167710p11968-resid.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-resid.jpg Stage coadds : Wall: 1.05 s, CPU: 0.39 s, VmPeak: 1026 MB, VmSize: 1016 MB, VmRSS: 198 MB, VmData: 226 MB, maxrss: 0.45462 MB Stage coadds finished: 2020-06-06T15:23:31.642691 Resources for stage coadds : 2.019 s worker CPU, 2.091 s worker Wall, Wall: 1.055 s, Cores in use: 2.28, Total efficiency (on 4 cores): 57.1 % Stage coadds finished Running stage wise_forced Running stage wise_forced at 2020-06-06T15:23:31.643962 Cut to 1 unWISE tiles Reading WISE tile 1684p121 band 2 Reading WISE tile 1684p121 band 3 Reading WISE tile 1684p121 band 1 Reading WISE tile 1684p121 band 4 Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/168/1684p121/unwise-1684p121-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/168/1684p121/unwise-1684p121-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/168/1684p121/unwise-1684p121-w4-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/168/1684p121/unwise-1684p121-w4-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/168/1684p121/unwise-1684p121-w4-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.03 s, CPU: 0.03 s, VmPeak: 957 MB, VmSize: 798 MB, VmRSS: 148 MB, VmData: 176 MB, maxrss: 0.190512 MB Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/168/1684p121/unwise-1684p121-w1-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/168/1684p121/unwise-1684p121-w1-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/168/1684p121/unwise-1684p121-w1-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/168/1684p121/unwise-1684p121-w1-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.03 s, CPU: 0.03 s, VmPeak: 961 MB, VmSize: 795 MB, VmRSS: 145 MB, VmData: 174 MB, maxrss: 0.2097 MB Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/168/1684p121/unwise-1684p121-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/168/1684p121/unwise-1684p121-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/168/1684p121/unwise-1684p121-w3-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/168/1684p121/unwise-1684p121-w3-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/168/1684p121/unwise-1684p121-w3-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.03 s, CPU: 0.03 s, VmPeak: 961 MB, VmSize: 791 MB, VmRSS: 141 MB, VmData: 170 MB, maxrss: 0.205944 MB Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/168/1684p121/unwise-1684p121-w2-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/168/1684p121/unwise-1684p121-w2-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/168/1684p121/unwise-1684p121-w2-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/168/1684p121/unwise-1684p121-w2-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.03 s, CPU: 0.03 s, VmPeak: 962 MB, VmSize: 791 MB, VmRSS: 141 MB, VmData: 170 MB, maxrss: 0.204468 MB Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-image-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-model-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-invvar-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-image-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-model-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-invvar-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-image-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-model-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-invvar-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-image-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-model-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-invvar-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/tmp-legacysurvey-custom-167710p11968-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/tmp-legacysurvey-custom-167710p11968-wisemodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-wisemodel.jpg Setting WISE apphot Setting WISE apphot Setting WISE apphot Setting WISE apphot Stage wise_forced : Wall: 12.25 s, CPU: 0.31 s, VmPeak: 1026 MB, VmSize: 1016 MB, VmRSS: 199 MB, VmData: 226 MB, maxrss: 0.45462 MB Stage wise_forced finished: 2020-06-06T15:23:43.894663 Resources for stage wise_forced : 2.821 s worker CPU, 21.549 s worker Wall, Wall: 12.252 s, Cores in use: 0.26, Total efficiency (on 4 cores): 6.4 % Stage wise_forced finished Running stage writecat Running stage writecat at 2020-06-06T15:23:43.896538 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/coadd/cus/custom-167710p11968/legacysurvey-custom-167710p11968-maskbits.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/tractor-i/cus/tractor-custom-167710p11968.fits Reading SFD maps... END Rejected (1), keyrecords following the END keyrecord were ignored. END Rejected (1), keyrecords following the END keyrecord were ignored. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/tractor/cus/tractor-custom-167710p11968.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/167/PGC1401624/tractor/cus/brick-custom-167710p11968.sha256sum Stage writecat : Wall: 1.12 s, CPU: 0.40 s, VmPeak: 1149 MB, VmSize: 1021 MB, VmRSS: 201 MB, VmData: 226 MB, maxrss: 0.45462 MB Stage writecat finished: 2020-06-06T15:23:45.015434 Resources for stage writecat : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 1.120 s, Cores in use: 0.36, Total efficiency (on 4 cores): 8.9 % Stage writecat finished All done: 26.543 s worker CPU, 428.757 s worker Wall, Wall: 145.107 s, Cores in use: 0.21, Total efficiency (on 4 cores): 5.3 % Finished galaxy PGC1401624 in 2.452 minutes.