Started working on galaxy PGC2733107 at Sun Jun 7 17:37:36 2020 Bands touching this brick, g r z python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 176.64388499999998 69.935489 --width 251 --height 251 --pixscale 0.262 --threads 4 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 runbrick.py starting at 2020-06-07T17:37:37.328696 legacypipe git version: DR9.5.2 Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '176.64388499999998', '69.935489', '--width', '251', '--height', '251', '--pixscale', '0.262', '--threads', '4', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 176.64388499999998 69.935489 --width 251 --height 251 --pixscale 0.262 --threads 4 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 Parsed RA,Dec 176.64388499999998 69.935489 Starting process 9272 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 632 MB, VmSize: 632 MB, VmRSS: 56 MB, VmData: 127 MB, maxrss: 0.057304 MB Starting process 9273 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 632 MB, VmSize: 632 MB, VmRSS: 56 MB, VmData: 127 MB, maxrss: 0.057308 MB Starting process 9275 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 632 MB, VmSize: 632 MB, VmRSS: 56 MB, VmData: 127 MB, maxrss: 0.057308 MB Starting process 9278 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 632 MB, VmSize: 632 MB, VmRSS: 56 MB, VmData: 127 MB, maxrss: 0.057308 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage fit_on_coadds Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-06-07T17:37:37.981286 Keeping 8 CCDs: k4m_171213_080628_ooi_zd_ls9.fits.fz k4m_171213_080628_ooi_zd_ls9-CCD2 z exptime 168.104 propid 2016A-0453 seeing 1.24 MJD 58100.338 object MzLS_713560_z k4m_170605_054327_ooi_zd_ls9.fits.fz k4m_170605_054327_ooi_zd_ls9-CCD2 z exptime 145.105 propid 2016A-0453 seeing 0.98 MJD 57909.239 object MzLS_713122_z k4m_171124_100323_ooi_zd_ls9.fits.fz k4m_171124_100323_ooi_zd_ls9-CCD1 z exptime 106.104 propid 2016A-0453 seeing 0.96 MJD 58081.419 object MzLS_713994_z k4m_170420_092701_ooi_zd_ls9.fits.fz k4m_170420_092701_ooi_zd_ls9-CCD4 z exptime 149.103 propid 2016A-0453 seeing 1.61 MJD 57863.394 object MzLS_713121_z ksb_180415_071429_ooi_g_ls9.fits.fz ksb_180415_071429_ooi_g_ls9-CCD2 g exptime 143.352 propid BASS seeing 1.77 MJD 58223.302 object 45951 ksb_180508_065824_ooi_r_ls9.fits.fz ksb_180508_065824_ooi_r_ls9-CCD2 r exptime 95.625 propid NOTBASS seeing 1.14 MJD 58246.291 object 45951 ksb_180409_080715_ooi_g_ls9.fits.fz ksb_180409_080715_ooi_g_ls9-CCD3 g exptime 153.09 propid BASS seeing 2.00 MJD 58217.338 object 45963 ksb_180409_080412_ooi_g_ls9.fits.fz ksb_180409_080412_ooi_g_ls9-CCD1 g exptime 146.61 propid BASS seeing 2.00 MJD 58217.336 object 45953 mosaic get_fwhm: self.fwhm = 3.77 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2941 3204 and amp range 2048 4095 and image y 506 764 and amp range 0 2039 Range in image: x 0 264 , y 0 259 (with image size 264 x 259) dzp 0.0044767857 -> scaling image by 1.004131784351374 mosaic get_fwhm: self.fwhm = 4.79 Found 4 relevant rows in amp-corrections file. Found overlap: image x 207 460 and amp range 0 2047 and image y 2942 3194 and amp range 2040 4078 Range in image: x 0 254 , y 0 253 (with image size 254 x 253) dzp 0.003927584 -> scaling image by 1.0036239891963392 mosaic get_fwhm: self.fwhm = 3.71 Found 4 relevant rows in amp-corrections file. Found overlap: image x 1780 2034 and amp range 0 2005 and image y 2039 2296 and amp range 0 2039 Range in image: x 0 226 , y 0 1 (with image size 255 x 258) dzp 0.009508088 -> scaling image by 1.0087957299815176 Found overlap: image x 1780 2034 and amp range 2006 4053 and image y 2039 2296 and amp range 0 2039 Range in image: x 226 255 , y 0 1 (with image size 255 x 258) dzp 0.0061123776 -> scaling image by 1.005645584425507 Found overlap: image x 1780 2034 and amp range 0 2005 and image y 2039 2296 and amp range 2040 4078 Range in image: x 0 226 , y 1 258 (with image size 255 x 258) dzp 0.014419514 -> scaling image by 1.0133694453441033 Found overlap: image x 1780 2034 and amp range 2006 4053 and image y 2039 2296 and amp range 2040 4078 Range in image: x 226 255 , y 1 258 (with image size 255 x 258) dzp 0.0036595785 -> scaling image by 1.003376283213225 mosaic get_fwhm: self.fwhm = 6.19 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2992 3252 and amp range 2048 4095 and image y 3998 4054 and amp range 2022 4060 Range in image: x 0 261 , y 0 57 (with image size 261 x 57) dzp -0.02499645 -> scaling image by 0.9772404160512723 Faked up EXPNUM 82230094 Faked up EXPNUM 82230094 Faked up EXPNUM 82230094 90prime get_fwhm: self.fwhm = 3.91 Found 4 relevant rows in amp-corrections file. Found overlap: image x 312 459 and amp range 0 2015 and image y 2197 2345 and amp range 2048 4095 Range in image: x 0 148 , y 0 149 (with image size 148 x 149) dzp 0.0459271 -> scaling image by 1.0432078344950917 Faked up EXPNUM 82460121 Faked up EXPNUM 82460121 Faked up EXPNUM 82460121 90prime get_fwhm: self.fwhm = 2.51 Found 4 relevant rows in amp-corrections file. Found overlap: image x 331 479 and amp range 0 2015 and image y 2172 2320 and amp range 2048 4095 Range in image: x 0 149 , y 0 149 (with image size 149 x 149) dzp 0.0503617 -> scaling image by 1.047477444828492 Faked up EXPNUM 82170118 Faked up EXPNUM 82170118 Faked up EXPNUM 82170118 90prime get_fwhm: self.fwhm = 4.41 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2547 2696 and amp range 2016 4031 and image y 3675 3824 and amp range 2048 4095 Range in image: x 0 150 , y 0 150 (with image size 150 x 150) dzp -0.01723723 -> scaling image by 0.9842492857213068 Faked up EXPNUM 82170117 Faked up EXPNUM 82170117 Faked up EXPNUM 82170117 90prime get_fwhm: self.fwhm = 4.41 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2463 2614 and amp range 2016 4031 and image y 226 377 and amp range 0 2047 Range in image: x 0 152 , y 0 152 (with image size 152 x 152) dzp -0.019046262 -> scaling image by 0.9826107140833663 Stage tims : Wall: 5.37 s, CPU: 1.12 s, VmPeak: 941 MB, VmSize: 941 MB, VmRSS: 170 MB, VmData: 204 MB, maxrss: 0.56654 MB Stage tims finished: 2020-06-07T17:37:43.350072 Resources for stage tims : 2.850 s worker CPU, 12.012 s worker Wall, Wall: 5.370 s, Cores in use: 0.74, Total efficiency (on 4 cores): 18.5 % Stage tims finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-tims.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-tims.p Running stage refs Running stage refs at 2020-06-07T17:37:43.366839 Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00337.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00391.fits Matched 12 large galaxies to Gaia stars. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/reference-custom-176643p69935.fits Stage refs : Wall: 2.46 s, CPU: 1.51 s, VmPeak: 972 MB, VmSize: 955 MB, VmRSS: 184 MB, VmData: 217 MB, maxrss: 0.56654 MB Stage refs finished: 2020-06-07T17:37:45.823191 Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 2.457 s, Cores in use: 0.61, Total efficiency (on 4 cores): 15.3 % Stage refs finished Running stage outliers Running stage outliers at 2020-06-07T17:37:45.824615 Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outlier-mask-custom-176643p69935.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outliers-pre-custom-176643p69935.jpg Image k4m_170420_092701_ooi_zd_ls9-CCD4 z : masking 1821 positive outlier pixels and 397 negative outlier pixels Image k4m_171213_080628_ooi_zd_ls9-CCD2 z : masking 339 positive outlier pixels and 101 negative outlier pixels Image k4m_170605_054327_ooi_zd_ls9-CCD2 z : masking 0 positive outlier pixels and 271 negative outlier pixels Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outlier-mask-custom-176643p69935.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outliers-post-custom-176643p69935.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outliers-masked-pos-custom-176643p69935.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outliers-masked-neg-custom-176643p69935.jpg Stage outliers : Wall: 0.87 s, CPU: 0.68 s, VmPeak: 972 MB, VmSize: 964 MB, VmRSS: 196 MB, VmData: 227 MB, maxrss: 0.56654 MB Stage outliers finished: 2020-06-07T17:37:46.698426 Resources for stage outliers : 0.664 s worker CPU, 0.667 s worker Wall, Wall: 0.875 s, Cores in use: 1.54, Total efficiency (on 4 cores): 38.5 % Stage outliers finished Running stage halos Running stage halos at 2020-06-07T17:37:46.700065 Stage halos : Wall: 0.24 s, CPU: 0.23 s, VmPeak: 972 MB, VmSize: 964 MB, VmRSS: 196 MB, VmData: 227 MB, maxrss: 0.56654 MB Stage halos finished: 2020-06-07T17:37:46.939335 Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Resources for stage halos : 0.000 s worker CPU, 0.001 s worker Wall, Wall: 0.240 s, Cores in use: 0.97, Total efficiency (on 4 cores): 24.3 % Stage halos finished Running stage fit_on_coadds Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Running stage fit_on_coadds at 2020-06-07T17:37:46.940859 PSF sigmas in coadd pixels: ['2.88', '3.25', '3.25'] Using average PSF sigma 3.124353901293587 Gaussian PSF norm 0.09028900076175157 vs pixelized 0.08135869 average tim pixel scale / coadd scale: 1.7303248728760552 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-copsf-g.fits.fz PSF sigmas in coadd pixels: ['1.85'] Using average PSF sigma 1.8481836232792843 Gaussian PSF norm 0.15263353068422358 vs pixelized 0.12905155 average tim pixel scale / coadd scale: 1.7303711279714835 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-copsf-r.fits.fz PSF sigmas in coadd pixels: ['2.02', '1.59', '1.56', '2.61'] Using average PSF sigma 1.9462635251182068 Gaussian PSF norm 0.14494172455744145 vs pixelized 0.119682044 average tim pixel scale / coadd scale: 0.9910524269576604 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-copsf-z.fits.fz Stage fit_on_coadds : Wall: 0.17 s, CPU: 0.14 s, VmPeak: 972 MB, VmSize: 964 MB, VmRSS: 199 MB, VmData: 228 MB, maxrss: 0.56654 MB Stage fit_on_coadds finished: 2020-06-07T17:37:47.109896 Resources for stage fit_on_coadds : 0.148 s worker CPU, 0.148 s worker Wall, Wall: 0.170 s, Cores in use: 1.69, Total efficiency (on 4 cores): 42.2 % Stage fit_on_coadds finished Running stage srcs Running stage srcs at 2020-06-07T17:37:47.112353 SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])] Found 6 potential peaks Of 6 potential peaks: 0 in veto map, 1 cut by saddle test, 0 cut by aper test, 5 kept SED z : 5 new peaks Found 17 potential peaks Of 17 potential peaks: 16 in veto map, 0 cut by saddle test, 0 cut by aper test, 1 kept SED r : 1 new peaks Found 6 potential peaks Of 6 potential peaks: 4 in veto map, 0 cut by saddle test, 0 cut by aper test, 2 kept SED g : 2 new peaks Found 11 potential peaks Of 11 potential peaks: 7 in veto map, 1 cut by saddle test, 0 cut by aper test, 3 kept SED Flat : 3 new peaks Found 9 potential peaks Of 9 potential peaks: 7 in veto map, 2 cut by saddle test, 0 cut by aper test, 0 kept SED Red : 0 new peaks Adding 11 new sources Median coadd sky for g : 1.51026225e-05 Median coadd sky for r : -9.553852e-05 Median coadd sky for z : 0.00021544908 Stage srcs : Wall: 0.56 s, CPU: 0.50 s, VmPeak: 972 MB, VmSize: 964 MB, VmRSS: 199 MB, VmData: 228 MB, maxrss: 0.56654 MB Stage srcs finished: 2020-06-07T17:37:47.667916 Resources for stage srcs : 0.109 s worker CPU, 0.119 s worker Wall, Wall: 0.556 s, Cores in use: 1.10, Total efficiency (on 4 cores): 27.4 % Stage srcs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-srcs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-srcs.p Running stage fitblobs Running stage fitblobs at 2020-06-07T17:37:47.680872 Blob 1 of 9, id: 4, sources: 2, size: 100x91, npix 6043, brick X: 79,179, Y: 72,163, one pixel: 140 72 Blob 2 of 9, id: 3, sources: 2, size: 67x32, npix 1849, brick X: 184,251, Y: 22,54, one pixel: 198 22 Blob 3 of 9, id: 6, sources: 1, size: 22x42, npix 701, brick X: 0,22, Y: 176,218, one pixel: 5 176 Blob 4 of 9, id: 8, sources: 1, size: 35x26, npix 646, brick X: 153,188, Y: 225,251, one pixel: 167 225 Blob 5 of 9, id: 0, sources: 2, size: 28x32, npix 444, brick X: 7,35, Y: 0,32, one pixel: 17 0 Blob 2 finished, total: Wall: 0.38 s, CPU: 0.38 s, VmPeak: 805 MB, VmSize: 712 MB, VmRSS: 136 MB, VmData: 178 MB, maxrss: 0.139604 MB Blob 6 of 9, id: 5, sources: 1, size: 26x24, npix 388, brick X: 204,230, Y: 96,120, one pixel: 216 96 Blob 4 finished, total: Wall: 1.00 s, CPU: 1.00 s, VmPeak: 803 MB, VmSize: 713 MB, VmRSS: 138 MB, VmData: 178 MB, maxrss: 0.14104 MB Blob 7 of 9, id: 7, sources: 1, size: 22x20, npix 250, brick X: 162,184, Y: 193,213, one pixel: 172 193 Blob 8 of 9, id: 1, sources: 1, size: 20x20, npix 228, brick X: 138,158, Y: 1,21, one pixel: 147 1 Blob 5 finished, total: Wall: 0.87 s, CPU: 0.87 s, VmPeak: 805 MB, VmSize: 712 MB, VmRSS: 138 MB, VmData: 178 MB, maxrss: 0.141548 MB Blob 9 of 9, id: 2, sources: 1, size: 19x19, npix 217, brick X: 211,230, Y: 3,22, one pixel: 219 3 Blob 6 finished, total: Wall: 0.50 s, CPU: 0.50 s, VmPeak: 803 MB, VmSize: 713 MB, VmRSS: 138 MB, VmData: 178 MB, maxrss: 0.14104 MB Blob 3 finished, total: Wall: 1.65 s, CPU: 1.65 s, VmPeak: 803 MB, VmSize: 713 MB, VmRSS: 138 MB, VmData: 178 MB, maxrss: 0.141776 MB Blob 9 finished, total: Wall: 0.07 s, CPU: 0.07 s, VmPeak: 803 MB, VmSize: 713 MB, VmRSS: 139 MB, VmData: 178 MB, maxrss: 0.142328 MB Blob 8 finished, total: Wall: 0.23 s, CPU: 0.23 s, VmPeak: 803 MB, VmSize: 713 MB, VmRSS: 138 MB, VmData: 178 MB, maxrss: 0.14104 MB Blob 7 finished, total: Wall: 0.55 s, CPU: 0.55 s, VmPeak: 805 MB, VmSize: 712 MB, VmRSS: 138 MB, VmData: 178 MB, maxrss: 0.141548 MB Known galaxy. Initial shape: EllipseWithPriors(0.25): log r_e=1.82709, ee1=-0.0734572, ee2=0.494995 Blob 1 finished, total: Wall: 3.51 s, CPU: 3.51 s, VmPeak: 805 MB, VmSize: 713 MB, VmRSS: 140 MB, VmData: 178 MB, maxrss: 0.143688 MB Old catalog: 17 New catalog: 9 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/blobs-custom-176643p69935.fits.gz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/all-models-custom-176643p69935.fits Stage fitblobs : Wall: 4.03 s, CPU: 0.43 s, VmPeak: 972 MB, VmSize: 964 MB, VmRSS: 199 MB, VmData: 228 MB, maxrss: 0.56654 MB Stage fitblobs finished: 2020-06-07T17:37:51.713347 Resources for stage fitblobs : 8.867 s worker CPU, 9.164 s worker Wall, Wall: 4.034 s, Cores in use: 2.31, Total efficiency (on 4 cores): 57.6 % Stage fitblobs finished Running stage coadds Running stage coadds at 2020-06-07T17:37:51.715133 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-ccds.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-image-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-invvar-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-nexp-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-depth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-galdepth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-psfsize-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-model-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-chi2-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-blobmodel-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-image-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-invvar-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-nexp-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-depth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-galdepth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-psfsize-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-model-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-chi2-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-blobmodel-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-image-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-invvar-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-nexp-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-depth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-galdepth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-psfsize-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-model-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-chi2-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-blobmodel-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-depth.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/tmp-legacysurvey-custom-176643p69935-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/tmp-legacysurvey-custom-176643p69935-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/tmp-legacysurvey-custom-176643p69935-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/tmp-legacysurvey-custom-176643p69935-resid.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-resid.jpg Stage coadds : Wall: 1.15 s, CPU: 0.55 s, VmPeak: 972 MB, VmSize: 964 MB, VmRSS: 200 MB, VmData: 228 MB, maxrss: 0.56654 MB Stage coadds finished: 2020-06-07T17:37:52.863030 Resources for stage coadds : 1.650 s worker CPU, 1.732 s worker Wall, Wall: 1.149 s, Cores in use: 1.92, Total efficiency (on 4 cores): 47.9 % Stage coadds finished Running stage wise_forced Running stage wise_forced at 2020-06-07T17:37:52.864712 Cut to 1 unWISE tiles Reading WISE tile 1758p696 band 1 Reading WISE tile 1758p696 band 2 Reading WISE tile 1758p696 band 3 Reading WISE tile 1758p696 band 4 Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.04 s, CPU: 0.04 s, VmPeak: 805 MB, VmSize: 794 MB, VmRSS: 146 MB, VmData: 175 MB, maxrss: 0.187784 MB Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.03 s, CPU: 0.03 s, VmPeak: 805 MB, VmSize: 788 MB, VmRSS: 140 MB, VmData: 169 MB, maxrss: 0.204152 MB Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w1-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w1-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w1-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w1-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.04 s, CPU: 0.04 s, VmPeak: 803 MB, VmSize: 792 MB, VmRSS: 144 MB, VmData: 173 MB, maxrss: 0.208248 MB Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w2-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w2-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w2-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w2-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.02 s, CPU: 0.02 s, VmPeak: 803 MB, VmSize: 788 MB, VmRSS: 140 MB, VmData: 169 MB, maxrss: 0.20452 MB Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-image-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-model-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-invvar-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-image-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-model-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-invvar-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-image-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-model-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-invvar-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-image-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-model-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-invvar-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/tmp-legacysurvey-custom-176643p69935-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/tmp-legacysurvey-custom-176643p69935-wisemodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-wisemodel.jpg Setting WISE apphot Setting WISE apphot Setting WISE apphot Setting WISE apphot Stage wise_forced : Wall: 2.48 s, CPU: 0.39 s, VmPeak: 972 MB, VmSize: 964 MB, VmRSS: 201 MB, VmData: 228 MB, maxrss: 0.56654 MB Stage wise_forced finished: 2020-06-07T17:37:55.342679 Resources for stage wise_forced : 5.144 s worker CPU, 6.498 s worker Wall, Wall: 2.479 s, Cores in use: 2.23, Total efficiency (on 4 cores): 55.8 % Stage wise_forced finished Running stage writecat Running stage writecat at 2020-06-07T17:37:55.344474 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-maskbits.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/tractor-i/cus/tractor-custom-176643p69935.fits Reading SFD maps... END Rejected (1), keyrecords following the END keyrecord were ignored. END Rejected (1), keyrecords following the END keyrecord were ignored. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/tractor/cus/tractor-custom-176643p69935.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/tractor/cus/brick-custom-176643p69935.sha256sum Stage writecat : Wall: 0.73 s, CPU: 0.50 s, VmPeak: 1097 MB, VmSize: 969 MB, VmRSS: 203 MB, VmData: 228 MB, maxrss: 0.56654 MB Stage writecat finished: 2020-06-07T17:37:56.069789 Resources for stage writecat : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 0.726 s, Cores in use: 0.70, Total efficiency (on 4 cores): 17.4 % Stage writecat finished All done: 19.434 s worker CPU, 30.343 s worker Wall, Wall: 18.094 s, Cores in use: 1.41, Total efficiency (on 4 cores): 35.3 % Finished galaxy PGC2733107 in 0.339 minutes. Started working on galaxy PGC2733107 at Thu Jul 23 04:39:37 2020 Bands touching this brick, g r z python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 176.64388499999998 69.935489 --width 251 --height 251 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 runbrick.py starting at 2020-07-23T04:39:38.648167 legacypipe git version: DR9.5.5-21-g66b1aacb Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '176.64388499999998', '69.935489', '--width', '251', '--height', '251', '--pixscale', '0.262', '--threads', '16', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 176.64388499999998 69.935489 --width 251 --height 251 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 Parsed RA,Dec 176.64388499999998 69.935489 Starting process 12920 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 635 MB, VmSize: 635 MB, VmRSS: 57 MB, VmData: 128 MB, maxrss: 0.05886 MB Starting process 12921 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 635 MB, VmSize: 635 MB, VmRSS: 57 MB, VmData: 128 MB, maxrss: 0.05886 MB Starting process 12922 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 635 MB, VmSize: 635 MB, VmRSS: 58 MB, VmData: 128 MB, maxrss: 0.058864 MB Starting process 12923 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 635 MB, VmSize: 635 MB, VmRSS: 58 MB, VmData: 128 MB, maxrss: 0.059156 MB Starting process 12924 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 635 MB, VmSize: 635 MB, VmRSS: 58 MB, VmData: 128 MB, maxrss: 0.05916 MB Starting process 12925 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 635 MB, VmSize: 635 MB, VmRSS: 58 MB, VmData: 128 MB, maxrss: 0.059164 MB Starting process 12926 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 635 MB, VmSize: 635 MB, VmRSS: 57 MB, VmData: 128 MB, maxrss: 0.058872 MB Starting process 12927 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 635 MB, VmSize: 635 MB, VmRSS: 57 MB, VmData: 128 MB, maxrss: 0.058872 MB Starting process 12928 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 635 MB, VmSize: 635 MB, VmRSS: 57 MB, VmData: 128 MB, maxrss: 0.058872 MB Starting process 12929 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 635 MB, VmSize: 635 MB, VmRSS: 57 MB, VmData: 128 MB, maxrss: 0.058872 MB Starting process 12930 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 635 MB, VmSize: 635 MB, VmRSS: 57 MB, VmData: 128 MB, maxrss: 0.058876 MB Starting process 12931 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 635 MB, VmSize: 635 MB, VmRSS: 58 MB, VmData: 128 MB, maxrss: 0.05888 MB Starting process 12932 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 635 MB, VmSize: 635 MB, VmRSS: 58 MB, VmData: 128 MB, maxrss: 0.05888 MB Starting process 12933 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 635 MB, VmSize: 635 MB, VmRSS: 58 MB, VmData: 128 MB, maxrss: 0.05888 MB Starting process 12934 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 635 MB, VmSize: 635 MB, VmRSS: 58 MB, VmData: 128 MB, maxrss: 0.05888 MB Starting process 12935 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 635 MB, VmSize: 635 MB, VmRSS: 58 MB, VmData: 128 MB, maxrss: 0.058888 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage fit_on_coadds Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-07-23T04:39:39.261653 Keeping 8 CCDs: k4m_171213_080628_ooi_zd_ls9.fits.fz k4m_171213_080628_ooi_zd_ls9-CCD2 z exptime 168.104 propid 2016A-0453 seeing 1.24 MJD 58100.338 object MzLS_713560_z k4m_170605_054327_ooi_zd_ls9.fits.fz k4m_170605_054327_ooi_zd_ls9-CCD2 z exptime 145.105 propid 2016A-0453 seeing 0.98 MJD 57909.239 object MzLS_713122_z k4m_171124_100323_ooi_zd_ls9.fits.fz k4m_171124_100323_ooi_zd_ls9-CCD1 z exptime 106.104 propid 2016A-0453 seeing 0.96 MJD 58081.419 object MzLS_713994_z k4m_170420_092701_ooi_zd_ls9.fits.fz k4m_170420_092701_ooi_zd_ls9-CCD4 z exptime 149.103 propid 2016A-0453 seeing 1.61 MJD 57863.394 object MzLS_713121_z ksb_180415_071429_ooi_g_ls9.fits.fz ksb_180415_071429_ooi_g_ls9-CCD2 g exptime 143.352 propid BASS seeing 1.77 MJD 58223.302 object 45951 ksb_180508_065824_ooi_r_ls9.fits.fz ksb_180508_065824_ooi_r_ls9-CCD2 r exptime 95.625 propid NOTBASS seeing 1.14 MJD 58246.291 object 45951 ksb_180409_080715_ooi_g_ls9.fits.fz ksb_180409_080715_ooi_g_ls9-CCD3 g exptime 153.09 propid BASS seeing 2.00 MJD 58217.338 object 45963 ksb_180409_080412_ooi_g_ls9.fits.fz ksb_180409_080412_ooi_g_ls9-CCD1 g exptime 146.61 propid BASS seeing 2.00 MJD 58217.336 object 45953 Faked up EXPNUM 82170118 Faked up EXPNUM 82170118 Faked up EXPNUM 82170118 90prime get_fwhm: self.fwhm = 4.41 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2547 2696 and amp range 2016 4031 and image y 3675 3824 and amp range 2048 4095 Range in image: x 0 150 , y 0 150 (with image size 150 x 150) dzp -0.01723723 -> scaling image by 0.9842492857213068 mosaic get_fwhm: self.fwhm = 3.71 Found 4 relevant rows in amp-corrections file. Found overlap: image x 1780 2034 and amp range 0 2005 and image y 2039 2296 and amp range 0 2039 Range in image: x 0 226 , y 0 1 (with image size 255 x 258) dzp 0.009508088 -> scaling image by 1.0087957299815176 Found overlap: image x 1780 2034 and amp range 2006 4053 and image y 2039 2296 and amp range 0 2039 Range in image: x 226 255 , y 0 1 (with image size 255 x 258) dzp 0.0061123776 -> scaling image by 1.005645584425507 Found overlap: image x 1780 2034 and amp range 0 2005 and image y 2039 2296 and amp range 2040 4078 Range in image: x 0 226 , y 1 258 (with image size 255 x 258) dzp 0.014419514 -> scaling image by 1.0133694453441033 Found overlap: image x 1780 2034 and amp range 2006 4053 and image y 2039 2296 and amp range 2040 4078 Range in image: x 226 255 , y 1 258 (with image size 255 x 258) dzp 0.0036595785 -> scaling image by 1.003376283213225 mosaic get_fwhm: self.fwhm = 3.77 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2941 3204 and amp range 2048 4095 and image y 506 764 and amp range 0 2039 Range in image: x 0 264 , y 0 259 (with image size 264 x 259) dzp 0.0044767857 -> scaling image by 1.004131784351374 mosaic get_fwhm: self.fwhm = 4.79 Found 4 relevant rows in amp-corrections file. Found overlap: image x 207 460 and amp range 0 2047 and image y 2942 3194 and amp range 2040 4078 Range in image: x 0 254 , y 0 253 (with image size 254 x 253) dzp 0.003927584 -> scaling image by 1.0036239891963392 Faked up EXPNUM 82170117 Faked up EXPNUM 82170117 Faked up EXPNUM 82170117 90prime get_fwhm: self.fwhm = 4.41 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2463 2614 and amp range 2016 4031 and image y 226 377 and amp range 0 2047 Range in image: x 0 152 , y 0 152 (with image size 152 x 152) dzp -0.019046262 -> scaling image by 0.9826107140833663 mosaic get_fwhm: self.fwhm = 6.19 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2992 3252 and amp range 2048 4095 and image y 3998 4054 and amp range 2022 4060 Range in image: x 0 261 , y 0 57 (with image size 261 x 57) dzp -0.02499645 -> scaling image by 0.9772404160512723 Faked up EXPNUM 82230094 Faked up EXPNUM 82230094 Faked up EXPNUM 82230094 90prime get_fwhm: self.fwhm = 3.91 Found 4 relevant rows in amp-corrections file. Found overlap: image x 312 459 and amp range 0 2015 and image y 2197 2345 and amp range 2048 4095 Range in image: x 0 148 , y 0 149 (with image size 148 x 149) dzp 0.0459271 -> scaling image by 1.0432078344950917 Faked up EXPNUM 82460121 Faked up EXPNUM 82460121 Faked up EXPNUM 82460121 90prime get_fwhm: self.fwhm = 2.51 Found 4 relevant rows in amp-corrections file. Found overlap: image x 331 479 and amp range 0 2015 and image y 2172 2320 and amp range 2048 4095 Range in image: x 0 149 , y 0 149 (with image size 149 x 149) dzp 0.0503617 -> scaling image by 1.047477444828492 Stage tims : Wall: 2.64 s, CPU: 0.78 s, VmPeak: 944 MB, VmSize: 944 MB, VmRSS: 171 MB, VmData: 205 MB, maxrss: 0.175488 MB Stage tims finished: 2020-07-23T04:39:41.897936 Resources for stage tims : 4.004 s worker CPU, 10.136 s worker Wall, Wall: 2.638 s, Cores in use: 1.82, Total efficiency (on 16 cores): 11.3 % Stage tims finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-tims.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-tims.p Running stage refs Running stage refs at 2020-07-23T04:39:42.597427 Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00337.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00391.fits Matched 12 large galaxies to Gaia stars. Matched 0 large galaxies to Tycho-2 stars. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/reference-custom-176643p69935.fits Stage refs : Wall: 5.42 s, CPU: 1.47 s, VmPeak: 975 MB, VmSize: 958 MB, VmRSS: 187 MB, VmData: 220 MB, maxrss: 0.191052 MB Stage refs finished: 2020-07-23T04:39:48.013643 Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 5.418 s, Cores in use: 0.27, Total efficiency (on 16 cores): 1.7 % Stage refs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-refs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-refs.p Running stage outliers Running stage outliers at 2020-07-23T04:39:48.036344 Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outlier-mask-custom-176643p69935.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outliers-pre-custom-176643p69935.jpg Image k4m_170420_092701_ooi_zd_ls9-CCD4 z : masking 1821 positive outlier pixels and 397 negative outlier pixels Image k4m_171213_080628_ooi_zd_ls9-CCD2 z : masking 339 positive outlier pixels and 101 negative outlier pixels Image k4m_170605_054327_ooi_zd_ls9-CCD2 z : masking 0 positive outlier pixels and 271 negative outlier pixels Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outlier-mask-custom-176643p69935.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outliers-post-custom-176643p69935.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outliers-masked-pos-custom-176643p69935.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outliers-masked-neg-custom-176643p69935.jpg Stage outliers : Wall: 1.36 s, CPU: 1.01 s, VmPeak: 975 MB, VmSize: 968 MB, VmRSS: 199 MB, VmData: 230 MB, maxrss: 0.204056 MB Stage outliers finished: 2020-07-23T04:39:49.394573 Resources for stage outliers : 0.641 s worker CPU, 0.791 s worker Wall, Wall: 1.360 s, Cores in use: 1.22, Total efficiency (on 16 cores): 7.6 % Stage outliers finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-outliers.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-outliers.p Running stage halos Running stage halos at 2020-07-23T04:39:49.419872 Stage halos : Wall: 0.87 s, CPU: 0.85 s, VmPeak: 975 MB, VmSize: 968 MB, VmRSS: 199 MB, VmData: 230 MB, maxrss: 0.20416 MB Stage halos finished: 2020-07-23T04:39:50.294708 Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.876 s, Cores in use: 0.98, Total efficiency (on 16 cores): 6.1 % Stage halos finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-halos.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-halos.p Running stage fit_on_coadds Warning: Stellar halo subtraction is only implemented for DECam Running stage fit_on_coadds at 2020-07-23T04:39:50.320118 PSF sigmas in coadd pixels: ['2.88', '3.25', '3.25'] Using average PSF sigma 3.124353901293587 Gaussian PSF norm 0.09028900076175157 vs pixelized 0.08135869 average tim pixel scale / coadd scale: 1.7303248728760552 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-copsf-g.fits.fz PSF sigmas in coadd pixels: ['1.85'] Using average PSF sigma 1.8481836232792843 Gaussian PSF norm 0.15263353068422358 vs pixelized 0.12905155 average tim pixel scale / coadd scale: 1.7303711279714835 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-copsf-r.fits.fz PSF sigmas in coadd pixels: ['2.02', '1.59', '1.56', '2.61'] Using average PSF sigma 1.9462635251182068 Gaussian PSF norm 0.14494172455744145 vs pixelized 0.119682044 average tim pixel scale / coadd scale: 0.9910524269576604 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-copsf-z.fits.fz Stage fit_on_coadds : Wall: 0.15 s, CPU: 0.12 s, VmPeak: 975 MB, VmSize: 968 MB, VmRSS: 201 MB, VmData: 231 MB, maxrss: 0.20582 MB Stage fit_on_coadds finished: 2020-07-23T04:39:50.467436 Warning: Stellar halo subtraction is only implemented for DECam Resources for stage fit_on_coadds : 0.148 s worker CPU, 0.148 s worker Wall, Wall: 0.149 s, Cores in use: 1.78, Total efficiency (on 16 cores): 11.1 % Stage fit_on_coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-fit_on_coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-fit_on_coadds.p Running stage srcs Running stage srcs at 2020-07-23T04:39:50.483042 SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])] Found 6 potential peaks Of 6 potential peaks: 0 in veto map, 1 cut by saddle test, 0 cut by aper test, 5 kept SED z : 5 new peaks Found 48 potential peaks Of 48 potential peaks: 47 in veto map, 0 cut by saddle test, 0 cut by aper test, 1 kept SED r : 1 new peaks Found 2565 potential peaks Of 2565 potential peaks: 2563 in veto map, 2 cut by saddle test, 0 cut by aper test, 0 kept SED g : 0 new peaks Found 57 potential peaks Of 57 potential peaks: 55 in veto map, 2 cut by saddle test, 0 cut by aper test, 0 kept SED Flat : 0 new peaks Found 67 potential peaks Of 67 potential peaks: 65 in veto map, 2 cut by saddle test, 0 cut by aper test, 0 kept SED Red : 0 new peaks Adding 6 new sources Median coadd sky for g : 4.428973e-05 Median coadd sky for r : -0.00014654305 Median coadd sky for z : 0.0003260696 Stage srcs : Wall: 0.68 s, CPU: 0.64 s, VmPeak: 975 MB, VmSize: 968 MB, VmRSS: 202 MB, VmData: 231 MB, maxrss: 0.206368 MB Stage srcs finished: 2020-07-23T04:39:51.162002 Resources for stage srcs : 0.076 s worker CPU, 0.076 s worker Wall, Wall: 0.681 s, Cores in use: 1.05, Total efficiency (on 16 cores): 6.5 % Stage srcs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-srcs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-srcs.p Running stage fitblobs Warning: Stellar halo subtraction is only implemented for DECam Running stage fitblobs at 2020-07-23T04:39:51.180478 Blob 1 of 6, id: 0, sources: 1, size: 251x71, npix 16136, brick X: 0,251, Y: 0,71, one pixel: 17 0 Blob 2 of 6, id: 1, sources: 2, size: 100x91, npix 6043, brick X: 79,179, Y: 72,163, one pixel: 140 72 Blob 3 of 6, id: 3, sources: 1, size: 22x42, npix 701, brick X: 0,22, Y: 176,218, one pixel: 5 176 Blob 4 of 6, id: 5, sources: 1, size: 35x26, npix 646, brick X: 153,188, Y: 225,251, one pixel: 167 225 Blob 5 of 6, id: 2, sources: 1, size: 26x24, npix 388, brick X: 204,230, Y: 96,120, one pixel: 216 96 Blob 6 of 6, id: 4, sources: 1, size: 22x20, npix 250, brick X: 162,184, Y: 193,213, one pixel: 172 193 Measuring 1 iterative sources Blob 5 finished, total: Wall: 0.36 s, CPU: 0.36 s, VmPeak: 799 MB, VmSize: 718 MB, VmRSS: 140 MB, VmData: 182 MB, maxrss: 0.142944 MB Blob 6 finished, total: Wall: 0.43 s, CPU: 0.43 s, VmPeak: 795 MB, VmSize: 718 MB, VmRSS: 141 MB, VmData: 182 MB, maxrss: 0.144776 MB Blob 4 finished, total: Wall: 0.66 s, CPU: 0.66 s, VmPeak: 664 MB, VmSize: 664 MB, VmRSS: 94 MB, VmData: 142 MB, maxrss: 0.096352 MB Blob 1 finished, total: Wall: 0.81 s, CPU: 0.80 s, VmPeak: 668 MB, VmSize: 664 MB, VmRSS: 95 MB, VmData: 141 MB, maxrss: 0.100728 MB Blob 3 finished, total: Wall: 0.94 s, CPU: 0.94 s, VmPeak: 661 MB, VmSize: 661 MB, VmRSS: 92 MB, VmData: 139 MB, maxrss: 0.094684 MB Known galaxy. Initial shape: EllipseWithPriors(0.25): log r_e=1.82629, ee1=-0.0734554, ee2=0.495109 Blob 2 finished, total: Wall: 2.15 s, CPU: 2.15 s, VmPeak: 795 MB, VmSize: 718 MB, VmRSS: 145 MB, VmData: 182 MB, maxrss: 0.148232 MB Old catalog: 12 New catalog: 8 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/blobs-custom-176643p69935.fits.gz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/all-models-custom-176643p69935.fits Stage fitblobs : Wall: 2.71 s, CPU: 0.49 s, VmPeak: 975 MB, VmSize: 968 MB, VmRSS: 202 MB, VmData: 231 MB, maxrss: 0.206484 MB Stage fitblobs finished: 2020-07-23T04:39:53.890138 Resources for stage fitblobs : 5.426 s worker CPU, 5.579 s worker Wall, Wall: 2.711 s, Cores in use: 2.18, Total efficiency (on 16 cores): 13.6 % Stage fitblobs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-fitblobs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-fitblobs.p Running stage coadds Running stage coadds at 2020-07-23T04:39:53.906823 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-ccds.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-image-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-invvar-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-nexp-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-depth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-galdepth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-psfsize-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-model-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-chi2-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-blobmodel-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-image-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-invvar-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-nexp-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-depth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-galdepth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-psfsize-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-model-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-chi2-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-blobmodel-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-image-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-invvar-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-nexp-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-depth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-galdepth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-psfsize-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-model-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-chi2-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-blobmodel-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-depth.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/tmp-legacysurvey-custom-176643p69935-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/tmp-legacysurvey-custom-176643p69935-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/tmp-legacysurvey-custom-176643p69935-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/tmp-legacysurvey-custom-176643p69935-resid.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-resid.jpg Stage coadds : Wall: 2.56 s, CPU: 1.24 s, VmPeak: 975 MB, VmSize: 968 MB, VmRSS: 202 MB, VmData: 231 MB, maxrss: 0.207124 MB Stage coadds finished: 2020-07-23T04:39:56.465081 Resources for stage coadds : 9.162 s worker CPU, 10.933 s worker Wall, Wall: 2.560 s, Cores in use: 4.06, Total efficiency (on 16 cores): 25.4 % Stage coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-coadds.p Running stage wise_forced Running stage wise_forced at 2020-07-23T04:39:56.475962 Cut to 1 unWISE tiles Reading WISE tile 1758p696 band 1 Reading WISE tile 1758p696 band 2 Reading WISE tile 1758p696 band 4 Reading WISE tile 1758p696 band 3 Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.03 s, CPU: 0.03 s, VmPeak: 799 MB, VmSize: 791 MB, VmRSS: 142 MB, VmData: 175 MB, maxrss: 0.186064 MB Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.02 s, CPU: 0.02 s, VmPeak: 799 MB, VmSize: 789 MB, VmRSS: 140 MB, VmData: 173 MB, maxrss: 0.203572 MB Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w2-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w2-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w2-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w2-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.02 s, CPU: 0.02 s, VmPeak: 794 MB, VmSize: 789 MB, VmRSS: 140 MB, VmData: 173 MB, maxrss: 0.203276 MB Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w1-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w1-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w1-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w1-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.02 s, CPU: 0.02 s, VmPeak: 799 MB, VmSize: 799 MB, VmRSS: 149 MB, VmData: 179 MB, maxrss: 0.2048 MB Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-image-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-model-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-invvar-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-image-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-model-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-invvar-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-image-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-model-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-invvar-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-image-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-model-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-invvar-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/tmp-legacysurvey-custom-176643p69935-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/tmp-legacysurvey-custom-176643p69935-wisemodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-wisemodel.jpg Setting WISE apphot Setting WISE apphot Setting WISE apphot Setting WISE apphot Stage wise_forced : Wall: 2.45 s, CPU: 0.33 s, VmPeak: 975 MB, VmSize: 968 MB, VmRSS: 203 MB, VmData: 231 MB, maxrss: 0.207832 MB Stage wise_forced finished: 2020-07-23T04:39:58.930547 Resources for stage wise_forced : 3.022 s worker CPU, 4.079 s worker Wall, Wall: 2.456 s, Cores in use: 1.37, Total efficiency (on 16 cores): 8.5 % Stage wise_forced finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-wise_forced.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-wise_forced.p Running stage writecat Running stage writecat at 2020-07-23T04:39:58.944168 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69935/legacysurvey-custom-176643p69935-maskbits.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/tractor-i/cus/tractor-custom-176643p69935.fits Reading SFD maps... END Rejected (1), keyrecords following the END keyrecord were ignored. END Rejected (1), keyrecords following the END keyrecord were ignored. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/tractor/cus/tractor-custom-176643p69935.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/tractor/cus/brick-custom-176643p69935.sha256sum Stage writecat : Wall: 1.12 s, CPU: 0.69 s, VmPeak: 1100 MB, VmSize: 972 MB, VmRSS: 205 MB, VmData: 231 MB, maxrss: 0.340708 MB Stage writecat finished: 2020-07-23T04:40:00.067950 Resources for stage writecat : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 1.125 s, Cores in use: 0.62, Total efficiency (on 16 cores): 3.9 % Stage writecat finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-writecat.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-writecat.p All done: 22.482 s worker CPU, 31.746 s worker Wall, Wall: 20.827 s, Cores in use: 1.45, Total efficiency (on 16 cores): 9.1 % Finished galaxy PGC2733107 in 0.386 minutes. Started working on galaxy PGC2733107 at Fri Aug 7 09:28:08 2020 Bands touching this brick, g r z python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 176.64314442568897 69.934906134442 --width 251 --height 251 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 runbrick.py starting at 2020-08-07T09:28:09.109579 legacypipe git version: DR9.5.11-watershed-3-gd509eaaf Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '176.64314442568897', '69.934906134442', '--width', '251', '--height', '251', '--pixscale', '0.262', '--threads', '16', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 176.64314442568897 69.934906134442 --width 251 --height 251 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 Parsed RA,Dec 176.64314442568897 69.934906134442 Starting process 33304 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059232 MB Starting process 33305 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB Starting process 33306 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB Starting process 33307 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB Starting process 33308 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05954 MB Starting process 33309 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059548 MB Starting process 33310 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059548 MB Starting process 33311 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB Starting process 33312 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB Starting process 33313 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 33314 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 33315 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 33316 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 33317 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB Starting process 33318 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB Starting process 33319 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage fit_on_coadds Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-08-07T09:28:09.560862 Keeping 8 CCDs: k4m_171213_080628_ooi_zd_ls9-CCD2 z expnum 150939 exptime 168.104 propid 2016A-0453 seeing 1.24 MJD 58100.338 object MzLS_713560_z mosaic/CP/V4.3/CP20171212/k4m_171213_080628_ooi_zd_ls9.fits.fz k4m_170605_054327_ooi_zd_ls9-CCD2 z expnum 131790 exptime 145.105 propid 2016A-0453 seeing 0.98 MJD 57909.239 object MzLS_713122_z mosaic/CP/V4.3/CP20170604/k4m_170605_054327_ooi_zd_ls9.fits.fz k4m_171124_100323_ooi_zd_ls9-CCD1 z expnum 148712 exptime 106.104 propid 2016A-0453 seeing 0.96 MJD 58081.419 object MzLS_713994_z mosaic/CP/V4.3/CP20171123/k4m_171124_100323_ooi_zd_ls9.fits.fz k4m_170420_092701_ooi_zd_ls9-CCD4 z expnum 123521 exptime 149.103 propid 2016A-0453 seeing 1.61 MJD 57863.394 object MzLS_713121_z mosaic/CP/V4.3/CP20170419/k4m_170420_092701_ooi_zd_ls9.fits.fz ksb_180415_071429_ooi_g_ls9-CCD2 g expnum 82230094 exptime 143.352 propid BASS seeing 1.77 MJD 58223.302 object 45951 90prime/CP/V2.3/CP20180414/ksb_180415_071429_ooi_g_ls9.fits.fz ksb_180508_065824_ooi_r_ls9-CCD2 r expnum 82460121 exptime 95.625 propid NOTBASS seeing 1.14 MJD 58246.291 object 45951 90prime/CP/V2.3/CP20180507/ksb_180508_065824_ooi_r_ls9.fits.fz ksb_180409_080715_ooi_g_ls9-CCD3 g expnum 82170118 exptime 153.09 propid BASS seeing 2.00 MJD 58217.338 object 45963 90prime/CP/V2.3/CP20180408/ksb_180409_080715_ooi_g_ls9.fits.fz ksb_180409_080412_ooi_g_ls9-CCD1 g expnum 82170117 exptime 146.61 propid BASS seeing 2.00 MJD 58217.336 object 45953 90prime/CP/V2.3/CP20180408/ksb_180409_080412_ooi_g_ls9.fits.fz mosaic get_fwhm: self.fwhm = 3.77 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2933 3196 and amp range 2048 4095 and image y 502 761 and amp range 0 2039 Range in image: x 0 264 , y 0 260 (with image size 264 x 260) dzp 0.0044767857 -> scaling image by 1.004131784351374 mosaic get_fwhm: self.fwhm = 3.71 Found 4 relevant rows in amp-corrections file. Found overlap: image x 1772 2026 and amp range 0 2005 and image y 2036 2293 and amp range 0 2039 Range in image: x 0 234 , y 0 4 (with image size 255 x 258) dzp 0.009508088 -> scaling image by 1.0087957299815176 Found overlap: image x 1772 2026 and amp range 2006 4053 and image y 2036 2293 and amp range 0 2039 Range in image: x 234 255 , y 0 4 (with image size 255 x 258) dzp 0.0061123776 -> scaling image by 1.005645584425507 Found overlap: image x 1772 2026 and amp range 0 2005 and image y 2036 2293 and amp range 2040 4078 Range in image: x 0 234 , y 4 258 (with image size 255 x 258) dzp 0.014419514 -> scaling image by 1.0133694453441033 Found overlap: image x 1772 2026 and amp range 2006 4053 and image y 2036 2293 and amp range 2040 4078 Range in image: x 234 255 , y 4 258 (with image size 255 x 258) dzp 0.0036595785 -> scaling image by 1.003376283213225 mosaic get_fwhm: self.fwhm = 6.19 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2984 3244 and amp range 2048 4095 and image y 3995 4054 and amp range 2022 4060 Range in image: x 0 261 , y 0 60 (with image size 261 x 60) dzp -0.02499645 -> scaling image by 0.9772404160512723 mosaic get_fwhm: self.fwhm = 4.79 Found 4 relevant rows in amp-corrections file. Found overlap: image x 199 452 and amp range 0 2047 and image y 2938 3190 and amp range 2040 4078 Range in image: x 0 254 , y 0 253 (with image size 254 x 253) dzp 0.003927584 -> scaling image by 1.0036239891963392 Faked up EXPNUM 82230094 Faked up EXPNUM 82230094 Faked up EXPNUM 82230094 90prime get_fwhm: self.fwhm = 3.91 Found 4 relevant rows in amp-corrections file. Found overlap: image x 316 464 and amp range 0 2015 and image y 2199 2347 and amp range 2048 4095 Range in image: x 0 149 , y 0 149 (with image size 149 x 149) dzp 0.0459271 -> scaling image by 1.0432078344950917 Faked up EXPNUM 82170117 Faked up EXPNUM 82170117 Faked up EXPNUM 82170117 90prime get_fwhm: self.fwhm = 4.41 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2468 2618 and amp range 2016 4031 and image y 228 379 and amp range 0 2047 Range in image: x 0 151 , y 0 152 (with image size 151 x 152) dzp -0.019046262 -> scaling image by 0.9826107140833663 Faked up EXPNUM 82170118 Faked up EXPNUM 82170118 Faked up EXPNUM 82170118 90prime get_fwhm: self.fwhm = 4.41 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2552 2700 and amp range 2016 4031 and image y 3677 3826 and amp range 2048 4095 Range in image: x 0 149 , y 0 150 (with image size 149 x 150) dzp -0.01723723 -> scaling image by 0.9842492857213068 Faked up EXPNUM 82460121 Faked up EXPNUM 82460121 Faked up EXPNUM 82460121 90prime get_fwhm: self.fwhm = 2.51 Found 4 relevant rows in amp-corrections file. Found overlap: image x 335 483 and amp range 0 2015 and image y 2174 2322 and amp range 2048 4095 Range in image: x 0 149 , y 0 149 (with image size 149 x 149) dzp 0.0503617 -> scaling image by 1.047477444828492 Stage tims : Wall: 37.50 s, CPU: 0.88 s, VmPeak: 951 MB, VmSize: 951 MB, VmRSS: 172 MB, VmData: 205 MB, maxrss: 0.176072 MB Stage tims finished: 2020-08-07T09:28:47.061375 Resources for stage tims : 3.727 s worker CPU, 154.285 s worker Wall, Wall: 37.502 s, Cores in use: 0.12, Total efficiency (on 16 cores): 0.8 % Stage tims finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-tims.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-tims.p Running stage refs Running stage refs at 2020-08-07T09:28:47.145451 Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00337.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00391.fits Matched 12 large galaxies to Gaia stars. Matched 0 large galaxies to Tycho-2 stars. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/reference-custom-176643p69934.fits Stage refs : Wall: 1.73 s, CPU: 1.41 s, VmPeak: 981 MB, VmSize: 965 MB, VmRSS: 187 MB, VmData: 220 MB, maxrss: 0.19154 MB Stage refs finished: 2020-08-07T09:28:48.877063 Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 1.733 s, Cores in use: 0.81, Total efficiency (on 16 cores): 5.1 % Stage refs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-refs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-refs.p Running stage outliers Running stage outliers at 2020-08-07T09:28:49.021232 Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outlier-mask-custom-176643p69934.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outliers-pre-custom-176643p69934.jpg Image k4m_170420_092701_ooi_zd_ls9-CCD4 z : masking 1673 positive outlier pixels and 337 negative outlier pixels Image k4m_171213_080628_ooi_zd_ls9-CCD2 z : masking 350 positive outlier pixels and 0 negative outlier pixels Image k4m_170605_054327_ooi_zd_ls9-CCD2 z : masking 0 positive outlier pixels and 270 negative outlier pixels Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outlier-mask-custom-176643p69934.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outliers-post-custom-176643p69934.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outliers-masked-pos-custom-176643p69934.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outliers-masked-neg-custom-176643p69934.jpg Stage outliers : Wall: 0.87 s, CPU: 0.51 s, VmPeak: 981 MB, VmSize: 975 MB, VmRSS: 200 MB, VmData: 230 MB, maxrss: 0.204484 MB Stage outliers finished: 2020-08-07T09:28:49.888358 Resources for stage outliers : 0.591 s worker CPU, 0.663 s worker Wall, Wall: 0.868 s, Cores in use: 1.27, Total efficiency (on 16 cores): 8.0 % Stage outliers finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-outliers.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-outliers.p Running stage halos Running stage halos at 2020-08-07T09:28:49.925825 Stage halos : Wall: 0.23 s, CPU: 0.22 s, VmPeak: 981 MB, VmSize: 975 MB, VmRSS: 200 MB, VmData: 230 MB, maxrss: 0.204624 MB Stage halos finished: 2020-08-07T09:28:50.156637 Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.232 s, Cores in use: 0.95, Total efficiency (on 16 cores): 5.9 % Stage halos finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-halos.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-halos.p Running stage fit_on_coadds Warning: Stellar halo subtraction is only implemented for DECam Running stage fit_on_coadds at 2020-08-07T09:28:50.261305 PSF sigmas in coadd pixels: ['2.88', '3.25', '3.25'] Using average PSF sigma 3.124353901293587 Gaussian PSF norm 0.09028900076175157 vs pixelized 0.08136483 average tim pixel scale / coadd scale: 1.7303248728760552 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-copsf-g.fits.fz PSF sigmas in coadd pixels: ['1.85'] Using average PSF sigma 1.8481836232792843 Gaussian PSF norm 0.15263353068422358 vs pixelized 0.12904963 average tim pixel scale / coadd scale: 1.7303711279714835 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-copsf-r.fits.fz PSF sigmas in coadd pixels: ['2.02', '1.59', '1.56', '2.61'] Using average PSF sigma 1.9462635251182068 Gaussian PSF norm 0.14494172455744145 vs pixelized 0.11968422 average tim pixel scale / coadd scale: 0.9910524269576604 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-copsf-z.fits.fz Stage fit_on_coadds : Wall: 0.20 s, CPU: 0.12 s, VmPeak: 981 MB, VmSize: 975 MB, VmRSS: 202 MB, VmData: 231 MB, maxrss: 0.20694 MB Stage fit_on_coadds finished: 2020-08-07T09:28:50.462372 Resources for stage fit_on_coadds : 0.130 s worker CPU, 0.130 s worker Wall, Wall: 0.202 s, Cores in use: 1.26, Total efficiency (on 16 cores): 7.9 % Stage fit_on_coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-fit_on_coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-fit_on_coadds.p Running stage srcs Running stage srcs at 2020-08-07T09:28:50.493100 SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])] Found 4 potential peaks Of 4 potential peaks: 1 in veto map, 1 cut by saddle test, 0 cut by aper test, 2 kept SED z : 2 new peaks Found 61 potential peaks Of 61 potential peaks: 58 in veto map, 0 cut by saddle test, 0 cut by aper test, 3 kept SED r : 3 new peaks Found 2582 potential peaks Of 2582 potential peaks: 2579 in veto map, 2 cut by saddle test, 0 cut by aper test, 1 kept SED g : 1 new peaks Found 53 potential peaks Of 53 potential peaks: 52 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED Flat : 0 new peaks Found 62 potential peaks Of 62 potential peaks: 61 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED Red : 0 new peaks Adding 6 new sources Median coadd sky for g : 4.4781977e-05 Median coadd sky for r : -0.00012375733 Median coadd sky for z : 0.0002685272 Stage srcs : Wall: 0.42 s, CPU: 0.37 s, VmPeak: 981 MB, VmSize: 975 MB, VmRSS: 203 MB, VmData: 231 MB, maxrss: 0.20758 MB Stage srcs finished: 2020-08-07T09:28:50.914829 Resources for stage srcs : 0.095 s worker CPU, 0.095 s worker Wall, Wall: 0.423 s, Cores in use: 1.10, Total efficiency (on 16 cores): 6.9 % Stage srcs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-srcs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-srcs.p Running stage fitblobs Running stage fitblobs at 2020-08-07T09:28:50.938803 Blob 1 of 6, id: 0, sources: 2, size: 251x79, npix 16357, brick X: 0,251, Y: 0,79, one pixel: 15 0 Blob 2 of 6, id: 1, sources: 1, size: 99x91, npix 6041, brick X: 76,175, Y: 80,171, one pixel: 136 80 Blob 3 of 6, id: 3, sources: 1, size: 18x37, npix 532, brick X: 0,18, Y: 188,225, one pixel: 3 188 Blob 4 of 6, id: 2, sources: 1, size: 25x24, npix 406, brick X: 201,226, Y: 104,128, one pixel: 212 104 Blob 5 of 6, id: 5, sources: 1, size: 30x18, npix 376, brick X: 151,181, Y: 233,251, one pixel: 163 233 Blob 6 of 6, id: 4, sources: 1, size: 21x20, npix 257, brick X: 159,180, Y: 201,221, one pixel: 169 201 kdtree_internal.c:2152:kdtree_build_2_ddd: Data, N, or D is zero Blob 3 finished, total: Wall: 0.23 s, CPU: 0.23 s, VmPeak: 809 MB, VmSize: 725 MB, VmRSS: 142 MB, VmData: 183 MB, maxrss: 0.14586 MB Blob 6 finished, total: Wall: 0.32 s, CPU: 0.32 s, VmPeak: 669 MB, VmSize: 669 MB, VmRSS: 93 MB, VmData: 140 MB, maxrss: 0.09488 MB Blob 4 finished, total: Wall: 0.34 s, CPU: 0.33 s, VmPeak: 808 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.146088 MB Exception in one_blob: brick custom-176643p69934, iblob 1 Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1637, in _bounce_one_blob result = one_blob(X) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 341, in run ras[not_galaxy], decs[not_galaxy], 1./3600.) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match (kd1,kd2) = _buildtrees(x1, x2) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees kd2 = spherematch_c.KdTree(fx2) SystemError: returned NULL without setting an error multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "/usr/local/lib/python/astrometry/util/timingpool.py", line 269, in __call__ R = self.func(*args, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1637, in _bounce_one_blob result = one_blob(X) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 341, in run ras[not_galaxy], decs[not_galaxy], 1./3600.) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match (kd1,kd2) = _buildtrees(x1, x2) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees kd2 = spherematch_c.KdTree(fx2) SystemError: returned NULL without setting an error """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3771, in sys.exit(main()) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3739, in main run_brick(opt.brick, survey, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3295, in run_brick initial_args=initargs, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage R = stagefunc(stage, **Px) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3284, in mystagefunc R = stagefunc(stage, mp=mp, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__ rtn = super(CallGlobalTime, self).__call__(stage, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__ return func(*self.args, **kwa) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1131, in stage_fitblobs r = Riter.next(timeout) File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 748, in next raise value SystemError: returned NULL without setting an error Something went wrong; please check the logfile. ERROR: galaxy PGC2733107; please check the logfile. Finished galaxy PGC2733107 in 0.722 minutes. Started working on galaxy PGC2733107 at Fri Aug 7 12:03:41 2020 Bands touching this brick, g r z python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 176.64314442568897 69.934906134442 --width 251 --height 251 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 runbrick.py starting at 2020-08-07T12:03:42.357732 legacypipe git version: DR9.5.11-watershed-4-gcc619972 Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '176.64314442568897', '69.934906134442', '--width', '251', '--height', '251', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 176.64314442568897 69.934906134442 --width 251 --height 251 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 Parsed RA,Dec 176.64314442568897 69.934906134442 Starting process 12085 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB Starting process 12086 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 12087 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 12088 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 12089 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059408 MB Starting process 12090 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059356 MB Starting process 12091 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059356 MB Starting process 12092 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 12093 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 12094 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 12095 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 12096 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB Starting process 12097 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 12098 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 12099 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 12100 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 12101 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB Starting process 12102 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059276 MB Starting process 12103 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 12104 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 12105 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 12106 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 12107 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB Starting process 12108 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB Starting process 12109 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB Starting process 12110 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB Starting process 12111 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB Starting process 12112 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059308 MB Starting process 12113 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB Starting process 12114 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB Starting process 12115 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB Starting process 12116 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059316 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-srcs.p and forcing stage srcs Runstage fit_on_coadds Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-fit_on_coadds.p and forcing stage fit_on_coadds Runstage halos Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-halos.p and forcing stage halos Runstage outliers Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-outliers.p and forcing stage outliers Runstage refs Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-refs.p and forcing stage refs Runstage tims Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-tims.p and forcing stage tims Running stage tims Running stage tims at 2020-08-07T12:03:42.926039 Keeping 8 CCDs: k4m_171213_080628_ooi_zd_ls9-CCD2 z expnum 150939 exptime 168.104 propid 2016A-0453 seeing 1.24 MJD 58100.338 object MzLS_713560_z mosaic/CP/V4.3/CP20171212/k4m_171213_080628_ooi_zd_ls9.fits.fz k4m_170605_054327_ooi_zd_ls9-CCD2 z expnum 131790 exptime 145.105 propid 2016A-0453 seeing 0.98 MJD 57909.239 object MzLS_713122_z mosaic/CP/V4.3/CP20170604/k4m_170605_054327_ooi_zd_ls9.fits.fz k4m_171124_100323_ooi_zd_ls9-CCD1 z expnum 148712 exptime 106.104 propid 2016A-0453 seeing 0.96 MJD 58081.419 object MzLS_713994_z mosaic/CP/V4.3/CP20171123/k4m_171124_100323_ooi_zd_ls9.fits.fz k4m_170420_092701_ooi_zd_ls9-CCD4 z expnum 123521 exptime 149.103 propid 2016A-0453 seeing 1.61 MJD 57863.394 object MzLS_713121_z mosaic/CP/V4.3/CP20170419/k4m_170420_092701_ooi_zd_ls9.fits.fz ksb_180415_071429_ooi_g_ls9-CCD2 g expnum 82230094 exptime 143.352 propid BASS seeing 1.77 MJD 58223.302 object 45951 90prime/CP/V2.3/CP20180414/ksb_180415_071429_ooi_g_ls9.fits.fz ksb_180508_065824_ooi_r_ls9-CCD2 r expnum 82460121 exptime 95.625 propid NOTBASS seeing 1.14 MJD 58246.291 object 45951 90prime/CP/V2.3/CP20180507/ksb_180508_065824_ooi_r_ls9.fits.fz ksb_180409_080715_ooi_g_ls9-CCD3 g expnum 82170118 exptime 153.09 propid BASS seeing 2.00 MJD 58217.338 object 45963 90prime/CP/V2.3/CP20180408/ksb_180409_080715_ooi_g_ls9.fits.fz ksb_180409_080412_ooi_g_ls9-CCD1 g expnum 82170117 exptime 146.61 propid BASS seeing 2.00 MJD 58217.336 object 45953 90prime/CP/V2.3/CP20180408/ksb_180409_080412_ooi_g_ls9.fits.fz mosaic get_fwhm: self.fwhm = 3.77 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2933 3196 and amp range 2048 4095 and image y 502 761 and amp range 0 2039 Range in image: x 0 264 , y 0 260 (with image size 264 x 260) dzp 0.0044767857 -> scaling image by 1.004131784351374 mosaic get_fwhm: self.fwhm = 3.71 Found 4 relevant rows in amp-corrections file. Found overlap: image x 1772 2026 and amp range 0 2005 and image y 2036 2293 and amp range 0 2039 Range in image: x 0 234 , y 0 4 (with image size 255 x 258) dzp 0.009508088 -> scaling image by 1.0087957299815176 Found overlap: image x 1772 2026 and amp range 2006 4053 and image y 2036 2293 and amp range 0 2039 Range in image: x 234 255 , y 0 4 (with image size 255 x 258) dzp 0.0061123776 -> scaling image by 1.005645584425507 Found overlap: image x 1772 2026 and amp range 0 2005 and image y 2036 2293 and amp range 2040 4078 Range in image: x 0 234 , y 4 258 (with image size 255 x 258) dzp 0.014419514 -> scaling image by 1.0133694453441033 Found overlap: image x 1772 2026 and amp range 2006 4053 and image y 2036 2293 and amp range 2040 4078 Range in image: x 234 255 , y 4 258 (with image size 255 x 258) dzp 0.0036595785 -> scaling image by 1.003376283213225 Faked up EXPNUM 82170117 Faked up EXPNUM 82170117 Faked up EXPNUM 82170117 90prime get_fwhm: self.fwhm = 4.41 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2468 2618 and amp range 2016 4031 and image y 228 379 and amp range 0 2047 Range in image: x 0 151 , y 0 152 (with image size 151 x 152) dzp -0.019046262 -> scaling image by 0.9826107140833663 Faked up EXPNUM 82170118 Faked up EXPNUM 82170118 Faked up EXPNUM 82170118 90prime get_fwhm: self.fwhm = 4.41 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2552 2700 and amp range 2016 4031 and image y 3677 3826 and amp range 2048 4095 Range in image: x 0 149 , y 0 150 (with image size 149 x 150) dzp -0.01723723 -> scaling image by 0.9842492857213068 Faked up EXPNUM 82460121 Faked up EXPNUM 82460121 Faked up EXPNUM 82460121 90prime get_fwhm: self.fwhm = 2.51 Found 4 relevant rows in amp-corrections file. Found overlap: image x 335 483 and amp range 0 2015 and image y 2174 2322 and amp range 2048 4095 Range in image: x 0 149 , y 0 149 (with image size 149 x 149) dzp 0.0503617 -> scaling image by 1.047477444828492 Faked up EXPNUM 82230094 Faked up EXPNUM 82230094 Faked up EXPNUM 82230094 90prime get_fwhm: self.fwhm = 3.91 Found 4 relevant rows in amp-corrections file. Found overlap: image x 316 464 and amp range 0 2015 and image y 2199 2347 and amp range 2048 4095 Range in image: x 0 149 , y 0 149 (with image size 149 x 149) dzp 0.0459271 -> scaling image by 1.0432078344950917 mosaic get_fwhm: self.fwhm = 6.19 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2984 3244 and amp range 2048 4095 and image y 3995 4054 and amp range 2022 4060 Range in image: x 0 261 , y 0 60 (with image size 261 x 60) dzp -0.02499645 -> scaling image by 0.9772404160512723 mosaic get_fwhm: self.fwhm = 4.79 Found 4 relevant rows in amp-corrections file. Found overlap: image x 199 452 and amp range 0 2047 and image y 2938 3190 and amp range 2040 4078 Range in image: x 0 254 , y 0 253 (with image size 254 x 253) dzp 0.003927584 -> scaling image by 1.0036239891963392 Stage tims : Wall: 2.19 s, CPU: 0.81 s, VmPeak: 951 MB, VmSize: 951 MB, VmRSS: 172 MB, VmData: 206 MB, maxrss: 0.176468 MB Stage tims finished: 2020-08-07T12:03:45.112254 Resources for stage tims : 3.950 s worker CPU, 8.293 s worker Wall, Wall: 2.189 s, Cores in use: 2.18, Total efficiency (on 32 cores): 6.8 % Stage tims finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-tims.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-tims.p Running stage refs Running stage refs at 2020-08-07T12:03:45.167056 Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00337.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00391.fits Matched 12 large galaxies to Gaia stars. Matched 0 large galaxies to Tycho-2 stars. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/reference-custom-176643p69934.fits Stage refs : Wall: 1.67 s, CPU: 1.44 s, VmPeak: 981 MB, VmSize: 965 MB, VmRSS: 188 MB, VmData: 220 MB, maxrss: 0.19206 MB Stage refs finished: 2020-08-07T12:03:46.832589 Resources for stage refs : 0.000 s worker CPU, 0.001 s worker Wall, Wall: 1.668 s, Cores in use: 0.87, Total efficiency (on 32 cores): 2.7 % Stage refs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-refs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-refs.p Running stage outliers Running stage outliers at 2020-08-07T12:03:46.852055 Stage outliers : Wall: 0.28 s, CPU: 0.26 s, VmPeak: 981 MB, VmSize: 965 MB, VmRSS: 188 MB, VmData: 220 MB, maxrss: 0.192796 MB Stage outliers finished: 2020-08-07T12:03:47.127693 Resources for stage outliers : 0.000 s worker CPU, 0.001 s worker Wall, Wall: 0.278 s, Cores in use: 0.95, Total efficiency (on 32 cores): 3.0 % Stage outliers finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-outliers.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-outliers.p Running stage halos Running stage halos at 2020-08-07T12:03:47.284578 Stage halos : Wall: 0.21 s, CPU: 0.20 s, VmPeak: 981 MB, VmSize: 965 MB, VmRSS: 191 MB, VmData: 223 MB, maxrss: 0.195628 MB Stage halos finished: 2020-08-07T12:03:47.492441 Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.210 s, Cores in use: 0.95, Total efficiency (on 32 cores): 3.0 % Stage halos finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-halos.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-halos.p Running stage fit_on_coadds Running stage fit_on_coadds at 2020-08-07T12:03:47.564991 PSF sigmas in coadd pixels: ['2.88', '3.25', '3.25'] Using average PSF sigma 3.124353901293587 Gaussian PSF norm 0.09028900076175157 vs pixelized 0.08136484 average tim pixel scale / coadd scale: 1.7303248728760552 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-copsf-g.fits.fz PSF sigmas in coadd pixels: ['1.85'] Using average PSF sigma 1.8481836232792843 Gaussian PSF norm 0.15263353068422358 vs pixelized 0.12904963 average tim pixel scale / coadd scale: 1.7303711279714835 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-copsf-r.fits.fz PSF sigmas in coadd pixels: ['2.02', '1.59', '1.56', '2.61'] Using average PSF sigma 1.9462635251182068 Gaussian PSF norm 0.14494172455744145 vs pixelized 0.11968422 average tim pixel scale / coadd scale: 0.9910524269576604 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-copsf-z.fits.fz Stage fit_on_coadds : Wall: 0.20 s, CPU: 0.12 s, VmPeak: 981 MB, VmSize: 970 MB, VmRSS: 200 MB, VmData: 230 MB, maxrss: 0.205216 MB Stage fit_on_coadds finished: 2020-08-07T12:03:47.764959 Warning: Stellar halo subtraction is only implemented for DECam Resources for stage fit_on_coadds : 0.180 s worker CPU, 0.267 s worker Wall, Wall: 0.202 s, Cores in use: 1.52, Total efficiency (on 32 cores): 4.7 % Stage fit_on_coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-fit_on_coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-fit_on_coadds.p Running stage srcs Running stage srcs at 2020-08-07T12:03:47.776239 SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])] Found 4 potential peaks Of 4 potential peaks: 1 in veto map, 1 cut by saddle test, 0 cut by aper test, 2 kept SED z : 2 new peaks Found 61 potential peaks Of 61 potential peaks: 58 in veto map, 0 cut by saddle test, 0 cut by aper test, 3 kept SED r : 3 new peaks Found 2582 potential peaks Of 2582 potential peaks: 2579 in veto map, 2 cut by saddle test, 0 cut by aper test, 1 kept SED g : 1 new peaks Found 53 potential peaks Of 53 potential peaks: 52 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED Flat : 0 new peaks Found 62 potential peaks Of 62 potential peaks: 61 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED Red : 0 new peaks Adding 6 new sources Median coadd sky for g : 4.4781984e-05 Median coadd sky for r : -0.00012375733 Median coadd sky for z : 0.00026852754 Stage srcs : Wall: 0.46 s, CPU: 0.41 s, VmPeak: 981 MB, VmSize: 970 MB, VmRSS: 201 MB, VmData: 230 MB, maxrss: 0.205864 MB Stage srcs finished: 2020-08-07T12:03:48.237812 Resources for stage srcs : 0.107 s worker CPU, 0.119 s worker Wall, Wall: 0.464 s, Cores in use: 1.11, Total efficiency (on 32 cores): 3.5 % Stage srcs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-srcs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-srcs.p Running stage fitblobs Running stage fitblobs at 2020-08-07T12:03:48.249788 Blob 1 of 6, id: 0, sources: 2, size: 251x79, npix 16357, brick X: 0,251, Y: 0,79, one pixel: 15 0 Blob 2 of 6, id: 1, sources: 1, size: 99x91, npix 6041, brick X: 76,175, Y: 80,171, one pixel: 136 80 Blob 3 of 6, id: 3, sources: 1, size: 18x37, npix 532, brick X: 0,18, Y: 188,225, one pixel: 3 188 Blob 4 of 6, id: 2, sources: 1, size: 25x24, npix 406, brick X: 201,226, Y: 104,128, one pixel: 212 104 Blob 5 of 6, id: 5, sources: 1, size: 30x18, npix 376, brick X: 151,181, Y: 233,251, one pixel: 163 233 Blob 6 of 6, id: 4, sources: 1, size: 21x20, npix 257, brick X: 159,180, Y: 201,221, one pixel: 169 201 Blob 3 finished, total: Wall: 0.06 s, CPU: 0.06 s, VmPeak: 801 MB, VmSize: 724 MB, VmRSS: 141 MB, VmData: 182 MB, maxrss: 0.142628 MB kdtree_internal.c:2152:kdtree_build_2_ddd: Data, N, or D is zero Blob 4 finished, total: Wall: 0.30 s, CPU: 0.30 s, VmPeak: 802 MB, VmSize: 724 MB, VmRSS: 142 MB, VmData: 182 MB, maxrss: 0.145656 MB Warning: Stellar halo subtraction is only implemented for DECam Exception in one_blob: brick custom-176643p69934, iblob 1 Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1641, in _bounce_one_blob result = one_blob(X) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 345, in run ras[not_galaxy], decs[not_galaxy], 1./3600.) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match (kd1,kd2) = _buildtrees(x1, x2) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees kd2 = spherematch_c.KdTree(fx2) SystemError: returned NULL without setting an error multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "/usr/local/lib/python/astrometry/util/timingpool.py", line 269, in __call__ R = self.func(*args, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1641, in _bounce_one_blob result = one_blob(X) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 345, in run ras[not_galaxy], decs[not_galaxy], 1./3600.) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match (kd1,kd2) = _buildtrees(x1, x2) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees kd2 = spherematch_c.KdTree(fx2) SystemError: returned NULL without setting an error """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3775, in sys.exit(main()) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3743, in main run_brick(opt.brick, survey, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3299, in run_brick initial_args=initargs, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage R = stagefunc(stage, **Px) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3288, in mystagefunc R = stagefunc(stage, mp=mp, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__ rtn = super(CallGlobalTime, self).__call__(stage, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__ return func(*self.args, **kwa) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1135, in stage_fitblobs r = Riter.next(timeout) File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 748, in next raise value SystemError: returned NULL without setting an error Something went wrong; please check the logfile. ERROR: galaxy PGC2733107; please check the logfile. Finished galaxy PGC2733107 in 0.124 minutes. Started working on galaxy PGC2733107 at Fri Aug 7 12:31:32 2020 Bands touching this brick, g r z python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 176.64314442568897 69.934906134442 --width 251 --height 251 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 runbrick.py starting at 2020-08-07T12:31:32.823253 legacypipe git version: DR9.5.11-watershed-4-gcc619972 Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '176.64314442568897', '69.934906134442', '--width', '251', '--height', '251', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 176.64314442568897 69.934906134442 --width 251 --height 251 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 Parsed RA,Dec 176.64314442568897 69.934906134442 Starting process 15920 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 15921 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 15922 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 15923 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 15924 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05954 MB Starting process 15925 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059552 MB Starting process 15926 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059552 MB Starting process 15927 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 15928 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 15929 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 15930 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 15931 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB Starting process 15932 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB Starting process 15933 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 15934 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 15935 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 15936 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB Starting process 15937 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059276 MB Starting process 15938 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 15939 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 15940 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 15941 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 15942 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB Starting process 15943 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB Starting process 15944 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB Starting process 15945 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB Starting process 15946 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB Starting process 15947 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059308 MB Starting process 15948 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB Starting process 15949 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB Starting process 15950 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB Starting process 15951 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059316 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage fit_on_coadds Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-08-07T12:31:33.244686 Keeping 8 CCDs: k4m_171213_080628_ooi_zd_ls9-CCD2 z expnum 150939 exptime 168.104 propid 2016A-0453 seeing 1.24 MJD 58100.338 object MzLS_713560_z mosaic/CP/V4.3/CP20171212/k4m_171213_080628_ooi_zd_ls9.fits.fz k4m_170605_054327_ooi_zd_ls9-CCD2 z expnum 131790 exptime 145.105 propid 2016A-0453 seeing 0.98 MJD 57909.239 object MzLS_713122_z mosaic/CP/V4.3/CP20170604/k4m_170605_054327_ooi_zd_ls9.fits.fz k4m_171124_100323_ooi_zd_ls9-CCD1 z expnum 148712 exptime 106.104 propid 2016A-0453 seeing 0.96 MJD 58081.419 object MzLS_713994_z mosaic/CP/V4.3/CP20171123/k4m_171124_100323_ooi_zd_ls9.fits.fz k4m_170420_092701_ooi_zd_ls9-CCD4 z expnum 123521 exptime 149.103 propid 2016A-0453 seeing 1.61 MJD 57863.394 object MzLS_713121_z mosaic/CP/V4.3/CP20170419/k4m_170420_092701_ooi_zd_ls9.fits.fz ksb_180415_071429_ooi_g_ls9-CCD2 g expnum 82230094 exptime 143.352 propid BASS seeing 1.77 MJD 58223.302 object 45951 90prime/CP/V2.3/CP20180414/ksb_180415_071429_ooi_g_ls9.fits.fz ksb_180508_065824_ooi_r_ls9-CCD2 r expnum 82460121 exptime 95.625 propid NOTBASS seeing 1.14 MJD 58246.291 object 45951 90prime/CP/V2.3/CP20180507/ksb_180508_065824_ooi_r_ls9.fits.fz ksb_180409_080715_ooi_g_ls9-CCD3 g expnum 82170118 exptime 153.09 propid BASS seeing 2.00 MJD 58217.338 object 45963 90prime/CP/V2.3/CP20180408/ksb_180409_080715_ooi_g_ls9.fits.fz ksb_180409_080412_ooi_g_ls9-CCD1 g expnum 82170117 exptime 146.61 propid BASS seeing 2.00 MJD 58217.336 object 45953 90prime/CP/V2.3/CP20180408/ksb_180409_080412_ooi_g_ls9.fits.fz Faked up EXPNUM 82170117 Faked up EXPNUM 82170117 Faked up EXPNUM 82170117 90prime get_fwhm: self.fwhm = 4.41 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2468 2618 and amp range 2016 4031 and image y 228 379 and amp range 0 2047 Range in image: x 0 151 , y 0 152 (with image size 151 x 152) dzp -0.019046262 -> scaling image by 0.9826107140833663 Faked up EXPNUM 82230094 Faked up EXPNUM 82230094 Faked up EXPNUM 82230094 90prime get_fwhm: self.fwhm = 3.91 Found 4 relevant rows in amp-corrections file. Found overlap: image x 316 464 and amp range 0 2015 and image y 2199 2347 and amp range 2048 4095 Range in image: x 0 149 , y 0 149 (with image size 149 x 149) dzp 0.0459271 -> scaling image by 1.0432078344950917 Faked up EXPNUM 82170118 Faked up EXPNUM 82170118 Faked up EXPNUM 82170118 90prime get_fwhm: self.fwhm = 4.41 Found 4 relevant rows in amp-corrections file. mosaic get_fwhm: self.fwhm = 3.71 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2552 2700 and amp range 2016 4031 and image y 3677 3826 and amp range 2048 4095 Found overlap: image x 1772 2026 and amp range 0 2005 and image y 2036 2293 and amp range 0 2039 Range in image: x 0 149 , y 0 150 (with image size 149 x 150) Range in image: x 0 234 , y 0 4 (with image size 255 x 258) dzp -0.01723723 -> scaling image by 0.9842492857213068 dzp 0.009508088 -> scaling image by 1.0087957299815176 Found overlap: image x 1772 2026 and amp range 2006 4053 and image y 2036 2293 and amp range 0 2039 Range in image: x 234 255 , y 0 4 (with image size 255 x 258) dzp 0.0061123776 -> scaling image by 1.005645584425507 Found overlap: image x 1772 2026 and amp range 0 2005 and image y 2036 2293 and amp range 2040 4078 Range in image: x 0 234 , y 4 258 (with image size 255 x 258) dzp 0.014419514 -> scaling image by 1.0133694453441033 Found overlap: image x 1772 2026 and amp range 2006 4053 and image y 2036 2293 and amp range 2040 4078 Range in image: x 234 255 , y 4 258 (with image size 255 x 258) dzp 0.0036595785 -> scaling image by 1.003376283213225 mosaic get_fwhm: self.fwhm = 3.77 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2933 3196 and amp range 2048 4095 and image y 502 761 and amp range 0 2039 Range in image: x 0 264 , y 0 260 (with image size 264 x 260) dzp 0.0044767857 -> scaling image by 1.004131784351374 mosaic get_fwhm: self.fwhm = 6.19 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2984 3244 and amp range 2048 4095 and image y 3995 4054 and amp range 2022 4060 Range in image: x 0 261 , y 0 60 (with image size 261 x 60) dzp -0.02499645 -> scaling image by 0.9772404160512723 Faked up EXPNUM 82460121 Faked up EXPNUM 82460121 Faked up EXPNUM 82460121 90prime get_fwhm: self.fwhm = 2.51 Found 4 relevant rows in amp-corrections file. Found overlap: image x 335 483 and amp range 0 2015 and image y 2174 2322 and amp range 2048 4095 Range in image: x 0 149 , y 0 149 (with image size 149 x 149) dzp 0.0503617 -> scaling image by 1.047477444828492 mosaic get_fwhm: self.fwhm = 4.79 Found 4 relevant rows in amp-corrections file. Found overlap: image x 199 452 and amp range 0 2047 and image y 2938 3190 and amp range 2040 4078 Range in image: x 0 254 , y 0 253 (with image size 254 x 253) dzp 0.003927584 -> scaling image by 1.0036239891963392 Stage tims : Wall: 2.29 s, CPU: 0.88 s, VmPeak: 951 MB, VmSize: 951 MB, VmRSS: 172 MB, VmData: 206 MB, maxrss: 0.176444 MB Stage tims finished: 2020-08-07T12:31:35.531770 Resources for stage tims : 4.062 s worker CPU, 8.880 s worker Wall, Wall: 2.289 s, Cores in use: 2.16, Total efficiency (on 32 cores): 6.7 % Stage tims finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-tims.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-tims.p Running stage refs Running stage refs at 2020-08-07T12:31:35.553547 Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00337.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00391.fits Matched 12 large galaxies to Gaia stars. Matched 0 large galaxies to Tycho-2 stars. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/reference-custom-176643p69934.fits Stage refs : Wall: 1.64 s, CPU: 1.37 s, VmPeak: 981 MB, VmSize: 965 MB, VmRSS: 188 MB, VmData: 220 MB, maxrss: 0.192256 MB Stage refs finished: 2020-08-07T12:31:37.197724 Resources for stage refs : 0.000 s worker CPU, 0.001 s worker Wall, Wall: 1.646 s, Cores in use: 0.84, Total efficiency (on 32 cores): 2.6 % Stage refs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-refs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-refs.p Running stage outliers Running stage outliers at 2020-08-07T12:31:37.323265 Stage outliers : Wall: 0.26 s, CPU: 0.23 s, VmPeak: 981 MB, VmSize: 965 MB, VmRSS: 188 MB, VmData: 220 MB, maxrss: 0.192984 MB Stage outliers finished: 2020-08-07T12:31:37.580687 Resources for stage outliers : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 0.260 s, Cores in use: 0.91, Total efficiency (on 32 cores): 2.8 % Stage outliers finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-outliers.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-outliers.p Running stage halos Running stage halos at 2020-08-07T12:31:37.672732 Stage halos : Wall: 0.20 s, CPU: 0.19 s, VmPeak: 981 MB, VmSize: 965 MB, VmRSS: 191 MB, VmData: 223 MB, maxrss: 0.195628 MB Stage halos finished: 2020-08-07T12:31:37.870554 Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.200 s, Cores in use: 0.95, Total efficiency (on 32 cores): 3.0 % Stage halos finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-halos.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-halos.p Running stage fit_on_coadds Warning: Stellar halo subtraction is only implemented for DECam Running stage fit_on_coadds at 2020-08-07T12:31:40.273823 PSF sigmas in coadd pixels: ['2.88', '3.25', '3.25'] Using average PSF sigma 3.124353901293587 Gaussian PSF norm 0.09028900076175157 vs pixelized 0.08136483 average tim pixel scale / coadd scale: 1.7303248728760552 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-copsf-g.fits.fz PSF sigmas in coadd pixels: ['1.85'] Using average PSF sigma 1.8481836232792843 Gaussian PSF norm 0.15263353068422358 vs pixelized 0.12904963 average tim pixel scale / coadd scale: 1.7303711279714835 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-copsf-r.fits.fz PSF sigmas in coadd pixels: ['2.02', '1.59', '1.56', '2.61'] Using average PSF sigma 1.9462635251182068 Gaussian PSF norm 0.14494172455744145 vs pixelized 0.11968422 average tim pixel scale / coadd scale: 0.9910524269576604 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-copsf-z.fits.fz Stage fit_on_coadds : Wall: 0.15 s, CPU: 0.12 s, VmPeak: 981 MB, VmSize: 970 MB, VmRSS: 201 MB, VmData: 230 MB, maxrss: 0.20536 MB Stage fit_on_coadds finished: 2020-08-07T12:31:40.426843 Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Resources for stage fit_on_coadds : 0.157 s worker CPU, 0.243 s worker Wall, Wall: 0.155 s, Cores in use: 1.82, Total efficiency (on 32 cores): 5.7 % Stage fit_on_coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-fit_on_coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-fit_on_coadds.p Running stage srcs Running stage srcs at 2020-08-07T12:31:40.439837 SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])] Found 4 potential peaks Of 4 potential peaks: 1 in veto map, 1 cut by saddle test, 0 cut by aper test, 2 kept SED z : 2 new peaks Found 61 potential peaks Of 61 potential peaks: 58 in veto map, 0 cut by saddle test, 0 cut by aper test, 3 kept SED r : 3 new peaks Found 2572 potential peaks Of 2572 potential peaks: 2569 in veto map, 2 cut by saddle test, 0 cut by aper test, 1 kept SED g : 1 new peaks Found 53 potential peaks Of 53 potential peaks: 52 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED Flat : 0 new peaks Found 62 potential peaks Of 62 potential peaks: 61 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED Red : 0 new peaks Adding 6 new sources Median coadd sky for g : 4.4781962e-05 Median coadd sky for r : -0.00012375733 Median coadd sky for z : 0.00026852736 Stage srcs : Wall: 0.50 s, CPU: 0.45 s, VmPeak: 981 MB, VmSize: 970 MB, VmRSS: 201 MB, VmData: 230 MB, maxrss: 0.205864 MB Stage srcs finished: 2020-08-07T12:31:40.944119 Resources for stage srcs : 0.098 s worker CPU, 0.111 s worker Wall, Wall: 0.507 s, Cores in use: 1.09, Total efficiency (on 32 cores): 3.4 % Stage srcs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-srcs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-srcs.p Running stage fitblobs Running stage fitblobs at 2020-08-07T12:31:40.956820 Blob 1 of 6, id: 0, sources: 2, size: 251x79, npix 16357, brick X: 0,251, Y: 0,79, one pixel: 15 0 Blob 2 of 6, id: 1, sources: 1, size: 99x91, npix 6041, brick X: 76,175, Y: 80,171, one pixel: 136 80 Blob 3 of 6, id: 3, sources: 1, size: 18x37, npix 532, brick X: 0,18, Y: 188,225, one pixel: 3 188 Blob 4 of 6, id: 2, sources: 1, size: 25x24, npix 406, brick X: 201,226, Y: 104,128, one pixel: 212 104 Blob 5 of 6, id: 5, sources: 1, size: 30x18, npix 376, brick X: 151,181, Y: 233,251, one pixel: 163 233 Blob 6 of 6, id: 4, sources: 1, size: 21x20, npix 257, brick X: 159,180, Y: 201,221, one pixel: 169 201 Blob 3 finished, total: Wall: 0.11 s, CPU: 0.08 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 82 MB, VmData: 137 MB, maxrss: 0.084232 MB kdtree_internal.c:2152:kdtree_build_2_ddd: Data, N, or D is zero Exception in one_blob: brick custom-176643p69934, iblob 1 Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1641, in _bounce_one_blob result = one_blob(X) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 345, in run ras[not_galaxy], decs[not_galaxy], 1./3600.) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match (kd1,kd2) = _buildtrees(x1, x2) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees kd2 = spherematch_c.KdTree(fx2) SystemError: returned NULL without setting an error multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "/usr/local/lib/python/astrometry/util/timingpool.py", line 269, in __call__ R = self.func(*args, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1641, in _bounce_one_blob result = one_blob(X) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 345, in run ras[not_galaxy], decs[not_galaxy], 1./3600.) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match (kd1,kd2) = _buildtrees(x1, x2) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees kd2 = spherematch_c.KdTree(fx2) SystemError: returned NULL without setting an error """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3775, in sys.exit(main()) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3743, in main run_brick(opt.brick, survey, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3299, in run_brick initial_args=initargs, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage R = stagefunc(stage, **Px) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3288, in mystagefunc R = stagefunc(stage, mp=mp, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__ rtn = super(CallGlobalTime, self).__call__(stage, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__ return func(*self.args, **kwa) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1135, in stage_fitblobs r = Riter.next(timeout) File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 748, in next raise value SystemError: returned NULL without setting an error Something went wrong; please check the logfile. ERROR: galaxy PGC2733107; please check the logfile. Finished galaxy PGC2733107 in 0.220 minutes. Started working on galaxy PGC2733107 at Fri Aug 7 13:31:08 2020 Bands touching this brick, g r z python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 176.64314442568897 69.934906134442 --width 251 --height 251 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 runbrick.py starting at 2020-08-07T13:31:09.794134 legacypipe git version: DR9.5.11-watershed-5-g77ea3463 Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '176.64314442568897', '69.934906134442', '--width', '251', '--height', '251', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 176.64314442568897 69.934906134442 --width 251 --height 251 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 Parsed RA,Dec 176.64314442568897 69.934906134442 Starting process 19609 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB Starting process 19610 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 19611 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 19612 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 19613 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05954 MB Starting process 19614 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059552 MB Starting process 19615 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059552 MB Starting process 19616 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 19617 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 19618 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 19619 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB Starting process 19620 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 19621 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 19622 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 19623 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 19624 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 19625 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB Starting process 19626 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059276 MB Starting process 19627 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 19628 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 19629 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 19630 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB Starting process 19631 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB Starting process 19632 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB Starting process 19633 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB Starting process 19634 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB Starting process 19635 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB Starting process 19636 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059308 MB Starting process 19637 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB Starting process 19638 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB Starting process 19639 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB Starting process 19640 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059316 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage fit_on_coadds Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-08-07T13:31:10.331733 Keeping 8 CCDs: k4m_171213_080628_ooi_zd_ls9-CCD2 z expnum 150939 exptime 168.104 propid 2016A-0453 seeing 1.24 MJD 58100.338 object MzLS_713560_z mosaic/CP/V4.3/CP20171212/k4m_171213_080628_ooi_zd_ls9.fits.fz k4m_170605_054327_ooi_zd_ls9-CCD2 z expnum 131790 exptime 145.105 propid 2016A-0453 seeing 0.98 MJD 57909.239 object MzLS_713122_z mosaic/CP/V4.3/CP20170604/k4m_170605_054327_ooi_zd_ls9.fits.fz k4m_171124_100323_ooi_zd_ls9-CCD1 z expnum 148712 exptime 106.104 propid 2016A-0453 seeing 0.96 MJD 58081.419 object MzLS_713994_z mosaic/CP/V4.3/CP20171123/k4m_171124_100323_ooi_zd_ls9.fits.fz k4m_170420_092701_ooi_zd_ls9-CCD4 z expnum 123521 exptime 149.103 propid 2016A-0453 seeing 1.61 MJD 57863.394 object MzLS_713121_z mosaic/CP/V4.3/CP20170419/k4m_170420_092701_ooi_zd_ls9.fits.fz ksb_180415_071429_ooi_g_ls9-CCD2 g expnum 82230094 exptime 143.352 propid BASS seeing 1.77 MJD 58223.302 object 45951 90prime/CP/V2.3/CP20180414/ksb_180415_071429_ooi_g_ls9.fits.fz ksb_180508_065824_ooi_r_ls9-CCD2 r expnum 82460121 exptime 95.625 propid NOTBASS seeing 1.14 MJD 58246.291 object 45951 90prime/CP/V2.3/CP20180507/ksb_180508_065824_ooi_r_ls9.fits.fz ksb_180409_080715_ooi_g_ls9-CCD3 g expnum 82170118 exptime 153.09 propid BASS seeing 2.00 MJD 58217.338 object 45963 90prime/CP/V2.3/CP20180408/ksb_180409_080715_ooi_g_ls9.fits.fz ksb_180409_080412_ooi_g_ls9-CCD1 g expnum 82170117 exptime 146.61 propid BASS seeing 2.00 MJD 58217.336 object 45953 90prime/CP/V2.3/CP20180408/ksb_180409_080412_ooi_g_ls9.fits.fz mosaic get_fwhm: self.fwhm = 3.71 Found 4 relevant rows in amp-corrections file. Found overlap: image x 1772 2026 and amp range 0 2005 and image y 2036 2293 and amp range 0 2039 Range in image: x 0 234 , y 0 4 (with image size 255 x 258) dzp 0.009508088 -> scaling image by 1.0087957299815176 Found overlap: image x 1772 2026 and amp range 2006 4053 and image y 2036 2293 and amp range 0 2039 Range in image: x 234 255 , y 0 4 (with image size 255 x 258) dzp 0.0061123776 -> scaling image by 1.005645584425507 Found overlap: image x 1772 2026 and amp range 0 2005 and image y 2036 2293 and amp range 2040 4078 Range in image: x 0 234 , y 4 258 (with image size 255 x 258) dzp 0.014419514 -> scaling image by 1.0133694453441033 Found overlap: image x 1772 2026 and amp range 2006 4053 and image y 2036 2293 and amp range 2040 4078 Range in image: x 234 255 , y 4 258 (with image size 255 x 258) dzp 0.0036595785 -> scaling image by 1.003376283213225 mosaic get_fwhm: self.fwhm = 3.77 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2933 3196 and amp range 2048 4095 and image y 502 761 and amp range 0 2039 Range in image: x 0 264 , y 0 260 (with image size 264 x 260) dzp 0.0044767857 -> scaling image by 1.004131784351374 Faked up EXPNUM 82170118 Faked up EXPNUM 82170118 Faked up EXPNUM 82170118 90prime get_fwhm: self.fwhm = 4.41 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2552 2700 and amp range 2016 4031 and image y 3677 3826 and amp range 2048 4095 Range in image: x 0 149 , y 0 150 (with image size 149 x 150) dzp -0.01723723 -> scaling image by 0.9842492857213068 Faked up EXPNUM 82460121 Faked up EXPNUM 82460121 Faked up EXPNUM 82460121 90prime get_fwhm: self.fwhm = 2.51 Found 4 relevant rows in amp-corrections file. Found overlap: image x 335 483 and amp range 0 2015 and image y 2174 2322 and amp range 2048 4095 Range in image: x 0 149 , y 0 149 (with image size 149 x 149) dzp 0.0503617 -> scaling image by 1.047477444828492 Faked up EXPNUM 82170117 Faked up EXPNUM 82170117 Faked up EXPNUM 82170117 90prime get_fwhm: self.fwhm = 4.41 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2468 2618 and amp range 2016 4031 and image y 228 379 and amp range 0 2047 Range in image: x 0 151 , y 0 152 (with image size 151 x 152) dzp -0.019046262 -> scaling image by 0.9826107140833663 mosaic get_fwhm: self.fwhm = 4.79 Found 4 relevant rows in amp-corrections file. Found overlap: image x 199 452 and amp range 0 2047 and image y 2938 3190 and amp range 2040 4078 Range in image: x 0 254 , y 0 253 (with image size 254 x 253) dzp 0.003927584 -> scaling image by 1.0036239891963392 Faked up EXPNUM 82230094 Faked up EXPNUM 82230094 Faked up EXPNUM 82230094 90prime get_fwhm: self.fwhm = 3.91 Found 4 relevant rows in amp-corrections file. Found overlap: image x 316 464 and amp range 0 2015 and image y 2199 2347 and amp range 2048 4095 Range in image: x 0 149 , y 0 149 (with image size 149 x 149) dzp 0.0459271 -> scaling image by 1.0432078344950917 mosaic get_fwhm: self.fwhm = 6.19 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2984 3244 and amp range 2048 4095 and image y 3995 4054 and amp range 2022 4060 Range in image: x 0 261 , y 0 60 (with image size 261 x 60) dzp -0.02499645 -> scaling image by 0.9772404160512723 Stage tims : Wall: 5.29 s, CPU: 0.86 s, VmPeak: 951 MB, VmSize: 951 MB, VmRSS: 172 MB, VmData: 205 MB, maxrss: 0.176136 MB Stage tims finished: 2020-08-07T13:31:15.623527 Resources for stage tims : 3.780 s worker CPU, 12.351 s worker Wall, Wall: 5.295 s, Cores in use: 0.88, Total efficiency (on 32 cores): 2.7 % Stage tims finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-tims.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-tims.p Running stage refs Running stage refs at 2020-08-07T13:31:15.882512 Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00337.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00391.fits Matched 12 large galaxies to Gaia stars. Matched 0 large galaxies to Tycho-2 stars. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/reference-custom-176643p69934.fits Stage refs : Wall: 2.09 s, CPU: 1.45 s, VmPeak: 981 MB, VmSize: 965 MB, VmRSS: 187 MB, VmData: 220 MB, maxrss: 0.191748 MB Stage refs finished: 2020-08-07T13:31:17.969057 Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 2.089 s, Cores in use: 0.70, Total efficiency (on 32 cores): 2.2 % Stage refs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-refs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-refs.p Running stage outliers Running stage outliers at 2020-08-07T13:31:18.002171 Stage outliers : Wall: 0.37 s, CPU: 0.34 s, VmPeak: 981 MB, VmSize: 965 MB, VmRSS: 188 MB, VmData: 220 MB, maxrss: 0.192472 MB Stage outliers finished: 2020-08-07T13:31:18.371177 Resources for stage outliers : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 0.371 s, Cores in use: 0.94, Total efficiency (on 32 cores): 2.9 % Stage outliers finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-outliers.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-outliers.p Running stage halos Running stage halos at 2020-08-07T13:31:18.398946 Stage halos : Wall: 0.27 s, CPU: 0.26 s, VmPeak: 981 MB, VmSize: 965 MB, VmRSS: 191 MB, VmData: 223 MB, maxrss: 0.195112 MB Stage halos finished: 2020-08-07T13:31:18.666347 Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.270 s, Cores in use: 0.97, Total efficiency (on 32 cores): 3.0 % Stage halos finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-halos.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-halos.p Running stage fit_on_coadds Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Running stage fit_on_coadds at 2020-08-07T13:31:18.685634 PSF sigmas in coadd pixels: ['2.88', '3.25', '3.25'] Using average PSF sigma 3.124353901293587 Gaussian PSF norm 0.09028900076175157 vs pixelized 0.08136483 average tim pixel scale / coadd scale: 1.7303248728760552 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-copsf-g.fits.fz PSF sigmas in coadd pixels: ['1.85'] Using average PSF sigma 1.8481836232792843 Gaussian PSF norm 0.15263353068422358 vs pixelized 0.12904963 average tim pixel scale / coadd scale: 1.7303711279714835 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-copsf-r.fits.fz PSF sigmas in coadd pixels: ['2.02', '1.59', '1.56', '2.61'] Using average PSF sigma 1.9462635251182068 Gaussian PSF norm 0.14494172455744145 vs pixelized 0.11968422 average tim pixel scale / coadd scale: 0.9910524269576604 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-copsf-z.fits.fz Stage fit_on_coadds : Wall: 0.21 s, CPU: 0.12 s, VmPeak: 981 MB, VmSize: 970 MB, VmRSS: 201 MB, VmData: 230 MB, maxrss: 0.205328 MB Stage fit_on_coadds finished: 2020-08-07T13:31:18.893179 Warning: Stellar halo subtraction is only implemented for DECam Resources for stage fit_on_coadds : 0.171 s worker CPU, 0.259 s worker Wall, Wall: 0.210 s, Cores in use: 1.41, Total efficiency (on 32 cores): 4.4 % Stage fit_on_coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-fit_on_coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-fit_on_coadds.p Running stage srcs Running stage srcs at 2020-08-07T13:31:18.904050 SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])] Found 4 potential peaks Of 4 potential peaks: 1 in veto map, 1 cut by saddle test, 0 cut by aper test, 2 kept SED z : 2 new peaks Found 61 potential peaks Of 61 potential peaks: 58 in veto map, 0 cut by saddle test, 0 cut by aper test, 3 kept SED r : 3 new peaks Found 2582 potential peaks Of 2582 potential peaks: 2579 in veto map, 2 cut by saddle test, 0 cut by aper test, 1 kept SED g : 1 new peaks Found 53 potential peaks Of 53 potential peaks: 52 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED Flat : 0 new peaks Found 62 potential peaks Of 62 potential peaks: 61 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED Red : 0 new peaks Adding 6 new sources Median coadd sky for g : 4.4781977e-05 Median coadd sky for r : -0.00012375733 Median coadd sky for z : 0.00026852754 Stage srcs : Wall: 0.67 s, CPU: 0.47 s, VmPeak: 981 MB, VmSize: 970 MB, VmRSS: 201 MB, VmData: 230 MB, maxrss: 0.205844 MB Stage srcs finished: 2020-08-07T13:31:19.569168 Resources for stage srcs : 0.107 s worker CPU, 0.118 s worker Wall, Wall: 0.667 s, Cores in use: 0.86, Total efficiency (on 32 cores): 2.7 % Stage srcs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-srcs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-srcs.p Running stage fitblobs Running stage fitblobs at 2020-08-07T13:31:19.581260 Blob 1 of 6, id: 0, sources: 2, size: 251x79, npix 16357, brick X: 0,251, Y: 0,79, one pixel: 15 0 Blob 2 of 6, id: 1, sources: 1, size: 99x91, npix 6041, brick X: 76,175, Y: 80,171, one pixel: 136 80 Blob 3 of 6, id: 3, sources: 1, size: 18x37, npix 532, brick X: 0,18, Y: 188,225, one pixel: 3 188 Blob 4 of 6, id: 2, sources: 1, size: 25x24, npix 406, brick X: 201,226, Y: 104,128, one pixel: 212 104 Blob 5 of 6, id: 5, sources: 1, size: 30x18, npix 376, brick X: 151,181, Y: 233,251, one pixel: 163 233 Blob 6 of 6, id: 4, sources: 1, size: 21x20, npix 257, brick X: 159,180, Y: 201,221, one pixel: 169 201 Blob 3 finished, total: Wall: 0.11 s, CPU: 0.07 s, VmPeak: 809 MB, VmSize: 725 MB, VmRSS: 142 MB, VmData: 183 MB, maxrss: 0.145132 MB kdtree_internal.c:2152:kdtree_build_2_ddd: Data, N, or D is zero Exception in one_blob: brick custom-176643p69934, iblob 1 Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1641, in _bounce_one_blob result = one_blob(X) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 345, in run ras[not_galaxy], decs[not_galaxy], 1./3600.) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match (kd1,kd2) = _buildtrees(x1, x2) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees kd2 = spherematch_c.KdTree(fx2) SystemError: returned NULL without setting an error multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "/usr/local/lib/python/astrometry/util/timingpool.py", line 269, in __call__ R = self.func(*args, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1641, in _bounce_one_blob result = one_blob(X) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 345, in run ras[not_galaxy], decs[not_galaxy], 1./3600.) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match (kd1,kd2) = _buildtrees(x1, x2) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees kd2 = spherematch_c.KdTree(fx2) SystemError: returned NULL without setting an error """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3775, in sys.exit(main()) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3743, in main run_brick(opt.brick, survey, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3299, in run_brick initial_args=initargs, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage R = stagefunc(stage, **Px) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3288, in mystagefunc R = stagefunc(stage, mp=mp, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__ rtn = super(CallGlobalTime, self).__call__(stage, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__ return func(*self.args, **kwa) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1135, in stage_fitblobs r = Riter.next(timeout) File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 748, in next raise value SystemError: returned NULL without setting an error Something went wrong; please check the logfile. ERROR: galaxy PGC2733107; please check the logfile. Finished galaxy PGC2733107 in 0.192 minutes. Started working on galaxy PGC2733107 at Fri Aug 7 14:33:28 2020 Bands touching this brick, g r z python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 176.64314442568897 69.934906134442 --width 251 --height 251 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 runbrick.py starting at 2020-08-07T14:33:29.031583 legacypipe git version: DR9.5.11-watershed-6-g6a2880fc Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '176.64314442568897', '69.934906134442', '--width', '251', '--height', '251', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 176.64314442568897 69.934906134442 --width 251 --height 251 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 Parsed RA,Dec 176.64314442568897 69.934906134442 Starting process 48146 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059232 MB Starting process 48147 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB Starting process 48148 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB Starting process 48149 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB Starting process 48150 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB Starting process 48151 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 48152 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 48153 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB Starting process 48154 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB Starting process 48155 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 48156 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 48157 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 48158 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 48159 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 48160 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB Starting process 48161 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 48162 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB Starting process 48163 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB Starting process 48164 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059276 MB Starting process 48165 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059276 MB Starting process 48166 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059276 MB Starting process 48167 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05928 MB Starting process 48168 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB Starting process 48169 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB Starting process 48170 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB Starting process 48171 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB Starting process 48172 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB Starting process 48173 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB Starting process 48174 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB Starting process 48175 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB Starting process 48176 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB Starting process 48177 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Reading pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-srcs.p Running stage fitblobs Running stage fitblobs at 2020-08-07T14:33:29.534271 Blob 1 of 6, id: 0, sources: 2, size: 251x79, npix 16357, brick X: 0,251, Y: 0,79, one pixel: 15 0 Blob 2 of 6, id: 1, sources: 1, size: 99x91, npix 6041, brick X: 76,175, Y: 80,171, one pixel: 136 80 Blob 3 of 6, id: 3, sources: 1, size: 18x37, npix 532, brick X: 0,18, Y: 188,225, one pixel: 3 188 Blob 4 of 6, id: 2, sources: 1, size: 25x24, npix 406, brick X: 201,226, Y: 104,128, one pixel: 212 104 Blob 5 of 6, id: 5, sources: 1, size: 30x18, npix 376, brick X: 151,181, Y: 233,251, one pixel: 163 233 Blob 6 of 6, id: 4, sources: 1, size: 21x20, npix 257, brick X: 159,180, Y: 201,221, one pixel: 169 201 Blob 3 finished, total: Wall: 0.08 s, CPU: 0.08 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 82 MB, VmData: 136 MB, maxrss: 0.084096 MB Blob 4 finished, total: Wall: 0.33 s, CPU: 0.32 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.089684 MB Blob 6 finished, total: Wall: 0.40 s, CPU: 0.36 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 87 MB, VmData: 137 MB, maxrss: 0.089548 MB Blob 5 finished, total: Wall: 0.48 s, CPU: 0.47 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.089688 MB Blob 1 finished, total: Wall: 1.01 s, CPU: 1.00 s, VmPeak: 670 MB, VmSize: 668 MB, VmRSS: 92 MB, VmData: 139 MB, maxrss: 0.09592 MB Known galaxy. Initial shape: EllipseWithPriors(0.25): log r_e=1.6938, ee1=-0.0719171, ee2=0.472062 Blob 2 finished, total: Wall: 1.26 s, CPU: 1.25 s, VmPeak: 675 MB, VmSize: 671 MB, VmRSS: 96 MB, VmData: 142 MB, maxrss: 0.101848 MB Old catalog: 11 New catalog: 5 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/blobs-custom-176643p69934.fits.gz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/all-models-custom-176643p69934.fits Stage fitblobs : Wall: 1.54 s, CPU: 0.26 s, VmPeak: 851 MB, VmSize: 848 MB, VmRSS: 110 MB, VmData: 145 MB, maxrss: 0.200352 MB Stage fitblobs finished: 2020-08-07T14:33:31.074601 Resources for stage fitblobs : 3.555 s worker CPU, 3.669 s worker Wall, Wall: 1.543 s, Cores in use: 2.47, Total efficiency (on 32 cores): 7.7 % Stage fitblobs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-fitblobs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-fitblobs.p Running stage coadds Running stage coadds at 2020-08-07T14:33:31.103963 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-ccds.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-image-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-invvar-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-nexp-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-depth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-galdepth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-psfsize-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-model-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-chi2-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-blobmodel-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-image-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-invvar-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-nexp-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-depth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-galdepth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-psfsize-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-model-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-chi2-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-blobmodel-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-image-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-invvar-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-nexp-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-depth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-galdepth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-psfsize-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-model-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-chi2-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-blobmodel-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-depth.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/tmp-legacysurvey-custom-176643p69934-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/tmp-legacysurvey-custom-176643p69934-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/tmp-legacysurvey-custom-176643p69934-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/tmp-legacysurvey-custom-176643p69934-resid.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-resid.jpg Stage coadds : Wall: 3.34 s, CPU: 0.82 s, VmPeak: 897 MB, VmSize: 887 MB, VmRSS: 141 MB, VmData: 173 MB, maxrss: 0.200352 MB Stage coadds finished: 2020-08-07T14:33:34.448305 Resources for stage coadds : 48.106 s worker CPU, 59.975 s worker Wall, Wall: 3.347 s, Cores in use: 14.62, Total efficiency (on 32 cores): 45.7 % Stage coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-coadds.p Running stage wise_forced Running stage wise_forced at 2020-08-07T14:33:34.460483 Cut to 1 unWISE tiles Reading WISE tile 1758p696 band 2 Reading WISE tile 1758p696 band 1 Reading WISE tile 1758p696 band 4 Reading WISE tile 1758p696 band 3 Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.02 s, CPU: 0.02 s, VmPeak: 798 MB, VmSize: 798 MB, VmRSS: 142 MB, VmData: 178 MB, maxrss: 0.179576 MB Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.02 s, CPU: 0.02 s, VmPeak: 789 MB, VmSize: 789 MB, VmRSS: 132 MB, VmData: 169 MB, maxrss: 0.19166 MB Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w2-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w2-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w2-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w2-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.02 s, CPU: 0.02 s, VmPeak: 861 MB, VmSize: 861 MB, VmRSS: 205 MB, VmData: 237 MB, maxrss: 0.209656 MB Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w1-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w1-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w1-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w1-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.02 s, CPU: 0.02 s, VmPeak: 800 MB, VmSize: 795 MB, VmRSS: 138 MB, VmData: 171 MB, maxrss: 0.20156 MB Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-image-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-model-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-invvar-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-image-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-model-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-invvar-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-image-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-model-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-invvar-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-image-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-model-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-invvar-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/tmp-legacysurvey-custom-176643p69934-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/tmp-legacysurvey-custom-176643p69934-wisemodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-wisemodel.jpg Setting WISE apphot Setting WISE apphot Setting WISE apphot Setting WISE apphot Stage wise_forced : Wall: 1.98 s, CPU: 0.54 s, VmPeak: 923 MB, VmSize: 923 MB, VmRSS: 156 MB, VmData: 186 MB, maxrss: 0.200352 MB Stage wise_forced finished: 2020-08-07T14:33:36.438259 Resources for stage wise_forced : 2.973 s worker CPU, 4.080 s worker Wall, Wall: 1.980 s, Cores in use: 1.78, Total efficiency (on 32 cores): 5.5 % Stage wise_forced finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-wise_forced.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-wise_forced.p Running stage writecat Running stage writecat at 2020-08-07T14:33:36.451651 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-maskbits.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/tractor-i/cus/tractor-custom-176643p69934.fits Reading SFD maps... END Rejected (1), keyrecords following the END keyrecord were ignored. END Rejected (1), keyrecords following the END keyrecord were ignored. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/tractor/cus/tractor-custom-176643p69934.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/tractor/cus/brick-custom-176643p69934.sha256sum Stage writecat : Wall: 0.54 s, CPU: 0.39 s, VmPeak: 1067 MB, VmSize: 939 MB, VmRSS: 172 MB, VmData: 199 MB, maxrss: 0.307448 MB Stage writecat finished: 2020-08-07T14:33:36.994155 Resources for stage writecat : 0.000 s worker CPU, 0.001 s worker Wall, Wall: 0.545 s, Cores in use: 0.71, Total efficiency (on 32 cores): 2.2 % Stage writecat finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-writecat.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-writecat.p All done: 54.636 s worker CPU, 67.727 s worker Wall, Wall: 7.540 s, Cores in use: 7.52, Total efficiency (on 32 cores): 23.5 % Finished galaxy PGC2733107 in 0.154 minutes. Started working on galaxy PGC2733107 at Fri Aug 7 14:45:04 2020 Bands touching this brick, g r z python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 176.64314442568897 69.934906134442 --width 251 --height 251 --pixscale 0.262 --threads 1 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 runbrick.py starting at 2020-08-07T14:45:05.010171 legacypipe git version: DR9.5.11-watershed-6-g6a2880fc Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '176.64314442568897', '69.934906134442', '--width', '251', '--height', '251', '--pixscale', '0.262', '--threads', '1', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 176.64314442568897 69.934906134442 --width 251 --height 251 --pixscale 0.262 --threads 1 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 Parsed RA,Dec 176.64314442568897 69.934906134442 Starting process 56168 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 641 MB, VmSize: 641 MB, VmRSS: 99 MB, VmData: 128 MB, maxrss: 0.107656 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage fit_on_coadds Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-08-07T14:45:05.512736 Keeping 8 CCDs: k4m_171213_080628_ooi_zd_ls9-CCD2 z expnum 150939 exptime 168.104 propid 2016A-0453 seeing 1.24 MJD 58100.338 object MzLS_713560_z mosaic/CP/V4.3/CP20171212/k4m_171213_080628_ooi_zd_ls9.fits.fz k4m_170605_054327_ooi_zd_ls9-CCD2 z expnum 131790 exptime 145.105 propid 2016A-0453 seeing 0.98 MJD 57909.239 object MzLS_713122_z mosaic/CP/V4.3/CP20170604/k4m_170605_054327_ooi_zd_ls9.fits.fz k4m_171124_100323_ooi_zd_ls9-CCD1 z expnum 148712 exptime 106.104 propid 2016A-0453 seeing 0.96 MJD 58081.419 object MzLS_713994_z mosaic/CP/V4.3/CP20171123/k4m_171124_100323_ooi_zd_ls9.fits.fz k4m_170420_092701_ooi_zd_ls9-CCD4 z expnum 123521 exptime 149.103 propid 2016A-0453 seeing 1.61 MJD 57863.394 object MzLS_713121_z mosaic/CP/V4.3/CP20170419/k4m_170420_092701_ooi_zd_ls9.fits.fz ksb_180415_071429_ooi_g_ls9-CCD2 g expnum 82230094 exptime 143.352 propid BASS seeing 1.77 MJD 58223.302 object 45951 90prime/CP/V2.3/CP20180414/ksb_180415_071429_ooi_g_ls9.fits.fz ksb_180508_065824_ooi_r_ls9-CCD2 r expnum 82460121 exptime 95.625 propid NOTBASS seeing 1.14 MJD 58246.291 object 45951 90prime/CP/V2.3/CP20180507/ksb_180508_065824_ooi_r_ls9.fits.fz ksb_180409_080715_ooi_g_ls9-CCD3 g expnum 82170118 exptime 153.09 propid BASS seeing 2.00 MJD 58217.338 object 45963 90prime/CP/V2.3/CP20180408/ksb_180409_080715_ooi_g_ls9.fits.fz ksb_180409_080412_ooi_g_ls9-CCD1 g expnum 82170117 exptime 146.61 propid BASS seeing 2.00 MJD 58217.336 object 45953 90prime/CP/V2.3/CP20180408/ksb_180409_080412_ooi_g_ls9.fits.fz mosaic get_fwhm: self.fwhm = 4.79 Found 4 relevant rows in amp-corrections file. Found overlap: image x 199 452 and amp range 0 2047 and image y 2938 3190 and amp range 2040 4078 Range in image: x 0 254 , y 0 253 (with image size 254 x 253) dzp 0.003927584 -> scaling image by 1.0036239891963392 mosaic get_fwhm: self.fwhm = 3.77 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2933 3196 and amp range 2048 4095 and image y 502 761 and amp range 0 2039 Range in image: x 0 264 , y 0 260 (with image size 264 x 260) dzp 0.0044767857 -> scaling image by 1.004131784351374 mosaic get_fwhm: self.fwhm = 3.71 Found 4 relevant rows in amp-corrections file. Found overlap: image x 1772 2026 and amp range 0 2005 and image y 2036 2293 and amp range 0 2039 Range in image: x 0 234 , y 0 4 (with image size 255 x 258) dzp 0.009508088 -> scaling image by 1.0087957299815176 Found overlap: image x 1772 2026 and amp range 2006 4053 and image y 2036 2293 and amp range 0 2039 Range in image: x 234 255 , y 0 4 (with image size 255 x 258) dzp 0.0061123776 -> scaling image by 1.005645584425507 Found overlap: image x 1772 2026 and amp range 0 2005 and image y 2036 2293 and amp range 2040 4078 Range in image: x 0 234 , y 4 258 (with image size 255 x 258) dzp 0.014419514 -> scaling image by 1.0133694453441033 Found overlap: image x 1772 2026 and amp range 2006 4053 and image y 2036 2293 and amp range 2040 4078 Range in image: x 234 255 , y 4 258 (with image size 255 x 258) dzp 0.0036595785 -> scaling image by 1.003376283213225 mosaic get_fwhm: self.fwhm = 6.19 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2984 3244 and amp range 2048 4095 and image y 3995 4054 and amp range 2022 4060 Range in image: x 0 261 , y 0 60 (with image size 261 x 60) dzp -0.02499645 -> scaling image by 0.9772404160512723 Faked up EXPNUM 82230094 Faked up EXPNUM 82230094 Faked up EXPNUM 82230094 90prime get_fwhm: self.fwhm = 3.91 Found 4 relevant rows in amp-corrections file. Found overlap: image x 316 464 and amp range 0 2015 and image y 2199 2347 and amp range 2048 4095 Range in image: x 0 149 , y 0 149 (with image size 149 x 149) dzp 0.0459271 -> scaling image by 1.0432078344950917 Faked up EXPNUM 82460121 Faked up EXPNUM 82460121 Faked up EXPNUM 82460121 90prime get_fwhm: self.fwhm = 2.51 Found 4 relevant rows in amp-corrections file. Found overlap: image x 335 483 and amp range 0 2015 and image y 2174 2322 and amp range 2048 4095 Range in image: x 0 149 , y 0 149 (with image size 149 x 149) dzp 0.0503617 -> scaling image by 1.047477444828492 Faked up EXPNUM 82170118 Faked up EXPNUM 82170118 Faked up EXPNUM 82170118 90prime get_fwhm: self.fwhm = 4.41 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2552 2700 and amp range 2016 4031 and image y 3677 3826 and amp range 2048 4095 Range in image: x 0 149 , y 0 150 (with image size 149 x 150) dzp -0.01723723 -> scaling image by 0.9842492857213068 Faked up EXPNUM 82170117 Faked up EXPNUM 82170117 Faked up EXPNUM 82170117 90prime get_fwhm: self.fwhm = 4.41 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2468 2618 and amp range 2016 4031 and image y 228 379 and amp range 0 2047 Range in image: x 0 151 , y 0 152 (with image size 151 x 152) dzp -0.019046262 -> scaling image by 0.9826107140833663 Stage tims : Wall: 6.85 s, CPU: 2.02 s, VmPeak: 859 MB, VmSize: 757 MB, VmRSS: 175 MB, VmData: 195 MB, maxrss: 0.179092 MB Stage tims finished: 2020-08-07T14:45:12.360230 Resources for stage tims : Wall: 6.85 s, CPU: 2.02 s Stage tims finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-tims.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-tims.p Running stage refs Running stage refs at 2020-08-07T14:45:12.428932 Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00337.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00391.fits Matched 12 large galaxies to Gaia stars. Matched 0 large galaxies to Tycho-2 stars. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/reference-custom-176643p69934.fits Stage refs : Wall: 1.77 s, CPU: 1.45 s, VmPeak: 859 MB, VmSize: 772 MB, VmRSS: 190 MB, VmData: 209 MB, maxrss: 0.194072 MB Stage refs finished: 2020-08-07T14:45:14.197108 Resources for stage refs : Wall: 1.77 s, CPU: 1.45 s Stage refs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-refs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-refs.p Running stage outliers Running stage outliers at 2020-08-07T14:45:14.289551 Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outlier-mask-custom-176643p69934.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outliers-pre-custom-176643p69934.jpg Image k4m_171213_080628_ooi_zd_ls9-CCD2 z : masking 350 positive outlier pixels and 0 negative outlier pixels Image k4m_170605_054327_ooi_zd_ls9-CCD2 z : masking 0 positive outlier pixels and 270 negative outlier pixels Image k4m_170420_092701_ooi_zd_ls9-CCD4 z : masking 1673 positive outlier pixels and 337 negative outlier pixels Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outlier-mask-custom-176643p69934.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outliers-post-custom-176643p69934.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outliers-masked-pos-custom-176643p69934.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/outliers-masked-neg-custom-176643p69934.jpg Stage outliers : Wall: 0.86 s, CPU: 0.81 s, VmPeak: 859 MB, VmSize: 776 MB, VmRSS: 192 MB, VmData: 209 MB, maxrss: 0.196168 MB Stage outliers finished: 2020-08-07T14:45:15.148997 Resources for stage outliers : Wall: 0.86 s, CPU: 0.81 s Stage outliers finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-outliers.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-outliers.p Running stage halos Running stage halos at 2020-08-07T14:45:15.215630 Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Stage halos : Wall: 0.28 s, CPU: 0.25 s, VmPeak: 859 MB, VmSize: 776 MB, VmRSS: 192 MB, VmData: 210 MB, maxrss: 0.196352 MB Stage halos finished: 2020-08-07T14:45:15.491029 Resources for stage halos : Wall: 0.28 s, CPU: 0.25 s Stage halos finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-halos.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-halos.p Running stage fit_on_coadds Running stage fit_on_coadds at 2020-08-07T14:45:15.589756 PSF sigmas in coadd pixels: ['2.88', '3.25', '3.25'] Using average PSF sigma 3.124353901293587 Gaussian PSF norm 0.09028900076175157 vs pixelized 0.08136484 average tim pixel scale / coadd scale: 1.7303248728760552 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-copsf-g.fits.fz PSF sigmas in coadd pixels: ['1.85'] Using average PSF sigma 1.8481836232792843 Gaussian PSF norm 0.15263353068422358 vs pixelized 0.12904963 average tim pixel scale / coadd scale: 1.7303711279714835 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-copsf-r.fits.fz PSF sigmas in coadd pixels: ['2.02', '1.59', '1.56', '2.61'] Using average PSF sigma 1.9462635251182068 Gaussian PSF norm 0.14494172455744145 vs pixelized 0.11968423 average tim pixel scale / coadd scale: 0.9910524269576604 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-copsf-z.fits.fz Stage fit_on_coadds : Wall: 0.20 s, CPU: 0.18 s, VmPeak: 859 MB, VmSize: 776 MB, VmRSS: 192 MB, VmData: 210 MB, maxrss: 0.196352 MB Stage fit_on_coadds finished: 2020-08-07T14:45:15.785328 Resources for stage fit_on_coadds : Wall: 0.20 s, CPU: 0.18 s Stage fit_on_coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-fit_on_coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-fit_on_coadds.p Running stage srcs Running stage srcs at 2020-08-07T14:45:15.792328 SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])] Found 4 potential peaks Of 4 potential peaks: 1 in veto map, 1 cut by saddle test, 0 cut by aper test, 2 kept SED z : 2 new peaks Found 61 potential peaks Of 61 potential peaks: 58 in veto map, 0 cut by saddle test, 0 cut by aper test, 3 kept SED r : 3 new peaks Found 2582 potential peaks Of 2582 potential peaks: 2579 in veto map, 2 cut by saddle test, 0 cut by aper test, 1 kept SED g : 1 new peaks Found 53 potential peaks Of 53 potential peaks: 52 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED Flat : 0 new peaks Found 62 potential peaks Of 62 potential peaks: 61 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED Red : 0 new peaks Adding 6 new sources Median coadd sky for g : 4.4781984e-05 Median coadd sky for r : -0.00012375733 Median coadd sky for z : 0.00026852754 Stage srcs : Wall: 0.44 s, CPU: 0.44 s, VmPeak: 859 MB, VmSize: 776 MB, VmRSS: 192 MB, VmData: 210 MB, maxrss: 0.196864 MB Stage srcs finished: 2020-08-07T14:45:16.236208 Resources for stage srcs : Wall: 0.44 s, CPU: 0.44 s Stage srcs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-srcs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-srcs.p Running stage fitblobs Running stage fitblobs at 2020-08-07T14:45:16.244207 Blob 1 of 6, id: 0, sources: 2, size: 251x79, npix 16357, brick X: 0,251, Y: 0,79, one pixel: 15 0 Blob 1 finished, total: Wall: 0.98 s, CPU: 0.95 s, VmPeak: 859 MB, VmSize: 785 MB, VmRSS: 196 MB, VmData: 210 MB, maxrss: 0.200296 MB Blob 2 of 6, id: 1, sources: 1, size: 99x91, npix 6041, brick X: 76,175, Y: 80,171, one pixel: 136 80 Known galaxy. Initial shape: EllipseWithPriors(0.25): log r_e=1.6938, ee1=-0.0719171, ee2=0.472062 Blob 2 finished, total: Wall: 1.12 s, CPU: 1.12 s, VmPeak: 859 MB, VmSize: 785 MB, VmRSS: 196 MB, VmData: 210 MB, maxrss: 0.201028 MB Blob 3 of 6, id: 3, sources: 1, size: 18x37, npix 532, brick X: 0,18, Y: 188,225, one pixel: 3 188 Blob 3 finished, total: Wall: 0.05 s, CPU: 0.05 s, VmPeak: 859 MB, VmSize: 785 MB, VmRSS: 196 MB, VmData: 210 MB, maxrss: 0.201028 MB Blob 4 of 6, id: 2, sources: 1, size: 25x24, npix 406, brick X: 201,226, Y: 104,128, one pixel: 212 104 Blob 4 finished, total: Wall: 0.26 s, CPU: 0.26 s, VmPeak: 859 MB, VmSize: 785 MB, VmRSS: 196 MB, VmData: 210 MB, maxrss: 0.201028 MB Blob 5 of 6, id: 5, sources: 1, size: 30x18, npix 376, brick X: 151,181, Y: 233,251, one pixel: 163 233 Blob 5 finished, total: Wall: 0.42 s, CPU: 0.42 s, VmPeak: 859 MB, VmSize: 785 MB, VmRSS: 196 MB, VmData: 210 MB, maxrss: 0.201028 MB Blob 6 of 6, id: 4, sources: 1, size: 21x20, npix 257, brick X: 159,180, Y: 201,221, one pixel: 169 201 Blob 6 finished, total: Wall: 0.29 s, CPU: 0.29 s, VmPeak: 859 MB, VmSize: 785 MB, VmRSS: 196 MB, VmData: 210 MB, maxrss: 0.201028 MB Old catalog: 11 New catalog: 5 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/blobs-custom-176643p69934.fits.gz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/metrics/cus/all-models-custom-176643p69934.fits Stage fitblobs : Wall: 3.45 s, CPU: 3.38 s, VmPeak: 859 MB, VmSize: 785 MB, VmRSS: 196 MB, VmData: 210 MB, maxrss: 0.201028 MB Stage fitblobs finished: 2020-08-07T14:45:19.694362 Resources for stage fitblobs : Wall: 3.45 s, CPU: 3.38 s Stage fitblobs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-fitblobs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-fitblobs.p Running stage coadds Running stage coadds at 2020-08-07T14:45:19.701492 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-ccds.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-image-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-invvar-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-nexp-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-depth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-galdepth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-psfsize-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-model-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-chi2-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-blobmodel-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-image-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-invvar-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-nexp-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-depth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-galdepth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-psfsize-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-model-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-chi2-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-blobmodel-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-image-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-invvar-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-nexp-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-depth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-galdepth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-psfsize-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-model-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-chi2-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-blobmodel-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-depth.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/tmp-legacysurvey-custom-176643p69934-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/tmp-legacysurvey-custom-176643p69934-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/tmp-legacysurvey-custom-176643p69934-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/tmp-legacysurvey-custom-176643p69934-resid.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-resid.jpg Stage coadds : Wall: 0.88 s, CPU: 0.69 s, VmPeak: 859 MB, VmSize: 785 MB, VmRSS: 197 MB, VmData: 210 MB, maxrss: 0.20218 MB Stage coadds finished: 2020-08-07T14:45:20.585930 Resources for stage coadds : Wall: 0.88 s, CPU: 0.69 s Stage coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-coadds.p Running stage wise_forced Running stage wise_forced at 2020-08-07T14:45:20.595583 Cut to 1 unWISE tiles Reading WISE tile 1758p696 band 1 Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w1-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w1-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w1-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w1-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.18 s, CPU: 0.07 s, VmPeak: 859 MB, VmSize: 835 MB, VmRSS: 190 MB, VmData: 198 MB, maxrss: 0.256648 MB Reading WISE tile 1758p696 band 2 Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w2-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w2-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w2-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w2-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.01 s, CPU: 0.01 s, VmPeak: 893 MB, VmSize: 835 MB, VmRSS: 191 MB, VmData: 198 MB, maxrss: 0.256648 MB Reading WISE tile 1758p696 band 3 Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w3-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.01 s, CPU: 0.01 s, VmPeak: 893 MB, VmSize: 835 MB, VmRSS: 191 MB, VmData: 198 MB, maxrss: 0.256648 MB Reading WISE tile 1758p696 band 4 Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/175/1758p696/unwise-1758p696-w4-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.01 s, CPU: 0.01 s, VmPeak: 893 MB, VmSize: 880 MB, VmRSS: 236 MB, VmData: 242 MB, maxrss: 0.256648 MB Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-image-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-model-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-invvar-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-image-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-model-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-invvar-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-image-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-model-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-invvar-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-image-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-model-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-invvar-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/tmp-legacysurvey-custom-176643p69934-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/tmp-legacysurvey-custom-176643p69934-wisemodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-wisemodel.jpg Setting WISE apphot Setting WISE apphot Setting WISE apphot Setting WISE apphot Stage wise_forced : Wall: 4.01 s, CPU: 3.01 s, VmPeak: 893 MB, VmSize: 880 MB, VmRSS: 236 MB, VmData: 242 MB, maxrss: 0.256648 MB Stage wise_forced finished: 2020-08-07T14:45:24.609343 Resources for stage wise_forced : Wall: 4.01 s, CPU: 3.01 s Stage wise_forced finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-wise_forced.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-wise_forced.p Running stage writecat Running stage writecat at 2020-08-07T14:45:24.618589 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/coadd/cus/custom-176643p69934/legacysurvey-custom-176643p69934-maskbits.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/tractor-i/cus/tractor-custom-176643p69934.fits Reading SFD maps... END Rejected (1), keyrecords following the END keyrecord were ignored. END Rejected (1), keyrecords following the END keyrecord were ignored. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/tractor/cus/tractor-custom-176643p69934.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/tractor/cus/brick-custom-176643p69934.sha256sum Stage writecat : Wall: 0.73 s, CPU: 0.41 s, VmPeak: 1012 MB, VmSize: 884 MB, VmRSS: 238 MB, VmData: 242 MB, maxrss: 0.374588 MB Stage writecat finished: 2020-08-07T14:45:25.348454 Resources for stage writecat : Wall: 0.73 s, CPU: 0.41 s Stage writecat finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-writecat.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/176/PGC2733107/PGC2733107-largegalaxy-writecat.p All done: Wall: 19.85 s, CPU: 12.71 s Finished galaxy PGC2733107 in 0.361 minutes.