Started working on galaxy PGC2605417 at Wed Aug 5 22:00:02 2020 Bands touching this brick, g r z python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 283.289223 60.68350600000001 --width 329 --height 329 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 runbrick.py starting at 2020-08-05T22:00:05.000257 legacypipe git version: DR9.5.10-12-gf22e529f Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '283.289223', '60.68350600000001', '--width', '329', '--height', '329', '--pixscale', '0.262', '--threads', '16', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-%%(stage)s.p', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 283.289223 60.68350600000001 --width 329 --height 329 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 Parsed RA,Dec 283.289223 60.68350600000001 Starting process 2215 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059236 MB Starting process 2216 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB Starting process 2217 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB Starting process 2218 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB Starting process 2219 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059472 MB Starting process 2220 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05942 MB Starting process 2221 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05942 MB Starting process 2222 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB Starting process 2223 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB Starting process 2224 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 2225 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 2226 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 2227 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 2228 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB Starting process 2229 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB Starting process 2230 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage fit_on_coadds Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-08-05T22:00:05.983617 CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' Keeping 9 CCDs: k4m_160607_101036_ooi_zd_ls9-CCD1 z expnum 94073 exptime 80.103 propid 2016A-0453 seeing 0.97 MJD 57546.424 object MzLS_688841_z mosaic/CP/V4.3/CP20160606/k4m_160607_101036_ooi_zd_ls9.fits.fz k4m_170927_053443_ooi_zd_ls9-CCD2 z expnum 141032 exptime 155.103 propid 2016A-0453 seeing 1.61 MJD 58023.232 object MzLS_688221_z mosaic/CP/V4.3/CP20170926/k4m_170927_053443_ooi_zd_ls9.fits.fz k4m_160606_094544_ooi_zd_ls9-CCD3 z expnum 93804 exptime 89.104 propid 2016A-0453 seeing 1.00 MJD 57545.407 object MzLS_689456_z mosaic/CP/V4.3/CP20160605/k4m_160606_094544_ooi_zd_ls9.fits.fz ksb_180519_100225_ooi_g_ls9-CCD2 g expnum 82570189 exptime 112.617 propid NOTBASS seeing 1.24 MJD 58257.418 object 38803 90prime/CP/V2.3/CP20180518/ksb_180519_100225_ooi_g_ls9.fits.fz ksb_170624_103052_ooi_r_ls9-CCD2 r expnum 79280167 exptime 136.0 propid BASS seeing 1.26 MJD 57928.438 object 38803 90prime/CP/V2.3/CP20170623/ksb_170624_103052_ooi_r_ls9.fits.fz ksb_180508_104536_ooi_r_ls9-CCD4 r expnum 82460215 exptime 117.421 propid NOTBASS seeing 1.28 MJD 58246.448 object 38801 90prime/CP/V2.3/CP20180507/ksb_180508_104536_ooi_r_ls9.fits.fz ksb_180517_111237_ooi_g_ls9-CCD4 g expnum 82550159 exptime 141.8 propid NOTBASS seeing 1.46 MJD 58255.467 object 38801 90prime/CP/V2.3/CP20180516/ksb_180517_111237_ooi_g_ls9.fits.fz ksb_160708_104905_ooi_g_ls9-CCD3 g expnum 75770145 exptime 212.6 propid BASS seeing 1.79 MJD 57577.451 object 37852 90prime/CP/V2.3/CP20160707/ksb_160708_104905_ooi_g_ls9.fits.fz ksb_160713_094902_ooi_r_ls9-CCD3 r expnum 75820120 exptime 109.0 propid BASS seeing 1.79 MJD 57582.409 object 37852 90prime/CP/V2.3/CP20160712/ksb_160713_094902_ooi_r_ls9.fits.fz Faked up EXPNUM 79280167 Faked up EXPNUM 79280167 Faked up EXPNUM 79280167 90prime get_fwhm: self.fwhm = 2.79 Found 4 relevant rows in amp-corrections file. Found overlap: image x 357 550 and amp range 0 2015 and image y 2349 2542 and amp range 2048 4095 Range in image: x 0 194 , y 0 194 (with image size 194 x 194) dzp 0.05525464 -> scaling image by 1.0522086198429996 CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' Faked up EXPNUM 82550159 Faked up EXPNUM 82550159 Faked up EXPNUM 82550159 90prime get_fwhm: self.fwhm = 3.23 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2350 2542 and amp range 2016 4031 and image y 15 80 and amp range 0 2047 Range in image: x 0 193 , y 0 66 (with image size 193 x 66) dzp 0.023266561 -> scaling image by 1.0216605511853067 CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' Faked up EXPNUM 82460215 Faked up EXPNUM 82460215 Faked up EXPNUM 82460215 90prime get_fwhm: self.fwhm = 2.83 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2367 2558 and amp range 2016 4031 and image y 15 45 and amp range 0 2047 Range in image: x 0 192 , y 0 31 (with image size 192 x 31) dzp 0.043530058 -> scaling image by 1.0409072255640852 CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' Faked up EXPNUM 82570189 Faked up EXPNUM 82570189 Faked up EXPNUM 82570189 90prime get_fwhm: self.fwhm = 2.73 Found 4 relevant rows in amp-corrections file. Found overlap: image x 335 528 and amp range 0 2015 and image y 2340 2534 and amp range 2048 4095 Range in image: x 0 194 , y 0 195 (with image size 194 x 195) dzp 0.0459271 -> scaling image by 1.0432078344950917 CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' Faked up EXPNUM 75770145 Faked up EXPNUM 75770145 Faked up EXPNUM 75770145 90prime get_fwhm: self.fwhm = 3.95 Found 4 relevant rows in amp-corrections file. Found overlap: image x 580 772 and amp range 0 2015 and image y 808 999 and amp range 0 2047 Range in image: x 0 193 , y 0 192 (with image size 193 x 192) dzp -0.03470582 -> scaling image by 0.9685402452207784 CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' Faked up EXPNUM 75820120 Faked up EXPNUM 75820120 Faked up EXPNUM 75820120 90prime get_fwhm: self.fwhm = 3.95 Found 4 relevant rows in amp-corrections file. Found overlap: image x 587 779 and amp range 0 2015 and image y 812 1003 and amp range 0 2047 Range in image: x 0 193 , y 0 192 (with image size 193 x 192) dzp -0.024507914 -> scaling image by 0.9776802328037623 CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' mosaic get_fwhm: self.fwhm = 6.19 Found 4 relevant rows in amp-corrections file. Found overlap: image x 1967 2300 and amp range 0 2047 and image y 2329 2663 and amp range 2040 4078 Range in image: x 0 81 , y 0 335 (with image size 334 x 335) dzp 0.003927584 -> scaling image by 1.0036239891963392 Found overlap: image x 1967 2300 and amp range 2048 4095 and image y 2329 2663 and amp range 2040 4078 Range in image: x 81 334 , y 0 335 (with image size 334 x 335) dzp -0.036442947 -> scaling image by 0.9669918668439786 CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' mosaic get_fwhm: self.fwhm = 3.85 Found 4 relevant rows in amp-corrections file. Found overlap: image x 853 1189 and amp range 0 2005 and image y 326 659 and amp range 0 2039 Range in image: x 0 337 , y 0 334 (with image size 337 x 334) dzp -0.005856539 -> scaling image by 0.9946204500110131 CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' mosaic get_fwhm: self.fwhm = 3.73 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3557 3888 and amp range 2006 4053 and image y 1453 1787 and amp range 0 2039 Range in image: x 0 332 , y 0 335 (with image size 332 x 335) dzp 0.001388291 -> scaling image by 1.0012794811153327 CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' Stage tims : Wall: 7.45 s, CPU: 0.99 s, VmPeak: 957 MB, VmSize: 957 MB, VmRSS: 177 MB, VmData: 209 MB, maxrss: 0.180816 MB Stage tims finished: 2020-08-05T22:00:13.434529 WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data] WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data] WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data] WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data] WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data] WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data] Resources for stage tims : 4.846 s worker CPU, 31.071 s worker Wall, Wall: 7.452 s, Cores in use: 0.78, Total efficiency (on 16 cores): 4.9 % Stage tims finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-tims.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-tims.p Running stage refs WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data] WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data] WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data] WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data] Running stage refs at 2020-08-05T22:00:13.497549 Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00906.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00743.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00822.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00823.fits Matched 8 large galaxies to Gaia stars. Matched 0 large galaxies to Tycho-2 stars. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/metrics/cus/reference-custom-283289p60683.fits Stage refs : Wall: 14.90 s, CPU: 5.11 s, VmPeak: 1026 MB, VmSize: 1020 MB, VmRSS: 243 MB, VmData: 272 MB, maxrss: 0.248472 MB Stage refs finished: 2020-08-05T22:00:28.396312 Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 14.900 s, Cores in use: 0.34, Total efficiency (on 16 cores): 2.1 % Stage refs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-refs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-refs.p Running stage outliers Running stage outliers at 2020-08-05T22:00:28.496760 Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/metrics/cus/outlier-mask-custom-283289p60683.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/metrics/cus/outliers-pre-custom-283289p60683.jpg Image ksb_180519_100225_ooi_g_ls9-CCD2 g : masking 386 positive outlier pixels and 0 negative outlier pixels Image ksb_180508_104536_ooi_r_ls9-CCD4 r : masking 58 positive outlier pixels and 0 negative outlier pixels Image ksb_170624_103052_ooi_r_ls9-CCD2 r : masking 89 positive outlier pixels and 0 negative outlier pixels Image k4m_170927_053443_ooi_zd_ls9-CCD2 z : masking 1794 positive outlier pixels and 0 negative outlier pixels Image k4m_160606_094544_ooi_zd_ls9-CCD3 z : masking 1058 positive outlier pixels and 0 negative outlier pixels Image k4m_160607_101036_ooi_zd_ls9-CCD1 z : masking 1249 positive outlier pixels and 0 negative outlier pixels Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/metrics/cus/outlier-mask-custom-283289p60683.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/metrics/cus/outliers-post-custom-283289p60683.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/metrics/cus/outliers-masked-pos-custom-283289p60683.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/metrics/cus/outliers-masked-neg-custom-283289p60683.jpg Stage outliers : Wall: 1.01 s, CPU: 0.69 s, VmPeak: 1030 MB, VmSize: 1030 MB, VmRSS: 260 MB, VmData: 286 MB, maxrss: 0.265924 MB Stage outliers finished: 2020-08-05T22:00:29.504706 Resources for stage outliers : 1.006 s worker CPU, 1.124 s worker Wall, Wall: 1.009 s, Cores in use: 1.68, Total efficiency (on 16 cores): 10.5 % Stage outliers finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-outliers.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-outliers.p Running stage halos Running stage halos at 2020-08-05T22:00:29.574027 Stage halos : Wall: 0.57 s, CPU: 0.56 s, VmPeak: 1030 MB, VmSize: 1030 MB, VmRSS: 260 MB, VmData: 286 MB, maxrss: 0.2661 MB Stage halos finished: 2020-08-05T22:00:30.143222 Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.571 s, Cores in use: 0.98, Total efficiency (on 16 cores): 6.1 % Stage halos finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-halos.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-halos.p Running stage fit_on_coadds Running stage fit_on_coadds at 2020-08-05T22:00:30.216432 PSF sigmas in coadd pixels: ['2.01', '2.38', '2.91'] Using average PSF sigma 2.432027803763198 Gaussian PSF norm 0.11599159817884436 vs pixelized 0.095841065 average tim pixel scale / coadd scale: 1.7302539572830646 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-copsf-g.fits.fz PSF sigmas in coadd pixels: ['2.05', '2.08', '2.91'] Using average PSF sigma 2.3476229855258617 Gaussian PSF norm 0.12016188012858874 vs pixelized 0.10434957 average tim pixel scale / coadd scale: 1.7295054778570436 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-copsf-r.fits.fz PSF sigmas in coadd pixels: ['1.57', '2.61', '1.62'] Using average PSF sigma 1.9355495361597093 Gaussian PSF norm 0.14574403108978426 vs pixelized 0.11683272 average tim pixel scale / coadd scale: 0.9909552679297635 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-copsf-z.fits.fz Stage fit_on_coadds : Wall: 0.21 s, CPU: 0.20 s, VmPeak: 1030 MB, VmSize: 1030 MB, VmRSS: 263 MB, VmData: 288 MB, maxrss: 0.26888 MB Stage fit_on_coadds finished: 2020-08-05T22:00:30.427085 Resources for stage fit_on_coadds : 0.253 s worker CPU, 0.253 s worker Wall, Wall: 0.212 s, Cores in use: 2.12, Total efficiency (on 16 cores): 13.2 % Stage fit_on_coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-fit_on_coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-fit_on_coadds.p Running stage srcs Running stage srcs at 2020-08-05T22:00:30.439456 SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])] Found 36 potential peaks Of 36 potential peaks: 14 in veto map, 2 cut by saddle test, 0 cut by aper test, 20 kept SED z : 20 new peaks Found 31 potential peaks Of 31 potential peaks: 30 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED r : 0 new peaks Found 28 potential peaks Of 28 potential peaks: 27 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED g : 0 new peaks Found 37 potential peaks Of 37 potential peaks: 33 in veto map, 3 cut by saddle test, 0 cut by aper test, 1 kept SED Flat : 1 new peaks Found 37 potential peaks Of 37 potential peaks: 35 in veto map, 1 cut by saddle test, 0 cut by aper test, 1 kept SED Red : 1 new peaks Adding 22 new sources Median coadd sky for g : 7.112807e-05 Median coadd sky for r : 3.249837e-05 Median coadd sky for z : -4.2964795e-05 Stage srcs : Wall: 0.55 s, CPU: 0.51 s, VmPeak: 1030 MB, VmSize: 1030 MB, VmRSS: 263 MB, VmData: 288 MB, maxrss: 0.269544 MB Stage srcs finished: 2020-08-05T22:00:30.993195 Resources for stage srcs : 0.117 s worker CPU, 0.117 s worker Wall, Wall: 0.555 s, Cores in use: 1.12, Total efficiency (on 16 cores): 7.0 % Stage srcs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-srcs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-srcs.p Running stage fitblobs Running stage fitblobs at 2020-08-05T22:00:31.008062 Blob 1 of 21, id: 10, sources: 2, size: 125x132, npix 7997, brick X: 107,232, Y: 98,230, one pixel: 205 98 Blob 2 of 21, id: 1, sources: 6, size: 75x138, npix 7285, brick X: 254,329, Y: 0,138, one pixel: 273 0 Blob 3 of 21, id: 18, sources: 4, size: 96x77, npix 4235, brick X: 111,207, Y: 252,329, one pixel: 158 252 Blob 4 of 21, id: 6, sources: 4, size: 68x74, npix 3097, brick X: 99,167, Y: 45,119, one pixel: 147 45 Blob 5 of 21, id: 13, sources: 4, size: 77x82, npix 3035, brick X: 0,77, Y: 154,236, one pixel: 43 154 Blob 6 of 21, id: 2, sources: 2, size: 55x57, npix 2302, brick X: 168,223, Y: 1,58, one pixel: 191 1 Blob 7 of 21, id: 7, sources: 2, size: 42x47, npix 1206, brick X: 167,209, Y: 50,97, one pixel: 176 50 Blob 8 of 21, id: 5, sources: 2, size: 38x43, npix 901, brick X: 37,75, Y: 38,81, one pixel: 49 38 Blob 9 of 21, id: 17, sources: 1, size: 36x34, npix 869, brick X: 161,197, Y: 221,255, one pixel: 174 221 Blob 10 of 21, id: 11, sources: 1, size: 37x33, npix 835, brick X: 1,38, Y: 119,152, one pixel: 17 119 Blob 11 of 21, id: 15, sources: 1, size: 32x32, npix 699, brick X: 60,92, Y: 212,244, one pixel: 72 212 Blob 12 of 21, id: 0, sources: 1, size: 33x29, npix 658, brick X: 40,73, Y: 0,29, one pixel: 46 0 Blob 13 of 21, id: 9, sources: 1, size: 27x33, npix 621, brick X: 70,97, Y: 91,124, one pixel: 81 91 Blob 14 of 21, id: 12, sources: 1, size: 31x31, npix 613, brick X: 251,282, Y: 139,170, one pixel: 265 139 Blob 15 of 21, id: 16, sources: 1, size: 28x28, npix 514, brick X: 219,247, Y: 215,243, one pixel: 231 215 Blob 16 of 21, id: 8, sources: 1, size: 27x25, npix 429, brick X: 225,252, Y: 73,98, one pixel: 236 73 Blob 17 of 21, id: 19, sources: 1, size: 26x26, npix 429, brick X: 263,289, Y: 276,302, one pixel: 274 276 Measuring 1 iterative sources Blob 15 finished, total: Wall: 0.71 s, CPU: 0.67 s, VmPeak: 813 MB, VmSize: 732 MB, VmRSS: 147 MB, VmData: 184 MB, maxrss: 0.15014 MB Blob 18 of 21, id: 4, sources: 3, size: 18x32, npix 413, brick X: 0,18, Y: 37,69, one pixel: 5 37 Blob 14 finished, total: Wall: 0.86 s, CPU: 0.83 s, VmPeak: 810 MB, VmSize: 731 MB, VmRSS: 148 MB, VmData: 183 MB, maxrss: 0.151288 MB Blob 19 of 21, id: 3, sources: 1, size: 23x22, npix 314, brick X: 103,126, Y: 7,29, one pixel: 112 7 Blob 20 of 21, id: 14, sources: 1, size: 23x22, npix 314, brick X: 296,319, Y: 175,197, one pixel: 306 175 Blob 9 finished, total: Wall: 0.91 s, CPU: 0.88 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 94 MB, VmData: 139 MB, maxrss: 0.096144 MB Blob 11 finished, total: Wall: 0.91 s, CPU: 0.91 s, VmPeak: 807 MB, VmSize: 731 MB, VmRSS: 148 MB, VmData: 183 MB, maxrss: 0.151312 MB Blob 21 of 21, id: 20, sources: 1, size: 23x21, npix 272, brick X: 42,65, Y: 294,315, one pixel: 51 294 Blob 16 finished, total: Wall: 0.93 s, CPU: 0.91 s, VmPeak: 670 MB, VmSize: 670 MB, VmRSS: 95 MB, VmData: 141 MB, maxrss: 0.09708 MB Measuring 3 iterative sources Measuring 1 iterative sources Blob 13 finished, total: Wall: 1.16 s, CPU: 1.16 s, VmPeak: 669 MB, VmSize: 669 MB, VmRSS: 95 MB, VmData: 140 MB, maxrss: 0.097544 MB Blob 19 finished, total: Wall: 0.29 s, CPU: 0.29 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 94 MB, VmData: 139 MB, maxrss: 0.096284 MB Blob 17 finished, total: Wall: 0.53 s, CPU: 0.53 s, VmPeak: 813 MB, VmSize: 732 MB, VmRSS: 147 MB, VmData: 184 MB, maxrss: 0.15014 MB Blob 10 finished, total: Wall: 1.25 s, CPU: 1.23 s, VmPeak: 812 MB, VmSize: 732 MB, VmRSS: 146 MB, VmData: 184 MB, maxrss: 0.149984 MB Blob 7 finished, total: Wall: 1.38 s, CPU: 1.35 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 92 MB, VmData: 138 MB, maxrss: 0.094552 MB Blob 8 finished, total: Wall: 1.39 s, CPU: 1.36 s, VmPeak: 670 MB, VmSize: 670 MB, VmRSS: 95 MB, VmData: 141 MB, maxrss: 0.097412 MB Blob 21 finished, total: Wall: 0.55 s, CPU: 0.55 s, VmPeak: 670 MB, VmSize: 670 MB, VmRSS: 95 MB, VmData: 141 MB, maxrss: 0.097296 MB Blob 20 finished, total: Wall: 0.66 s, CPU: 0.65 s, VmPeak: 807 MB, VmSize: 731 MB, VmRSS: 148 MB, VmData: 183 MB, maxrss: 0.151648 MB Blob 12 finished, total: Wall: 1.66 s, CPU: 1.66 s, VmPeak: 669 MB, VmSize: 669 MB, VmRSS: 95 MB, VmData: 140 MB, maxrss: 0.097096 MB Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (283.31563, 60.67591), pm (-2.4, -1.9), parallax -0.183 with NanoMaggies: g=18.5, r=16.3, z=15 Reset brightness to NanoMaggies: g=19.1, r=18.7, z=18.6 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (283.32413, 60.67599), pm (-4.3, -6.2), parallax 0.243 with NanoMaggies: g=17.7, r=17.2, z=16.9 Reset brightness to NanoMaggies: g=17.7, r=17.2, z=16.9 Blob 18 finished, total: Wall: 0.97 s, CPU: 0.97 s, VmPeak: 810 MB, VmSize: 731 MB, VmRSS: 149 MB, VmData: 183 MB, maxrss: 0.15224 MB Blob 5 finished, total: Wall: 2.70 s, CPU: 2.68 s, VmPeak: 808 MB, VmSize: 732 MB, VmRSS: 149 MB, VmData: 184 MB, maxrss: 0.152068 MB Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (283.26563, 60.67036), pm (2.7, -14.3), parallax 0.969 with NanoMaggies: g=16.4, r=15.5, z=14.8 Reset brightness to NanoMaggies: g=20, r=18.6, z=17.4 Measuring 2 iterative sources Measuring 1 iterative sources Measuring 1 iterative sources Blob 3 finished, total: Wall: 3.18 s, CPU: 3.16 s, VmPeak: 806 MB, VmSize: 732 MB, VmRSS: 149 MB, VmData: 183 MB, maxrss: 0.15254 MB Known galaxy. Initial shape: EllipseWithPriors(0.25): log r_e=1.50229, ee1=0.20349, ee2=0.631099 Measuring 1 iterative sources Blob 4 finished, total: Wall: 3.60 s, CPU: 3.58 s, VmPeak: 810 MB, VmSize: 732 MB, VmRSS: 149 MB, VmData: 184 MB, maxrss: 0.153012 MB Blob 6 finished, total: Wall: 3.79 s, CPU: 3.77 s, VmPeak: 669 MB, VmSize: 669 MB, VmRSS: 96 MB, VmData: 140 MB, maxrss: 0.097812 MB Blob 1 finished, total: Wall: 4.26 s, CPU: 4.25 s, VmPeak: 811 MB, VmSize: 732 MB, VmRSS: 151 MB, VmData: 184 MB, maxrss: 0.154616 MB Blob 2 finished, total: Wall: 5.43 s, CPU: 5.42 s, VmPeak: 811 MB, VmSize: 732 MB, VmRSS: 151 MB, VmData: 184 MB, maxrss: 0.154452 MB Old catalog: 47 New catalog: 51 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/metrics/cus/blobs-custom-283289p60683.fits.gz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/metrics/cus/all-models-custom-283289p60683.fits Stage fitblobs : Wall: 14.57 s, CPU: 0.67 s, VmPeak: 1030 MB, VmSize: 1030 MB, VmRSS: 264 MB, VmData: 288 MB, maxrss: 0.269836 MB Stage fitblobs finished: 2020-08-05T22:00:45.573510 Resources for stage fitblobs : 37.152 s worker CPU, 38.779 s worker Wall, Wall: 14.567 s, Cores in use: 2.60, Total efficiency (on 16 cores): 16.2 % Stage fitblobs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-fitblobs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-fitblobs.p Running stage coadds Running stage coadds at 2020-08-05T22:00:45.654531 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-ccds.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-image-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-invvar-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-nexp-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-depth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-galdepth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-psfsize-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-model-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-chi2-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-blobmodel-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-image-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-invvar-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-nexp-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-depth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-galdepth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-psfsize-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-model-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-chi2-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-blobmodel-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-image-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-invvar-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-nexp-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-depth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-galdepth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-psfsize-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-model-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-chi2-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-blobmodel-z.fits.fz CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-depth.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/tmp-legacysurvey-custom-283289p60683-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/tmp-legacysurvey-custom-283289p60683-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/tmp-legacysurvey-custom-283289p60683-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/tmp-legacysurvey-custom-283289p60683-resid.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-resid.jpg Stage coadds : Wall: 5.03 s, CPU: 0.75 s, VmPeak: 1030 MB, VmSize: 1030 MB, VmRSS: 264 MB, VmData: 288 MB, maxrss: 0.270188 MB Stage coadds finished: 2020-08-05T22:00:50.682784 WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data] WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data] WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data] Resources for stage coadds : 10.244 s worker CPU, 25.333 s worker Wall, Wall: 5.030 s, Cores in use: 2.19, Total efficiency (on 16 cores): 13.7 % Stage coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-coadds.p Running stage wise_forced WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data] WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data] WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data] Running stage wise_forced at 2020-08-05T22:00:50.700077 Cut to 1 unWISE tiles Reading WISE tile 2820p605 band 1 Reading WISE tile 2820p605 band 2 Reading WISE tile 2820p605 band 3 Reading WISE tile 2820p605 band 4 Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w3-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w3-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w3-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.08 s, CPU: 0.08 s, VmPeak: 811 MB, VmSize: 807 MB, VmRSS: 149 MB, VmData: 175 MB, maxrss: 0.213584 MB Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w4-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w4-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w4-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.10 s, CPU: 0.10 s, VmPeak: 811 MB, VmSize: 810 MB, VmRSS: 152 MB, VmData: 178 MB, maxrss: 0.195112 MB Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w1-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w1-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w1-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w1-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.07 s, CPU: 0.07 s, VmPeak: 812 MB, VmSize: 807 MB, VmRSS: 147 MB, VmData: 175 MB, maxrss: 0.21074 MB Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w2-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w2-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w2-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w2-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.07 s, CPU: 0.07 s, VmPeak: 807 MB, VmSize: 804 MB, VmRSS: 145 MB, VmData: 172 MB, maxrss: 0.208208 MB Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-image-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-model-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-invvar-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-image-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-model-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-invvar-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-image-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-model-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-invvar-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-image-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-model-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-invvar-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/tmp-legacysurvey-custom-283289p60683-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/tmp-legacysurvey-custom-283289p60683-wisemodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-wisemodel.jpg Setting WISE apphot Setting WISE apphot Setting WISE apphot Setting WISE apphot Stage wise_forced : Wall: 7.53 s, CPU: 0.56 s, VmPeak: 1030 MB, VmSize: 1030 MB, VmRSS: 264 MB, VmData: 288 MB, maxrss: 0.270716 MB Stage wise_forced finished: 2020-08-05T22:00:58.231486 WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data] Resources for stage wise_forced : 3.331 s worker CPU, 9.852 s worker Wall, Wall: 7.533 s, Cores in use: 0.52, Total efficiency (on 16 cores): 3.2 % Stage wise_forced finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-wise_forced.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-wise_forced.p Running stage writecat Running stage writecat at 2020-08-05T22:00:58.253657 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/coadd/cus/custom-283289p60683/legacysurvey-custom-283289p60683-maskbits.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/tractor-i/cus/tractor-custom-283289p60683.fits Reading SFD maps... END Rejected (1), keyrecords following the END keyrecord were ignored. END Rejected (1), keyrecords following the END keyrecord were ignored. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/tractor/cus/tractor-custom-283289p60683.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/tractor/cus/brick-custom-283289p60683.sha256sum Stage writecat : Wall: 2.46 s, CPU: 1.03 s, VmPeak: 1162 MB, VmSize: 1034 MB, VmRSS: 266 MB, VmData: 288 MB, maxrss: 0.403596 MB Stage writecat finished: 2020-08-05T22:01:00.715820 Resources for stage writecat : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 2.464 s, Cores in use: 0.42, Total efficiency (on 16 cores): 2.6 % Stage writecat finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-writecat.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2605417/PGC2605417-largegalaxy-writecat.p All done: 56.953 s worker CPU, 106.533 s worker Wall, Wall: 54.843 s, Cores in use: 1.24, Total efficiency (on 16 cores): 7.8 % Finished galaxy PGC2605417 in 0.974 minutes. Started working on galaxy PGC2605417 at Fri Aug 7 09:58:39 2020 Bands touching this brick, g r z python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 283.28852889391 60.6835168525139 --width 329 --height 329 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 runbrick.py starting at 2020-08-07T09:58:40.333955 legacypipe git version: DR9.5.11-watershed-3-gd509eaaf Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '283.28852889391', '60.6835168525139', '--width', '329', '--height', '329', '--pixscale', '0.262', '--threads', '16', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 283.28852889391 60.6835168525139 --width 329 --height 329 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 Parsed RA,Dec 283.28852889391 60.6835168525139 Starting process 52802 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB Starting process 52803 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 52804 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 52805 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 52806 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05954 MB Starting process 52807 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059548 MB Starting process 52808 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059552 MB Starting process 52809 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 52810 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 52811 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 52812 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 52813 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB Starting process 52814 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 52816 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 52815 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 52817 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage fit_on_coadds Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-08-07T09:58:40.767150 Keeping 9 CCDs: k4m_160607_101036_ooi_zd_ls9-CCD1 z expnum 94073 exptime 80.103 propid 2016A-0453 seeing 0.97 MJD 57546.424 object MzLS_688841_z mosaic/CP/V4.3/CP20160606/k4m_160607_101036_ooi_zd_ls9.fits.fz k4m_170927_053443_ooi_zd_ls9-CCD2 z expnum 141032 exptime 155.103 propid 2016A-0453 seeing 1.61 MJD 58023.232 object MzLS_688221_z mosaic/CP/V4.3/CP20170926/k4m_170927_053443_ooi_zd_ls9.fits.fz k4m_160606_094544_ooi_zd_ls9-CCD3 z expnum 93804 exptime 89.104 propid 2016A-0453 seeing 1.00 MJD 57545.407 object MzLS_689456_z mosaic/CP/V4.3/CP20160605/k4m_160606_094544_ooi_zd_ls9.fits.fz ksb_180519_100225_ooi_g_ls9-CCD2 g expnum 82570189 exptime 112.617 propid NOTBASS seeing 1.24 MJD 58257.418 object 38803 90prime/CP/V2.3/CP20180518/ksb_180519_100225_ooi_g_ls9.fits.fz ksb_170624_103052_ooi_r_ls9-CCD2 r expnum 79280167 exptime 136.0 propid BASS seeing 1.26 MJD 57928.438 object 38803 90prime/CP/V2.3/CP20170623/ksb_170624_103052_ooi_r_ls9.fits.fz ksb_180508_104536_ooi_r_ls9-CCD4 r expnum 82460215 exptime 117.421 propid NOTBASS seeing 1.28 MJD 58246.448 object 38801 90prime/CP/V2.3/CP20180507/ksb_180508_104536_ooi_r_ls9.fits.fz ksb_180517_111237_ooi_g_ls9-CCD4 g expnum 82550159 exptime 141.8 propid NOTBASS seeing 1.46 MJD 58255.467 object 38801 90prime/CP/V2.3/CP20180516/ksb_180517_111237_ooi_g_ls9.fits.fz ksb_160708_104905_ooi_g_ls9-CCD3 g expnum 75770145 exptime 212.6 propid BASS seeing 1.79 MJD 57577.451 object 37852 90prime/CP/V2.3/CP20160707/ksb_160708_104905_ooi_g_ls9.fits.fz ksb_160713_094902_ooi_r_ls9-CCD3 r expnum 75820120 exptime 109.0 propid BASS seeing 1.79 MJD 57582.409 object 37852 90prime/CP/V2.3/CP20160712/ksb_160713_094902_ooi_r_ls9.fits.fz Faked up EXPNUM 82460215 Faked up EXPNUM 82460215 Faked up EXPNUM 82460215 90prime get_fwhm: self.fwhm = 2.83 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2366 2558 and amp range 2016 4031 and image y 15 48 and amp range 0 2047 Range in image: x 0 193 , y 0 34 (with image size 193 x 34) dzp 0.043530058 -> scaling image by 1.0409072255640852 Faked up EXPNUM 75770145 Faked up EXPNUM 75770145 Faked up EXPNUM 75770145 90prime get_fwhm: self.fwhm = 3.95 Found 4 relevant rows in amp-corrections file. Found overlap: image x 580 772 and amp range 0 2015 and image y 811 1002 and amp range 0 2047 Range in image: x 0 193 , y 0 192 (with image size 193 x 192) dzp -0.03470582 -> scaling image by 0.9685402452207784 Faked up EXPNUM 82550159 Faked up EXPNUM 82550159 Faked up EXPNUM 82550159 90prime get_fwhm: self.fwhm = 3.23 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2350 2542 and amp range 2016 4031 and image y 15 83 and amp range 0 2047 Range in image: x 0 193 , y 0 69 (with image size 193 x 69) dzp 0.023266561 -> scaling image by 1.0216605511853067 Faked up EXPNUM 82570189 Faked up EXPNUM 82570189 Faked up EXPNUM 82570189 90prime get_fwhm: self.fwhm = 2.73 Found 4 relevant rows in amp-corrections file. Found overlap: image x 335 528 and amp range 0 2015 and image y 2343 2537 and amp range 2048 4095 Range in image: x 0 194 , y 0 195 (with image size 194 x 195) dzp 0.0459271 -> scaling image by 1.0432078344950917 Faked up EXPNUM 75820120 Faked up EXPNUM 75820120 Faked up EXPNUM 75820120 90prime get_fwhm: self.fwhm = 3.95 Found 4 relevant rows in amp-corrections file. Found overlap: image x 587 779 and amp range 0 2015 and image y 814 1006 and amp range 0 2047 Range in image: x 0 193 , y 0 193 (with image size 193 x 193) dzp -0.024507914 -> scaling image by 0.9776802328037623 Faked up EXPNUM 79280167 Faked up EXPNUM 79280167 Faked up EXPNUM 79280167 90prime get_fwhm: self.fwhm = 2.79 Found 4 relevant rows in amp-corrections file. Found overlap: image x 357 550 and amp range 0 2015 and image y 2351 2545 and amp range 2048 4095 Range in image: x 0 194 , y 0 195 (with image size 194 x 195) dzp 0.05525464 -> scaling image by 1.0522086198429996 mosaic get_fwhm: self.fwhm = 6.19 Found 4 relevant rows in amp-corrections file. Found overlap: image x 1967 2300 and amp range 0 2047 and image y 2324 2658 and amp range 2040 4078 Range in image: x 0 81 , y 0 335 (with image size 334 x 335) dzp 0.003927584 -> scaling image by 1.0036239891963392 Found overlap: image x 1967 2300 and amp range 2048 4095 and image y 2324 2658 and amp range 2040 4078 Range in image: x 81 334 , y 0 335 (with image size 334 x 335) dzp -0.036442947 -> scaling image by 0.9669918668439786 mosaic get_fwhm: self.fwhm = 3.85 Found 4 relevant rows in amp-corrections file. Found overlap: image x 853 1189 and amp range 0 2005 and image y 321 654 and amp range 0 2039 Range in image: x 0 337 , y 0 334 (with image size 337 x 334) dzp -0.005856539 -> scaling image by 0.9946204500110131 mosaic get_fwhm: self.fwhm = 3.73 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3557 3888 and amp range 2006 4053 and image y 1448 1782 and amp range 0 2039 Range in image: x 0 332 , y 0 335 (with image size 332 x 335) dzp 0.001388291 -> scaling image by 1.0012794811153327 Stage tims : Wall: 27.12 s, CPU: 0.92 s, VmPeak: 951 MB, VmSize: 951 MB, VmRSS: 175 MB, VmData: 208 MB, maxrss: 0.178872 MB Stage tims finished: 2020-08-07T09:59:07.883531 Resources for stage tims : 4.227 s worker CPU, 65.832 s worker Wall, Wall: 27.118 s, Cores in use: 0.19, Total efficiency (on 16 cores): 1.2 % Stage tims finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-tims.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-tims.p Running stage refs Running stage refs at 2020-08-07T09:59:07.955621 Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00906.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00743.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00822.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00823.fits Matched 8 large galaxies to Gaia stars. Matched 0 large galaxies to Tycho-2 stars. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/metrics/cus/reference-custom-283288p60683.fits Stage refs : Wall: 44.44 s, CPU: 4.47 s, VmPeak: 1020 MB, VmSize: 1014 MB, VmRSS: 240 MB, VmData: 272 MB, maxrss: 0.246104 MB Stage refs finished: 2020-08-07T09:59:52.397128 Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 44.444 s, Cores in use: 0.10, Total efficiency (on 16 cores): 0.6 % Stage refs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-refs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-refs.p Running stage outliers Running stage outliers at 2020-08-07T09:59:53.147113 Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/metrics/cus/outlier-mask-custom-283288p60683.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/metrics/cus/outliers-pre-custom-283288p60683.jpg Image ksb_180519_100225_ooi_g_ls9-CCD2 g : masking 383 positive outlier pixels and 0 negative outlier pixels Image ksb_180508_104536_ooi_r_ls9-CCD4 r : masking 102 positive outlier pixels and 0 negative outlier pixels Image ksb_170624_103052_ooi_r_ls9-CCD2 r : masking 88 positive outlier pixels and 0 negative outlier pixels Image k4m_170927_053443_ooi_zd_ls9-CCD2 z : masking 1761 positive outlier pixels and 0 negative outlier pixels Image k4m_160606_094544_ooi_zd_ls9-CCD3 z : masking 944 positive outlier pixels and 0 negative outlier pixels Image k4m_160607_101036_ooi_zd_ls9-CCD1 z : masking 1266 positive outlier pixels and 0 negative outlier pixels Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/metrics/cus/outlier-mask-custom-283288p60683.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/metrics/cus/outliers-post-custom-283288p60683.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/metrics/cus/outliers-masked-pos-custom-283288p60683.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/metrics/cus/outliers-masked-neg-custom-283288p60683.jpg Stage outliers : Wall: 3.97 s, CPU: 0.73 s, VmPeak: 1024 MB, VmSize: 1024 MB, VmRSS: 256 MB, VmData: 285 MB, maxrss: 0.262536 MB Stage outliers finished: 2020-08-07T09:59:57.115788 Resources for stage outliers : 1.012 s worker CPU, 1.121 s worker Wall, Wall: 3.970 s, Cores in use: 0.44, Total efficiency (on 16 cores): 2.7 % Stage outliers finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-outliers.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-outliers.p Running stage halos Running stage halos at 2020-08-07T09:59:57.135332 Stage halos : Wall: 0.34 s, CPU: 0.33 s, VmPeak: 1024 MB, VmSize: 1024 MB, VmRSS: 257 MB, VmData: 285 MB, maxrss: 0.262776 MB Stage halos finished: 2020-08-07T09:59:57.474574 Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.341 s, Cores in use: 0.97, Total efficiency (on 16 cores): 6.1 % Stage halos finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-halos.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-halos.p Running stage fit_on_coadds Warning: Stellar halo subtraction is only implemented for DECam Running stage fit_on_coadds at 2020-08-07T09:59:57.497741 PSF sigmas in coadd pixels: ['2.01', '2.38', '2.91'] Using average PSF sigma 2.432027803763198 Gaussian PSF norm 0.11599159817884436 vs pixelized 0.095838696 average tim pixel scale / coadd scale: 1.7302539572830646 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-copsf-g.fits.fz PSF sigmas in coadd pixels: ['2.05', '2.08', '2.91'] Using average PSF sigma 2.3476229855258617 Gaussian PSF norm 0.12016188012858874 vs pixelized 0.10435013 average tim pixel scale / coadd scale: 1.7295054778570436 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-copsf-r.fits.fz PSF sigmas in coadd pixels: ['1.57', '2.61', '1.62'] Using average PSF sigma 1.9355495361597093 Gaussian PSF norm 0.14574403108978426 vs pixelized 0.116829835 average tim pixel scale / coadd scale: 0.9909552679297635 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-copsf-z.fits.fz Stage fit_on_coadds : Wall: 0.21 s, CPU: 0.19 s, VmPeak: 1024 MB, VmSize: 1024 MB, VmRSS: 261 MB, VmData: 288 MB, maxrss: 0.267 MB Stage fit_on_coadds finished: 2020-08-07T09:59:57.705623 Warning: Stellar halo subtraction is only implemented for DECam Resources for stage fit_on_coadds : 0.227 s worker CPU, 0.227 s worker Wall, Wall: 0.209 s, Cores in use: 1.98, Total efficiency (on 16 cores): 12.4 % Stage fit_on_coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-fit_on_coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-fit_on_coadds.p Running stage srcs Warning: Stellar halo subtraction is only implemented for DECam Running stage srcs at 2020-08-07T09:59:57.783138 SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])] Found 37 potential peaks Of 37 potential peaks: 15 in veto map, 2 cut by saddle test, 0 cut by aper test, 20 kept SED z : 20 new peaks Found 31 potential peaks Of 31 potential peaks: 30 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED r : 0 new peaks Found 29 potential peaks Of 29 potential peaks: 28 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED g : 0 new peaks Found 36 potential peaks Of 36 potential peaks: 33 in veto map, 2 cut by saddle test, 0 cut by aper test, 1 kept SED Flat : 1 new peaks Found 39 potential peaks Of 39 potential peaks: 36 in veto map, 1 cut by saddle test, 0 cut by aper test, 2 kept SED Red : 2 new peaks Adding 23 new sources Median coadd sky for g : 5.9140428e-05 Median coadd sky for r : 4.192812e-05 Median coadd sky for z : -1.7749027e-05 Stage srcs : Wall: 0.51 s, CPU: 0.47 s, VmPeak: 1024 MB, VmSize: 1024 MB, VmRSS: 261 MB, VmData: 288 MB, maxrss: 0.267696 MB Stage srcs finished: 2020-08-07T09:59:58.295931 Resources for stage srcs : 0.116 s worker CPU, 0.116 s worker Wall, Wall: 0.514 s, Cores in use: 1.14, Total efficiency (on 16 cores): 7.1 % Stage srcs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-srcs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-srcs.p Running stage fitblobs Running stage fitblobs at 2020-08-07T09:59:58.354556 Blob 1 of 23, id: 10, sources: 1, size: 126x132, npix 7991, brick X: 102,228, Y: 98,230, one pixel: 200 98 Blob 2 of 23, id: 1, sources: 7, size: 80x137, npix 7535, brick X: 249,329, Y: 0,137, one pixel: 268 0 Blob 3 of 23, id: 6, sources: 4, size: 67x74, npix 3083, brick X: 95,162, Y: 45,119, one pixel: 142 45 Blob 4 of 23, id: 13, sources: 4, size: 72x83, npix 2948, brick X: 0,72, Y: 153,236, one pixel: 42 153 Blob 5 of 23, id: 21, sources: 2, size: 81x48, npix 2631, brick X: 106,187, Y: 281,329, one pixel: 124 281 Blob 6 of 23, id: 2, sources: 2, size: 54x57, npix 2299, brick X: 164,218, Y: 1,58, one pixel: 185 1 Blob 7 of 23, id: 19, sources: 2, size: 63x45, npix 1595, brick X: 139,202, Y: 252,297, one pixel: 153 252 Blob 8 of 23, id: 7, sources: 2, size: 41x46, npix 1200, brick X: 163,204, Y: 50,96, one pixel: 171 50 Blob 9 of 23, id: 4, sources: 2, size: 38x43, npix 908, brick X: 32,70, Y: 38,81, one pixel: 44 38 Blob 10 of 23, id: 17, sources: 1, size: 36x35, npix 871, brick X: 156,192, Y: 220,255, one pixel: 172 220 Blob 11 of 23, id: 11, sources: 1, size: 33x33, npix 801, brick X: 0,33, Y: 119,152, one pixel: 12 119 Blob 12 of 23, id: 15, sources: 1, size: 32x32, npix 701, brick X: 55,87, Y: 212,244, one pixel: 67 212 Blob 13 of 23, id: 0, sources: 1, size: 34x29, npix 667, brick X: 35,69, Y: 0,29, one pixel: 41 0 Blob 14 of 23, id: 9, sources: 1, size: 28x33, npix 627, brick X: 65,93, Y: 91,124, one pixel: 76 91 Blob 15 of 23, id: 12, sources: 1, size: 31x31, npix 624, brick X: 246,277, Y: 139,170, one pixel: 260 139 Blob 16 of 23, id: 16, sources: 1, size: 27x28, npix 511, brick X: 215,242, Y: 215,243, one pixel: 226 215 Blob 17 of 23, id: 20, sources: 1, size: 26x26, npix 426, brick X: 258,284, Y: 276,302, one pixel: 269 276 Measuring 1 iterative sources Blob 16 finished, total: Wall: 0.76 s, CPU: 0.76 s, VmPeak: 670 MB, VmSize: 670 MB, VmRSS: 95 MB, VmData: 141 MB, maxrss: 0.097276 MB Blob 18 of 23, id: 8, sources: 1, size: 27x26, npix 420, brick X: 220,247, Y: 72,98, one pixel: 234 72 Blob 10 finished, total: Wall: 0.87 s, CPU: 0.87 s, VmPeak: 810 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.147776 MB Blob 19 of 23, id: 14, sources: 1, size: 23x22, npix 316, brick X: 291,314, Y: 175,197, one pixel: 301 175 Blob 12 finished, total: Wall: 0.89 s, CPU: 0.89 s, VmPeak: 808 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.146456 MB Blob 20 of 23, id: 3, sources: 1, size: 23x23, npix 307, brick X: 98,121, Y: 6,29, one pixel: 109 6 Measuring 1 iterative sources Blob 15 finished, total: Wall: 0.93 s, CPU: 0.93 s, VmPeak: 807 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.147924 MB Blob 21 of 23, id: 22, sources: 1, size: 22x22, npix 281, brick X: 38,60, Y: 293,315, one pixel: 48 293 Blob 22 of 23, id: 18, sources: 1, size: 17x21, npix 233, brick X: 312,329, Y: 229,250, one pixel: 320 229 Measuring 1 iterative sources Blob 11 finished, total: Wall: 1.24 s, CPU: 1.24 s, VmPeak: 811 MB, VmSize: 725 MB, VmRSS: 145 MB, VmData: 183 MB, maxrss: 0.148588 MB Blob 14 finished, total: Wall: 1.22 s, CPU: 1.20 s, VmPeak: 806 MB, VmSize: 725 MB, VmRSS: 145 MB, VmData: 183 MB, maxrss: 0.148288 MB Blob 23 of 23, id: 5, sources: 2, size: 12x20, npix 153, brick X: 0,12, Y: 43,63, one pixel: 2 43 Blob 9 finished, total: Wall: 1.30 s, CPU: 1.30 s, VmPeak: 813 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.146064 MB Blob 20 finished, total: Wall: 0.32 s, CPU: 0.32 s, VmPeak: 807 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.147924 MB Blob 17 finished, total: Wall: 0.55 s, CPU: 0.55 s, VmPeak: 670 MB, VmSize: 670 MB, VmRSS: 95 MB, VmData: 141 MB, maxrss: 0.097372 MB Blob 22 finished, total: Wall: 0.21 s, CPU: 0.21 s, VmPeak: 806 MB, VmSize: 725 MB, VmRSS: 145 MB, VmData: 183 MB, maxrss: 0.148288 MB Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (283.31563, 60.67591), pm (-2.4, -1.9), parallax -0.183 with NanoMaggies: g=17.7, r=16.3, z=15.3 Reset brightness to NanoMaggies: g=19.1, r=18.7, z=18.6 Blob 23 finished, total: Wall: 0.25 s, CPU: 0.25 s, VmPeak: 813 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.14652 MB Blob 18 finished, total: Wall: 0.73 s, CPU: 0.73 s, VmPeak: 810 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.147776 MB Blob 13 finished, total: Wall: 1.63 s, CPU: 1.63 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 95 MB, VmData: 139 MB, maxrss: 0.09702 MB Blob 19 finished, total: Wall: 0.74 s, CPU: 0.74 s, VmPeak: 808 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.146456 MB Blob 8 finished, total: Wall: 1.68 s, CPU: 1.66 s, VmPeak: 669 MB, VmSize: 669 MB, VmRSS: 93 MB, VmData: 140 MB, maxrss: 0.095272 MB Blob 7 finished, total: Wall: 1.72 s, CPU: 1.71 s, VmPeak: 669 MB, VmSize: 669 MB, VmRSS: 94 MB, VmData: 140 MB, maxrss: 0.096668 MB Blob 21 finished, total: Wall: 0.50 s, CPU: 0.50 s, VmPeak: 811 MB, VmSize: 725 MB, VmRSS: 145 MB, VmData: 183 MB, maxrss: 0.148588 MB Blob 6 finished, total: Wall: 1.77 s, CPU: 1.77 s, VmPeak: 812 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.147656 MB Measuring 1 iterative sources kdtree_internal.c:2152:kdtree_build_2_ddd: Data, N, or D is zero Blob 5 finished, total: Wall: 2.23 s, CPU: 2.22 s, VmPeak: 669 MB, VmSize: 669 MB, VmRSS: 96 MB, VmData: 140 MB, maxrss: 0.097868 MB Exception in one_blob: brick custom-283288p60683, iblob 10 Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1637, in _bounce_one_blob result = one_blob(X) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 341, in run ras[not_galaxy], decs[not_galaxy], 1./3600.) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match (kd1,kd2) = _buildtrees(x1, x2) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees kd2 = spherematch_c.KdTree(fx2) SystemError: returned NULL without setting an error multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "/usr/local/lib/python/astrometry/util/timingpool.py", line 269, in __call__ R = self.func(*args, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1637, in _bounce_one_blob result = one_blob(X) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 341, in run ras[not_galaxy], decs[not_galaxy], 1./3600.) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match (kd1,kd2) = _buildtrees(x1, x2) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees kd2 = spherematch_c.KdTree(fx2) SystemError: returned NULL without setting an error """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3771, in sys.exit(main()) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3739, in main run_brick(opt.brick, survey, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3295, in run_brick initial_args=initargs, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage R = stagefunc(stage, **Px) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3284, in mystagefunc R = stagefunc(stage, mp=mp, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__ rtn = super(CallGlobalTime, self).__call__(stage, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__ return func(*self.args, **kwa) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1131, in stage_fitblobs r = Riter.next(timeout) File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 748, in next raise value SystemError: returned NULL without setting an error Something went wrong; please check the logfile. ERROR: galaxy PGC2605417; please check the logfile. Finished galaxy PGC2605417 in 1.368 minutes. Started working on galaxy PGC2605417 at Fri Aug 7 12:08:26 2020 Bands touching this brick, g r z python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 283.28852889391 60.6835168525139 --width 329 --height 329 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 runbrick.py starting at 2020-08-07T12:08:27.678567 legacypipe git version: DR9.5.11-watershed-4-gcc619972 Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '283.28852889391', '60.6835168525139', '--width', '329', '--height', '329', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 283.28852889391 60.6835168525139 --width 329 --height 329 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 Parsed RA,Dec 283.28852889391 60.6835168525139 Starting process 16972 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB Starting process 16973 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 16974 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 16975 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 16976 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05954 MB Starting process 16977 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059552 MB Starting process 16978 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059552 MB Starting process 16979 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 16980 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 16981 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 16982 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB Starting process 16983 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB Starting process 16984 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 16985 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 16986 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 16987 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 16988 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB Starting process 16989 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059276 MB Starting process 16990 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 16991 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 16992 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 16993 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB Starting process 16994 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB Starting process 16995 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB Starting process 16996 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB Starting process 16998 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB Starting process 16999 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059308 MB Starting process 16997 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB Starting process 17000 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB Starting process 17001 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB Starting process 17002 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB Starting process 17003 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059316 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-srcs.p and forcing stage srcs Runstage fit_on_coadds Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-fit_on_coadds.p and forcing stage fit_on_coadds Runstage halos Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-halos.p and forcing stage halos Runstage outliers Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-outliers.p and forcing stage outliers Runstage refs Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-refs.p and forcing stage refs Runstage tims Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-tims.p and forcing stage tims Running stage tims Running stage tims at 2020-08-07T12:08:28.143427 Keeping 9 CCDs: k4m_160607_101036_ooi_zd_ls9-CCD1 z expnum 94073 exptime 80.103 propid 2016A-0453 seeing 0.97 MJD 57546.424 object MzLS_688841_z mosaic/CP/V4.3/CP20160606/k4m_160607_101036_ooi_zd_ls9.fits.fz k4m_170927_053443_ooi_zd_ls9-CCD2 z expnum 141032 exptime 155.103 propid 2016A-0453 seeing 1.61 MJD 58023.232 object MzLS_688221_z mosaic/CP/V4.3/CP20170926/k4m_170927_053443_ooi_zd_ls9.fits.fz k4m_160606_094544_ooi_zd_ls9-CCD3 z expnum 93804 exptime 89.104 propid 2016A-0453 seeing 1.00 MJD 57545.407 object MzLS_689456_z mosaic/CP/V4.3/CP20160605/k4m_160606_094544_ooi_zd_ls9.fits.fz ksb_180519_100225_ooi_g_ls9-CCD2 g expnum 82570189 exptime 112.617 propid NOTBASS seeing 1.24 MJD 58257.418 object 38803 90prime/CP/V2.3/CP20180518/ksb_180519_100225_ooi_g_ls9.fits.fz ksb_170624_103052_ooi_r_ls9-CCD2 r expnum 79280167 exptime 136.0 propid BASS seeing 1.26 MJD 57928.438 object 38803 90prime/CP/V2.3/CP20170623/ksb_170624_103052_ooi_r_ls9.fits.fz ksb_180508_104536_ooi_r_ls9-CCD4 r expnum 82460215 exptime 117.421 propid NOTBASS seeing 1.28 MJD 58246.448 object 38801 90prime/CP/V2.3/CP20180507/ksb_180508_104536_ooi_r_ls9.fits.fz ksb_180517_111237_ooi_g_ls9-CCD4 g expnum 82550159 exptime 141.8 propid NOTBASS seeing 1.46 MJD 58255.467 object 38801 90prime/CP/V2.3/CP20180516/ksb_180517_111237_ooi_g_ls9.fits.fz ksb_160708_104905_ooi_g_ls9-CCD3 g expnum 75770145 exptime 212.6 propid BASS seeing 1.79 MJD 57577.451 object 37852 90prime/CP/V2.3/CP20160707/ksb_160708_104905_ooi_g_ls9.fits.fz ksb_160713_094902_ooi_r_ls9-CCD3 r expnum 75820120 exptime 109.0 propid BASS seeing 1.79 MJD 57582.409 object 37852 90prime/CP/V2.3/CP20160712/ksb_160713_094902_ooi_r_ls9.fits.fz Faked up EXPNUM 82460215 Faked up EXPNUM 82460215 Faked up EXPNUM 82460215 90prime get_fwhm: self.fwhm = 2.83 Found 4 relevant rows in amp-corrections file. Faked up EXPNUM 82550159 Faked up EXPNUM 82550159 Faked up EXPNUM 82550159 90prime get_fwhm: self.fwhm = 3.23 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2366 2558 and amp range 2016 4031 and image y 15 48 and amp range 0 2047 Found overlap: image x 2350 2542 and amp range 2016 4031 and image y 15 83 and amp range 0 2047 Range in image: x 0 193 , y 0 34 (with image size 193 x 34) Range in image: x 0 193 , y 0 69 (with image size 193 x 69) dzp 0.043530058 -> scaling image by 1.0409072255640852 dzp 0.023266561 -> scaling image by 1.0216605511853067 mosaic get_fwhm: self.fwhm = 3.73 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3557 3888 and amp range 2006 4053 and image y 1448 1782 and amp range 0 2039 Range in image: x 0 332 , y 0 335 (with image size 332 x 335) dzp 0.001388291 -> scaling image by 1.0012794811153327 Faked up EXPNUM 75820120 Faked up EXPNUM 75820120 Faked up EXPNUM 75820120 90prime get_fwhm: self.fwhm = 3.95 Found 4 relevant rows in amp-corrections file. Found overlap: image x 587 779 and amp range 0 2015 and image y 814 1006 and amp range 0 2047 Range in image: x 0 193 , y 0 193 (with image size 193 x 193) dzp -0.024507914 -> scaling image by 0.9776802328037623 mosaic get_fwhm: self.fwhm = 3.85 Found 4 relevant rows in amp-corrections file. Found overlap: image x 853 1189 and amp range 0 2005 and image y 321 654 and amp range 0 2039 Range in image: x 0 337 , y 0 334 (with image size 337 x 334) dzp -0.005856539 -> scaling image by 0.9946204500110131 Faked up EXPNUM 75770145 Faked up EXPNUM 75770145 Faked up EXPNUM 75770145 90prime get_fwhm: self.fwhm = 3.95 Found 4 relevant rows in amp-corrections file. Found overlap: image x 580 772 and amp range 0 2015 and image y 811 1002 and amp range 0 2047 Range in image: x 0 193 , y 0 192 (with image size 193 x 192) dzp -0.03470582 -> scaling image by 0.9685402452207784 mosaic get_fwhm: self.fwhm = 6.19 Found 4 relevant rows in amp-corrections file. Found overlap: image x 1967 2300 and amp range 0 2047 and image y 2324 2658 and amp range 2040 4078 Range in image: x 0 81 , y 0 335 (with image size 334 x 335) dzp 0.003927584 -> scaling image by 1.0036239891963392 Found overlap: image x 1967 2300 and amp range 2048 4095 and image y 2324 2658 and amp range 2040 4078 Range in image: x 81 334 , y 0 335 (with image size 334 x 335) dzp -0.036442947 -> scaling image by 0.9669918668439786 Faked up EXPNUM 82570189 Faked up EXPNUM 82570189 Faked up EXPNUM 82570189 90prime get_fwhm: self.fwhm = 2.73 Found 4 relevant rows in amp-corrections file. Found overlap: image x 335 528 and amp range 0 2015 and image y 2343 2537 and amp range 2048 4095 Range in image: x 0 194 , y 0 195 (with image size 194 x 195) dzp 0.0459271 -> scaling image by 1.0432078344950917 Faked up EXPNUM 79280167 Faked up EXPNUM 79280167 Faked up EXPNUM 79280167 90prime get_fwhm: self.fwhm = 2.79 Found 4 relevant rows in amp-corrections file. Found overlap: image x 357 550 and amp range 0 2015 and image y 2351 2545 and amp range 2048 4095 Range in image: x 0 194 , y 0 195 (with image size 194 x 195) dzp 0.05525464 -> scaling image by 1.0522086198429996 Stage tims : Wall: 2.35 s, CPU: 0.82 s, VmPeak: 951 MB, VmSize: 951 MB, VmRSS: 175 MB, VmData: 208 MB, maxrss: 0.178704 MB Stage tims finished: 2020-08-07T12:08:30.491698 Resources for stage tims : 4.630 s worker CPU, 10.182 s worker Wall, Wall: 2.351 s, Cores in use: 2.32, Total efficiency (on 32 cores): 7.2 % Stage tims finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-tims.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-tims.p Running stage refs Running stage refs at 2020-08-07T12:08:30.512879 Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00906.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00743.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00822.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00823.fits Matched 8 large galaxies to Gaia stars. Matched 0 large galaxies to Tycho-2 stars. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/metrics/cus/reference-custom-283288p60683.fits Stage refs : Wall: 5.81 s, CPU: 4.35 s, VmPeak: 1020 MB, VmSize: 1014 MB, VmRSS: 240 MB, VmData: 271 MB, maxrss: 0.245736 MB Stage refs finished: 2020-08-07T12:08:36.325216 Resources for stage refs : 0.000 s worker CPU, 0.001 s worker Wall, Wall: 5.815 s, Cores in use: 0.75, Total efficiency (on 32 cores): 2.3 % Stage refs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-refs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-refs.p Running stage outliers Running stage outliers at 2020-08-07T12:08:36.505206 Stage outliers : Wall: 0.72 s, CPU: 0.31 s, VmPeak: 1020 MB, VmSize: 1014 MB, VmRSS: 241 MB, VmData: 272 MB, maxrss: 0.246752 MB Stage outliers finished: 2020-08-07T12:08:37.221209 Resources for stage outliers : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.719 s, Cores in use: 0.43, Total efficiency (on 32 cores): 1.3 % Stage outliers finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-outliers.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-outliers.p Running stage halos Running stage halos at 2020-08-07T12:08:43.164018 Stage halos : Wall: 0.33 s, CPU: 0.32 s, VmPeak: 1020 MB, VmSize: 1014 MB, VmRSS: 245 MB, VmData: 277 MB, maxrss: 0.250992 MB Stage halos finished: 2020-08-07T12:08:43.497162 Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.335 s, Cores in use: 0.97, Total efficiency (on 32 cores): 3.0 % Stage halos finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-halos.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-halos.p Running stage fit_on_coadds Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Running stage fit_on_coadds at 2020-08-07T12:08:43.523640 PSF sigmas in coadd pixels: ['2.01', '2.38', '2.91'] Using average PSF sigma 2.432027803763198 Gaussian PSF norm 0.11599159817884436 vs pixelized 0.095838696 average tim pixel scale / coadd scale: 1.7302539572830646 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-copsf-g.fits.fz PSF sigmas in coadd pixels: ['2.05', '2.08', '2.91'] Using average PSF sigma 2.3476229855258617 Gaussian PSF norm 0.12016188012858874 vs pixelized 0.10435013 average tim pixel scale / coadd scale: 1.7295054778570436 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-copsf-r.fits.fz PSF sigmas in coadd pixels: ['1.57', '2.61', '1.62'] Using average PSF sigma 1.9355495361597093 Gaussian PSF norm 0.14574403108978426 vs pixelized 0.116829835 average tim pixel scale / coadd scale: 0.9909552679297635 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-copsf-z.fits.fz Stage fit_on_coadds : Wall: 0.23 s, CPU: 0.20 s, VmPeak: 1020 MB, VmSize: 1019 MB, VmRSS: 258 MB, VmData: 286 MB, maxrss: 0.263968 MB Stage fit_on_coadds finished: 2020-08-07T12:08:43.752963 Warning: Stellar halo subtraction is only implemented for DECam Resources for stage fit_on_coadds : 0.249 s worker CPU, 0.396 s worker Wall, Wall: 0.231 s, Cores in use: 1.93, Total efficiency (on 32 cores): 6.0 % Stage fit_on_coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-fit_on_coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-fit_on_coadds.p Running stage srcs Running stage srcs at 2020-08-07T12:08:43.766460 SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])] Found 37 potential peaks Of 37 potential peaks: 15 in veto map, 2 cut by saddle test, 0 cut by aper test, 20 kept SED z : 20 new peaks Found 31 potential peaks Of 31 potential peaks: 30 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED r : 0 new peaks Found 29 potential peaks Of 29 potential peaks: 28 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED g : 0 new peaks Found 36 potential peaks Of 36 potential peaks: 33 in veto map, 2 cut by saddle test, 0 cut by aper test, 1 kept SED Flat : 1 new peaks Found 39 potential peaks Of 39 potential peaks: 36 in veto map, 1 cut by saddle test, 0 cut by aper test, 2 kept SED Red : 2 new peaks Adding 23 new sources Median coadd sky for g : 5.9140428e-05 Median coadd sky for r : 4.192812e-05 Median coadd sky for z : -1.774911e-05 Stage srcs : Wall: 0.52 s, CPU: 0.44 s, VmPeak: 1020 MB, VmSize: 1019 MB, VmRSS: 259 MB, VmData: 286 MB, maxrss: 0.264724 MB Stage srcs finished: 2020-08-07T12:08:44.290862 Resources for stage srcs : 0.151 s worker CPU, 0.176 s worker Wall, Wall: 0.527 s, Cores in use: 1.14, Total efficiency (on 32 cores): 3.5 % Stage srcs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-srcs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-srcs.p Running stage fitblobs Running stage fitblobs at 2020-08-07T12:08:44.310421 Blob 1 of 23, id: 10, sources: 1, size: 126x132, npix 7991, brick X: 102,228, Y: 98,230, one pixel: 200 98 Blob 2 of 23, id: 1, sources: 7, size: 80x137, npix 7535, brick X: 249,329, Y: 0,137, one pixel: 268 0 Blob 3 of 23, id: 6, sources: 4, size: 67x74, npix 3083, brick X: 95,162, Y: 45,119, one pixel: 142 45 Blob 4 of 23, id: 13, sources: 4, size: 72x83, npix 2948, brick X: 0,72, Y: 153,236, one pixel: 42 153 Blob 5 of 23, id: 21, sources: 2, size: 81x48, npix 2631, brick X: 106,187, Y: 281,329, one pixel: 124 281 Blob 6 of 23, id: 2, sources: 2, size: 54x57, npix 2299, brick X: 164,218, Y: 1,58, one pixel: 185 1 Blob 7 of 23, id: 19, sources: 2, size: 63x45, npix 1595, brick X: 139,202, Y: 252,297, one pixel: 153 252 Blob 8 of 23, id: 7, sources: 2, size: 41x46, npix 1200, brick X: 163,204, Y: 50,96, one pixel: 171 50 Blob 9 of 23, id: 4, sources: 2, size: 38x43, npix 908, brick X: 32,70, Y: 38,81, one pixel: 44 38 Blob 10 of 23, id: 17, sources: 1, size: 36x35, npix 871, brick X: 156,192, Y: 220,255, one pixel: 172 220 Blob 11 of 23, id: 11, sources: 1, size: 33x33, npix 801, brick X: 0,33, Y: 119,152, one pixel: 12 119 Blob 12 of 23, id: 15, sources: 1, size: 32x32, npix 701, brick X: 55,87, Y: 212,244, one pixel: 67 212 Blob 13 of 23, id: 0, sources: 1, size: 34x29, npix 667, brick X: 35,69, Y: 0,29, one pixel: 41 0 Blob 14 of 23, id: 9, sources: 1, size: 28x33, npix 627, brick X: 65,93, Y: 91,124, one pixel: 76 91 Blob 15 of 23, id: 12, sources: 1, size: 31x31, npix 624, brick X: 246,277, Y: 139,170, one pixel: 260 139 Blob 16 of 23, id: 16, sources: 1, size: 27x28, npix 511, brick X: 215,242, Y: 215,243, one pixel: 226 215 Blob 17 of 23, id: 20, sources: 1, size: 26x26, npix 426, brick X: 258,284, Y: 276,302, one pixel: 269 276 Blob 18 of 23, id: 8, sources: 1, size: 27x26, npix 420, brick X: 220,247, Y: 72,98, one pixel: 234 72 Blob 19 of 23, id: 14, sources: 1, size: 23x22, npix 316, brick X: 291,314, Y: 175,197, one pixel: 301 175 Blob 20 of 23, id: 3, sources: 1, size: 23x23, npix 307, brick X: 98,121, Y: 6,29, one pixel: 109 6 Blob 21 of 23, id: 22, sources: 1, size: 22x22, npix 281, brick X: 38,60, Y: 293,315, one pixel: 48 293 Blob 22 of 23, id: 18, sources: 1, size: 17x21, npix 233, brick X: 312,329, Y: 229,250, one pixel: 320 229 Blob 23 of 23, id: 5, sources: 2, size: 12x20, npix 153, brick X: 0,12, Y: 43,63, one pixel: 2 43 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (283.31563, 60.67591), pm (-2.4, -1.9), parallax -0.183 with NanoMaggies: g=18.1, r=16.8, z=15.4 Reset brightness to NanoMaggies: g=19.1, r=18.7, z=18.6 Blob 20 finished, total: Wall: 0.26 s, CPU: 0.25 s, VmPeak: 807 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.14708 MB Blob 22 finished, total: Wall: 0.26 s, CPU: 0.23 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 87 MB, VmData: 137 MB, maxrss: 0.08956 MB Blob 23 finished, total: Wall: 0.33 s, CPU: 0.29 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.089948 MB Blob 17 finished, total: Wall: 0.37 s, CPU: 0.35 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.08986 MB Blob 21 finished, total: Wall: 0.36 s, CPU: 0.36 s, VmPeak: 811 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.147144 MB Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (283.26563, 60.67036), pm (2.7, -14.3), parallax 0.969 with NanoMaggies: g=16.5, r=15.6, z=15.1 Reset brightness to NanoMaggies: g=20, r=18.6, z=17.4 Measuring 1 iterative sources Blob 19 finished, total: Wall: 0.50 s, CPU: 0.50 s, VmPeak: 669 MB, VmSize: 669 MB, VmRSS: 93 MB, VmData: 140 MB, maxrss: 0.094888 MB Blob 15 finished, total: Wall: 0.57 s, CPU: 0.55 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 88 MB, VmData: 138 MB, maxrss: 0.0903 MB Blob 16 finished, total: Wall: 0.57 s, CPU: 0.57 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 91 MB, VmData: 139 MB, maxrss: 0.093612 MB Measuring 1 iterative sources Blob 10 finished, total: Wall: 0.80 s, CPU: 0.79 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 90 MB, VmData: 138 MB, maxrss: 0.092452 MB Blob 18 finished, total: Wall: 0.80 s, CPU: 0.80 s, VmPeak: 810 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 182 MB, maxrss: 0.146784 MB Blob 12 finished, total: Wall: 0.83 s, CPU: 0.81 s, VmPeak: 810 MB, VmSize: 725 MB, VmRSS: 142 MB, VmData: 183 MB, maxrss: 0.144988 MB Blob 11 finished, total: Wall: 0.86 s, CPU: 0.83 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 89 MB, VmData: 138 MB, maxrss: 0.090712 MB Measuring 1 iterative sources Blob 14 finished, total: Wall: 0.99 s, CPU: 0.97 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 90 MB, VmData: 138 MB, maxrss: 0.091928 MB Blob 13 finished, total: Wall: 1.10 s, CPU: 1.10 s, VmPeak: 812 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.14604 MB Blob 9 finished, total: Wall: 1.11 s, CPU: 1.05 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 89 MB, VmData: 138 MB, maxrss: 0.091016 MB Blob 6 finished, total: Wall: 1.33 s, CPU: 1.27 s, VmPeak: 668 MB, VmSize: 667 MB, VmRSS: 90 MB, VmData: 138 MB, maxrss: 0.092984 MB Blob 7 finished, total: Wall: 1.34 s, CPU: 1.28 s, VmPeak: 668 MB, VmSize: 667 MB, VmRSS: 89 MB, VmData: 138 MB, maxrss: 0.09154 MB Blob 8 finished, total: Wall: 1.44 s, CPU: 1.44 s, VmPeak: 806 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.14716 MB Measuring 1 iterative sources Blob 5 finished, total: Wall: 1.86 s, CPU: 1.86 s, VmPeak: 811 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.146328 MB kdtree_internal.c:2152:kdtree_build_2_ddd: Data, N, or D is zero Exception in one_blob: brick custom-283288p60683, iblob 10 Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1641, in _bounce_one_blob result = one_blob(X) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 345, in run ras[not_galaxy], decs[not_galaxy], 1./3600.) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match (kd1,kd2) = _buildtrees(x1, x2) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees kd2 = spherematch_c.KdTree(fx2) SystemError: returned NULL without setting an error multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "/usr/local/lib/python/astrometry/util/timingpool.py", line 269, in __call__ R = self.func(*args, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1641, in _bounce_one_blob result = one_blob(X) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 345, in run ras[not_galaxy], decs[not_galaxy], 1./3600.) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match (kd1,kd2) = _buildtrees(x1, x2) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees kd2 = spherematch_c.KdTree(fx2) SystemError: returned NULL without setting an error """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3775, in sys.exit(main()) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3743, in main run_brick(opt.brick, survey, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3299, in run_brick initial_args=initargs, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage R = stagefunc(stage, **Px) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3288, in mystagefunc R = stagefunc(stage, mp=mp, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__ rtn = super(CallGlobalTime, self).__call__(stage, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__ return func(*self.args, **kwa) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1135, in stage_fitblobs r = Riter.next(timeout) File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 748, in next raise value SystemError: returned NULL without setting an error Something went wrong; please check the logfile. ERROR: galaxy PGC2605417; please check the logfile. Finished galaxy PGC2605417 in 0.335 minutes. Started working on galaxy PGC2605417 at Fri Aug 7 12:33:18 2020 Bands touching this brick, g r z python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 283.28852889391 60.6835168525139 --width 329 --height 329 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 runbrick.py starting at 2020-08-07T12:33:19.373184 legacypipe git version: DR9.5.11-watershed-4-gcc619972 Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '283.28852889391', '60.6835168525139', '--width', '329', '--height', '329', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 283.28852889391 60.6835168525139 --width 329 --height 329 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 Parsed RA,Dec 283.28852889391 60.6835168525139 Starting process 19784 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB Starting process 19785 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 19786 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 19787 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 19788 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059408 MB Starting process 19789 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059356 MB Starting process 19790 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059356 MB Starting process 19791 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 19792 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 19793 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 19794 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 19795 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB Starting process 19796 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 19797 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 19798 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 19799 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 19800 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB Starting process 19801 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059276 MB Starting process 19802 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 19803 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 19804 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 19805 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 19806 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB Starting process 19807 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB Starting process 19808 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB Starting process 19809 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB Starting process 19810 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB Starting process 19811 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059308 MB Starting process 19812 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB Starting process 19813 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB Starting process 19814 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB Starting process 19815 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059316 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage fit_on_coadds Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-08-07T12:33:19.756412 Keeping 9 CCDs: k4m_160607_101036_ooi_zd_ls9-CCD1 z expnum 94073 exptime 80.103 propid 2016A-0453 seeing 0.97 MJD 57546.424 object MzLS_688841_z mosaic/CP/V4.3/CP20160606/k4m_160607_101036_ooi_zd_ls9.fits.fz k4m_170927_053443_ooi_zd_ls9-CCD2 z expnum 141032 exptime 155.103 propid 2016A-0453 seeing 1.61 MJD 58023.232 object MzLS_688221_z mosaic/CP/V4.3/CP20170926/k4m_170927_053443_ooi_zd_ls9.fits.fz k4m_160606_094544_ooi_zd_ls9-CCD3 z expnum 93804 exptime 89.104 propid 2016A-0453 seeing 1.00 MJD 57545.407 object MzLS_689456_z mosaic/CP/V4.3/CP20160605/k4m_160606_094544_ooi_zd_ls9.fits.fz ksb_180519_100225_ooi_g_ls9-CCD2 g expnum 82570189 exptime 112.617 propid NOTBASS seeing 1.24 MJD 58257.418 object 38803 90prime/CP/V2.3/CP20180518/ksb_180519_100225_ooi_g_ls9.fits.fz ksb_170624_103052_ooi_r_ls9-CCD2 r expnum 79280167 exptime 136.0 propid BASS seeing 1.26 MJD 57928.438 object 38803 90prime/CP/V2.3/CP20170623/ksb_170624_103052_ooi_r_ls9.fits.fz ksb_180508_104536_ooi_r_ls9-CCD4 r expnum 82460215 exptime 117.421 propid NOTBASS seeing 1.28 MJD 58246.448 object 38801 90prime/CP/V2.3/CP20180507/ksb_180508_104536_ooi_r_ls9.fits.fz ksb_180517_111237_ooi_g_ls9-CCD4 g expnum 82550159 exptime 141.8 propid NOTBASS seeing 1.46 MJD 58255.467 object 38801 90prime/CP/V2.3/CP20180516/ksb_180517_111237_ooi_g_ls9.fits.fz ksb_160708_104905_ooi_g_ls9-CCD3 g expnum 75770145 exptime 212.6 propid BASS seeing 1.79 MJD 57577.451 object 37852 90prime/CP/V2.3/CP20160707/ksb_160708_104905_ooi_g_ls9.fits.fz ksb_160713_094902_ooi_r_ls9-CCD3 r expnum 75820120 exptime 109.0 propid BASS seeing 1.79 MJD 57582.409 object 37852 90prime/CP/V2.3/CP20160712/ksb_160713_094902_ooi_r_ls9.fits.fz Faked up EXPNUM 82460215 Faked up EXPNUM 82460215 Faked up EXPNUM 82460215 90prime get_fwhm: self.fwhm = 2.83 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2366 2558 and amp range 2016 4031 and image y 15 48 and amp range 0 2047 Range in image: x 0 193 , y 0 34 (with image size 193 x 34) dzp 0.043530058 -> scaling image by 1.0409072255640852 Faked up EXPNUM 79280167 Faked up EXPNUM 79280167 Faked up EXPNUM 79280167 90prime get_fwhm: self.fwhm = 2.79 Found 4 relevant rows in amp-corrections file. Found overlap: image x 357 550 and amp range 0 2015 and image y 2351 2545 and amp range 2048 4095 Range in image: x 0 194 , y 0 195 (with image size 194 x 195) dzp 0.05525464 -> scaling image by 1.0522086198429996 Faked up EXPNUM 82550159 Faked up EXPNUM 82550159 Faked up EXPNUM 82550159 90prime get_fwhm: self.fwhm = 3.23 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2350 2542 and amp range 2016 4031 and image y 15 83 and amp range 0 2047 Range in image: x 0 193 , y 0 69 (with image size 193 x 69) dzp 0.023266561 -> scaling image by 1.0216605511853067 Faked up EXPNUM 75820120 Faked up EXPNUM 75820120 Faked up EXPNUM 75820120 90prime get_fwhm: self.fwhm = 3.95 Found 4 relevant rows in amp-corrections file. Found overlap: image x 587 779 and amp range 0 2015 and image y 814 1006 and amp range 0 2047 Range in image: x 0 193 , y 0 193 (with image size 193 x 193) dzp -0.024507914 -> scaling image by 0.9776802328037623 mosaic get_fwhm: self.fwhm = 3.85 Found 4 relevant rows in amp-corrections file. Found overlap: image x 853 1189 and amp range 0 2005 and image y 321 654 and amp range 0 2039 Range in image: x 0 337 , y 0 334 (with image size 337 x 334) dzp -0.005856539 -> scaling image by 0.9946204500110131 mosaic get_fwhm: self.fwhm = 3.73 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3557 3888 and amp range 2006 4053 and image y 1448 1782 and amp range 0 2039 Range in image: x 0 332 , y 0 335 (with image size 332 x 335) dzp 0.001388291 -> scaling image by 1.0012794811153327 Faked up EXPNUM 82570189 Faked up EXPNUM 82570189 Faked up EXPNUM 82570189 90prime get_fwhm: self.fwhm = 2.73 Found 4 relevant rows in amp-corrections file. Found overlap: image x 335 528 and amp range 0 2015 and image y 2343 2537 and amp range 2048 4095 Range in image: x 0 194 , y 0 195 (with image size 194 x 195) dzp 0.0459271 -> scaling image by 1.0432078344950917 mosaic get_fwhm: self.fwhm = 6.19 Found 4 relevant rows in amp-corrections file. Found overlap: image x 1967 2300 and amp range 0 2047 and image y 2324 2658 and amp range 2040 4078 Range in image: x 0 81 , y 0 335 (with image size 334 x 335) dzp 0.003927584 -> scaling image by 1.0036239891963392 Found overlap: image x 1967 2300 and amp range 2048 4095 and image y 2324 2658 and amp range 2040 4078 Range in image: x 81 334 , y 0 335 (with image size 334 x 335) dzp -0.036442947 -> scaling image by 0.9669918668439786 Faked up EXPNUM 75770145 Faked up EXPNUM 75770145 Faked up EXPNUM 75770145 90prime get_fwhm: self.fwhm = 3.95 Found 4 relevant rows in amp-corrections file. Found overlap: image x 580 772 and amp range 0 2015 and image y 811 1002 and amp range 0 2047 Range in image: x 0 193 , y 0 192 (with image size 193 x 192) dzp -0.03470582 -> scaling image by 0.9685402452207784 Stage tims : Wall: 2.21 s, CPU: 0.76 s, VmPeak: 951 MB, VmSize: 951 MB, VmRSS: 175 MB, VmData: 208 MB, maxrss: 0.178768 MB Stage tims finished: 2020-08-07T12:33:21.971235 Resources for stage tims : 4.570 s worker CPU, 10.373 s worker Wall, Wall: 2.217 s, Cores in use: 2.40, Total efficiency (on 32 cores): 7.5 % Stage tims finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-tims.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-tims.p Running stage refs Running stage refs at 2020-08-07T12:33:21.992899 Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00906.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00743.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00822.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00823.fits Matched 8 large galaxies to Gaia stars. Matched 0 large galaxies to Tycho-2 stars. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/metrics/cus/reference-custom-283288p60683.fits Stage refs : Wall: 4.81 s, CPU: 4.32 s, VmPeak: 1020 MB, VmSize: 1014 MB, VmRSS: 240 MB, VmData: 271 MB, maxrss: 0.2458 MB Stage refs finished: 2020-08-07T12:33:26.806162 Resources for stage refs : 0.000 s worker CPU, 0.001 s worker Wall, Wall: 4.816 s, Cores in use: 0.90, Total efficiency (on 32 cores): 2.8 % Stage refs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-refs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-refs.p Running stage outliers Running stage outliers at 2020-08-07T12:33:26.829375 Stage outliers : Wall: 1.11 s, CPU: 0.33 s, VmPeak: 1020 MB, VmSize: 1014 MB, VmRSS: 241 MB, VmData: 272 MB, maxrss: 0.246812 MB Stage outliers finished: 2020-08-07T12:33:27.937673 Resources for stage outliers : 0.000 s worker CPU, 0.001 s worker Wall, Wall: 1.111 s, Cores in use: 0.30, Total efficiency (on 32 cores): 0.9 % Stage outliers finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-outliers.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-outliers.p Running stage halos Running stage halos at 2020-08-07T12:33:28.528986 Stage halos : Wall: 0.33 s, CPU: 0.31 s, VmPeak: 1020 MB, VmSize: 1014 MB, VmRSS: 245 MB, VmData: 277 MB, maxrss: 0.251 MB Stage halos finished: 2020-08-07T12:33:28.855688 Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.329 s, Cores in use: 0.96, Total efficiency (on 32 cores): 3.0 % Stage halos finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-halos.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-halos.p Running stage fit_on_coadds Warning: Stellar halo subtraction is only implemented for DECam Running stage fit_on_coadds at 2020-08-07T12:33:28.880423 PSF sigmas in coadd pixels: ['2.01', '2.38', '2.91'] Using average PSF sigma 2.432027803763198 Gaussian PSF norm 0.11599159817884436 vs pixelized 0.095838696 average tim pixel scale / coadd scale: 1.7302539572830646 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-copsf-g.fits.fz PSF sigmas in coadd pixels: ['2.05', '2.08', '2.91'] Using average PSF sigma 2.3476229855258617 Gaussian PSF norm 0.12016188012858874 vs pixelized 0.10435013 average tim pixel scale / coadd scale: 1.7295054778570436 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-copsf-r.fits.fz PSF sigmas in coadd pixels: ['1.57', '2.61', '1.62'] Using average PSF sigma 1.9355495361597093 Gaussian PSF norm 0.14574403108978426 vs pixelized 0.116829835 average tim pixel scale / coadd scale: 0.9909552679297635 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-copsf-z.fits.fz Stage fit_on_coadds : Wall: 1.00 s, CPU: 0.19 s, VmPeak: 1020 MB, VmSize: 1019 MB, VmRSS: 258 MB, VmData: 286 MB, maxrss: 0.263984 MB Stage fit_on_coadds finished: 2020-08-07T12:33:29.880457 Resources for stage fit_on_coadds : 0.244 s worker CPU, 0.366 s worker Wall, Wall: 1.002 s, Cores in use: 0.44, Total efficiency (on 32 cores): 1.4 % Stage fit_on_coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-fit_on_coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-fit_on_coadds.p Running stage srcs Warning: Stellar halo subtraction is only implemented for DECam Running stage srcs at 2020-08-07T12:33:29.950419 SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])] Found 37 potential peaks Of 37 potential peaks: 15 in veto map, 2 cut by saddle test, 0 cut by aper test, 20 kept SED z : 20 new peaks Found 31 potential peaks Of 31 potential peaks: 30 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED r : 0 new peaks Found 29 potential peaks Of 29 potential peaks: 28 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED g : 0 new peaks Found 36 potential peaks Of 36 potential peaks: 33 in veto map, 2 cut by saddle test, 0 cut by aper test, 1 kept SED Flat : 1 new peaks Found 39 potential peaks Of 39 potential peaks: 36 in veto map, 1 cut by saddle test, 0 cut by aper test, 2 kept SED Red : 2 new peaks Adding 23 new sources Median coadd sky for g : 5.9140417e-05 Median coadd sky for r : 4.192812e-05 Median coadd sky for z : -1.7749027e-05 Stage srcs : Wall: 0.66 s, CPU: 0.58 s, VmPeak: 1020 MB, VmSize: 1019 MB, VmRSS: 259 MB, VmData: 286 MB, maxrss: 0.264724 MB Stage srcs finished: 2020-08-07T12:33:30.610868 Resources for stage srcs : 0.157 s worker CPU, 0.182 s worker Wall, Wall: 0.663 s, Cores in use: 1.12, Total efficiency (on 32 cores): 3.5 % Stage srcs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-srcs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-srcs.p Running stage fitblobs Running stage fitblobs at 2020-08-07T12:33:30.638674 Blob 1 of 23, id: 10, sources: 1, size: 126x132, npix 7991, brick X: 102,228, Y: 98,230, one pixel: 200 98 Blob 2 of 23, id: 1, sources: 7, size: 80x137, npix 7535, brick X: 249,329, Y: 0,137, one pixel: 268 0 Blob 3 of 23, id: 6, sources: 4, size: 67x74, npix 3083, brick X: 95,162, Y: 45,119, one pixel: 142 45 Blob 4 of 23, id: 13, sources: 4, size: 72x83, npix 2948, brick X: 0,72, Y: 153,236, one pixel: 42 153 Blob 5 of 23, id: 21, sources: 2, size: 81x48, npix 2631, brick X: 106,187, Y: 281,329, one pixel: 124 281 Blob 6 of 23, id: 2, sources: 2, size: 54x57, npix 2299, brick X: 164,218, Y: 1,58, one pixel: 185 1 Blob 7 of 23, id: 19, sources: 2, size: 63x45, npix 1595, brick X: 139,202, Y: 252,297, one pixel: 153 252 Blob 8 of 23, id: 7, sources: 2, size: 41x46, npix 1200, brick X: 163,204, Y: 50,96, one pixel: 171 50 Blob 9 of 23, id: 4, sources: 2, size: 38x43, npix 908, brick X: 32,70, Y: 38,81, one pixel: 44 38 Blob 10 of 23, id: 17, sources: 1, size: 36x35, npix 871, brick X: 156,192, Y: 220,255, one pixel: 172 220 Blob 11 of 23, id: 11, sources: 1, size: 33x33, npix 801, brick X: 0,33, Y: 119,152, one pixel: 12 119 Blob 12 of 23, id: 15, sources: 1, size: 32x32, npix 701, brick X: 55,87, Y: 212,244, one pixel: 67 212 Blob 13 of 23, id: 0, sources: 1, size: 34x29, npix 667, brick X: 35,69, Y: 0,29, one pixel: 41 0 Blob 14 of 23, id: 9, sources: 1, size: 28x33, npix 627, brick X: 65,93, Y: 91,124, one pixel: 76 91 Blob 15 of 23, id: 12, sources: 1, size: 31x31, npix 624, brick X: 246,277, Y: 139,170, one pixel: 260 139 Blob 16 of 23, id: 16, sources: 1, size: 27x28, npix 511, brick X: 215,242, Y: 215,243, one pixel: 226 215 Blob 17 of 23, id: 20, sources: 1, size: 26x26, npix 426, brick X: 258,284, Y: 276,302, one pixel: 269 276 Blob 18 of 23, id: 8, sources: 1, size: 27x26, npix 420, brick X: 220,247, Y: 72,98, one pixel: 234 72 Blob 19 of 23, id: 14, sources: 1, size: 23x22, npix 316, brick X: 291,314, Y: 175,197, one pixel: 301 175 Blob 20 of 23, id: 3, sources: 1, size: 23x23, npix 307, brick X: 98,121, Y: 6,29, one pixel: 109 6 Blob 21 of 23, id: 22, sources: 1, size: 22x22, npix 281, brick X: 38,60, Y: 293,315, one pixel: 48 293 Blob 22 of 23, id: 18, sources: 1, size: 17x21, npix 233, brick X: 312,329, Y: 229,250, one pixel: 320 229 Blob 23 of 23, id: 5, sources: 2, size: 12x20, npix 153, brick X: 0,12, Y: 43,63, one pixel: 2 43 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (283.31563, 60.67591), pm (-2.4, -1.9), parallax -0.183 with NanoMaggies: g=18.1, r=16.8, z=15.4 Reset brightness to NanoMaggies: g=19.1, r=18.7, z=18.6 Blob 23 finished, total: Wall: 0.25 s, CPU: 0.19 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.089988 MB Blob 20 finished, total: Wall: 0.27 s, CPU: 0.27 s, VmPeak: 813 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.147108 MB Blob 22 finished, total: Wall: 0.30 s, CPU: 0.23 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.08972 MB Blob 17 finished, total: Wall: 0.35 s, CPU: 0.34 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 92 MB, VmData: 139 MB, maxrss: 0.09376 MB Blob 21 finished, total: Wall: 0.39 s, CPU: 0.39 s, VmPeak: 669 MB, VmSize: 669 MB, VmRSS: 93 MB, VmData: 140 MB, maxrss: 0.094748 MB Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (283.26563, 60.67036), pm (2.7, -14.3), parallax 0.969 with NanoMaggies: g=16.5, r=15.6, z=15.1 Reset brightness to NanoMaggies: g=20, r=18.6, z=17.4 Blob 12 finished, total: Wall: 0.62 s, CPU: 0.59 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 138 MB, maxrss: 0.090212 MB Blob 18 finished, total: Wall: 0.62 s, CPU: 0.62 s, VmPeak: 810 MB, VmSize: 725 MB, VmRSS: 142 MB, VmData: 183 MB, maxrss: 0.145488 MB Blob 19 finished, total: Wall: 0.63 s, CPU: 0.62 s, VmPeak: 807 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 182 MB, maxrss: 0.147124 MB Measuring 1 iterative sources Blob 16 finished, total: Wall: 0.68 s, CPU: 0.68 s, VmPeak: 811 MB, VmSize: 725 MB, VmRSS: 141 MB, VmData: 183 MB, maxrss: 0.144884 MB Measuring 1 iterative sources Blob 10 finished, total: Wall: 0.73 s, CPU: 0.73 s, VmPeak: 806 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.146952 MB Measuring 1 iterative sources Blob 15 finished, total: Wall: 0.79 s, CPU: 0.78 s, VmPeak: 810 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 182 MB, maxrss: 0.146796 MB Blob 14 finished, total: Wall: 0.99 s, CPU: 0.98 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 90 MB, VmData: 138 MB, maxrss: 0.092244 MB Blob 9 finished, total: Wall: 1.09 s, CPU: 1.05 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 89 MB, VmData: 138 MB, maxrss: 0.091072 MB Blob 11 finished, total: Wall: 1.10 s, CPU: 1.07 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 89 MB, VmData: 138 MB, maxrss: 0.091 MB Blob 7 finished, total: Wall: 1.17 s, CPU: 1.13 s, VmPeak: 668 MB, VmSize: 667 MB, VmRSS: 89 MB, VmData: 138 MB, maxrss: 0.09158 MB Blob 8 finished, total: Wall: 1.18 s, CPU: 1.18 s, VmPeak: 812 MB, VmSize: 725 MB, VmRSS: 142 MB, VmData: 183 MB, maxrss: 0.145688 MB Blob 6 finished, total: Wall: 1.31 s, CPU: 1.27 s, VmPeak: 668 MB, VmSize: 667 MB, VmRSS: 90 MB, VmData: 138 MB, maxrss: 0.092968 MB Blob 13 finished, total: Wall: 1.36 s, CPU: 1.34 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 90 MB, VmData: 138 MB, maxrss: 0.091904 MB Measuring 1 iterative sources Blob 5 finished, total: Wall: 1.73 s, CPU: 1.69 s, VmPeak: 669 MB, VmSize: 669 MB, VmRSS: 92 MB, VmData: 140 MB, maxrss: 0.093812 MB kdtree_internal.c:2152:kdtree_build_2_ddd: Data, N, or D is zero Exception in one_blob: brick custom-283288p60683, iblob 10 Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1641, in _bounce_one_blob result = one_blob(X) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 345, in run ras[not_galaxy], decs[not_galaxy], 1./3600.) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match (kd1,kd2) = _buildtrees(x1, x2) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees kd2 = spherematch_c.KdTree(fx2) SystemError: returned NULL without setting an error multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "/usr/local/lib/python/astrometry/util/timingpool.py", line 269, in __call__ R = self.func(*args, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1641, in _bounce_one_blob result = one_blob(X) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 345, in run ras[not_galaxy], decs[not_galaxy], 1./3600.) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match (kd1,kd2) = _buildtrees(x1, x2) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees kd2 = spherematch_c.KdTree(fx2) SystemError: returned NULL without setting an error """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3775, in sys.exit(main()) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3743, in main run_brick(opt.brick, survey, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3299, in run_brick initial_args=initargs, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage R = stagefunc(stage, **Px) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3288, in mystagefunc R = stagefunc(stage, mp=mp, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__ rtn = super(CallGlobalTime, self).__call__(stage, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__ return func(*self.args, **kwa) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1135, in stage_fitblobs r = Riter.next(timeout) File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 748, in next raise value SystemError: returned NULL without setting an error Something went wrong; please check the logfile. ERROR: galaxy PGC2605417; please check the logfile. Finished galaxy PGC2605417 in 0.243 minutes. Started working on galaxy PGC2605417 at Fri Aug 7 13:33:57 2020 Bands touching this brick, g r z python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 283.28852889391 60.6835168525139 --width 329 --height 329 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 runbrick.py starting at 2020-08-07T13:33:58.033443 legacypipe git version: DR9.5.11-watershed-5-g77ea3463 Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '283.28852889391', '60.6835168525139', '--width', '329', '--height', '329', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 283.28852889391 60.6835168525139 --width 329 --height 329 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 Parsed RA,Dec 283.28852889391 60.6835168525139 Starting process 23539 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB Starting process 23540 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 23541 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 23542 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 23543 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05954 MB Starting process 23544 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059552 MB Starting process 23545 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059552 MB Starting process 23546 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 23547 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 23548 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 23549 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 23550 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB Starting process 23551 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB Starting process 23552 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 23553 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 23554 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB Starting process 23555 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB Starting process 23556 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059276 MB Starting process 23557 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 23558 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 23559 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 23560 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 23561 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB Starting process 23562 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB Starting process 23563 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB Starting process 23564 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB Starting process 23565 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB Starting process 23566 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059308 MB Starting process 23567 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB Starting process 23568 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB Starting process 23569 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB Starting process 23570 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059316 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage fit_on_coadds Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-08-07T13:33:58.551660 Keeping 9 CCDs: k4m_160607_101036_ooi_zd_ls9-CCD1 z expnum 94073 exptime 80.103 propid 2016A-0453 seeing 0.97 MJD 57546.424 object MzLS_688841_z mosaic/CP/V4.3/CP20160606/k4m_160607_101036_ooi_zd_ls9.fits.fz k4m_170927_053443_ooi_zd_ls9-CCD2 z expnum 141032 exptime 155.103 propid 2016A-0453 seeing 1.61 MJD 58023.232 object MzLS_688221_z mosaic/CP/V4.3/CP20170926/k4m_170927_053443_ooi_zd_ls9.fits.fz k4m_160606_094544_ooi_zd_ls9-CCD3 z expnum 93804 exptime 89.104 propid 2016A-0453 seeing 1.00 MJD 57545.407 object MzLS_689456_z mosaic/CP/V4.3/CP20160605/k4m_160606_094544_ooi_zd_ls9.fits.fz ksb_180519_100225_ooi_g_ls9-CCD2 g expnum 82570189 exptime 112.617 propid NOTBASS seeing 1.24 MJD 58257.418 object 38803 90prime/CP/V2.3/CP20180518/ksb_180519_100225_ooi_g_ls9.fits.fz ksb_170624_103052_ooi_r_ls9-CCD2 r expnum 79280167 exptime 136.0 propid BASS seeing 1.26 MJD 57928.438 object 38803 90prime/CP/V2.3/CP20170623/ksb_170624_103052_ooi_r_ls9.fits.fz ksb_180508_104536_ooi_r_ls9-CCD4 r expnum 82460215 exptime 117.421 propid NOTBASS seeing 1.28 MJD 58246.448 object 38801 90prime/CP/V2.3/CP20180507/ksb_180508_104536_ooi_r_ls9.fits.fz ksb_180517_111237_ooi_g_ls9-CCD4 g expnum 82550159 exptime 141.8 propid NOTBASS seeing 1.46 MJD 58255.467 object 38801 90prime/CP/V2.3/CP20180516/ksb_180517_111237_ooi_g_ls9.fits.fz ksb_160708_104905_ooi_g_ls9-CCD3 g expnum 75770145 exptime 212.6 propid BASS seeing 1.79 MJD 57577.451 object 37852 90prime/CP/V2.3/CP20160707/ksb_160708_104905_ooi_g_ls9.fits.fz ksb_160713_094902_ooi_r_ls9-CCD3 r expnum 75820120 exptime 109.0 propid BASS seeing 1.79 MJD 57582.409 object 37852 90prime/CP/V2.3/CP20160712/ksb_160713_094902_ooi_r_ls9.fits.fz Faked up EXPNUM 75820120 Faked up EXPNUM 75820120 Faked up EXPNUM 75820120 90prime get_fwhm: self.fwhm = 3.95 Found 4 relevant rows in amp-corrections file. Found overlap: image x 587 779 and amp range 0 2015 and image y 814 1006 and amp range 0 2047 Range in image: x 0 193 , y 0 193 (with image size 193 x 193) dzp -0.024507914 -> scaling image by 0.9776802328037623 Faked up EXPNUM 82460215 Faked up EXPNUM 82460215 Faked up EXPNUM 82460215 90prime get_fwhm: self.fwhm = 2.83 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2366 2558 and amp range 2016 4031 and image y 15 48 and amp range 0 2047 Range in image: x 0 193 , y 0 34 (with image size 193 x 34) dzp 0.043530058 -> scaling image by 1.0409072255640852 Faked up EXPNUM 79280167 Faked up EXPNUM 79280167 Faked up EXPNUM 79280167 90prime get_fwhm: self.fwhm = 2.79 Found 4 relevant rows in amp-corrections file. Found overlap: image x 357 550 and amp range 0 2015 and image y 2351 2545 and amp range 2048 4095 Range in image: x 0 194 , y 0 195 (with image size 194 x 195) dzp 0.05525464 -> scaling image by 1.0522086198429996 Faked up EXPNUM 82570189 Faked up EXPNUM 82570189 Faked up EXPNUM 82570189 90prime get_fwhm: self.fwhm = 2.73 Found 4 relevant rows in amp-corrections file. mosaic get_fwhm: self.fwhm = 3.73 Found 4 relevant rows in amp-corrections file. Found overlap: image x 335 528 and amp range 0 2015 and image y 2343 2537 and amp range 2048 4095 Range in image: x 0 194 , y 0 195 (with image size 194 x 195) dzp 0.0459271 -> scaling image by 1.0432078344950917 Found overlap: image x 3557 3888 and amp range 2006 4053 and image y 1448 1782 and amp range 0 2039 Range in image: x 0 332 , y 0 335 (with image size 332 x 335) dzp 0.001388291 -> scaling image by 1.0012794811153327 mosaic get_fwhm: self.fwhm = 6.19 Found 4 relevant rows in amp-corrections file. Found overlap: image x 1967 2300 and amp range 0 2047 and image y 2324 2658 and amp range 2040 4078 Range in image: x 0 81 , y 0 335 (with image size 334 x 335) dzp 0.003927584 -> scaling image by 1.0036239891963392 Found overlap: image x 1967 2300 and amp range 2048 4095 and image y 2324 2658 and amp range 2040 4078 Range in image: x 81 334 , y 0 335 (with image size 334 x 335) dzp -0.036442947 -> scaling image by 0.9669918668439786 Faked up EXPNUM 82550159 Faked up EXPNUM 82550159 Faked up EXPNUM 82550159 90prime get_fwhm: self.fwhm = 3.23 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2350 2542 and amp range 2016 4031 and image y 15 83 and amp range 0 2047 Range in image: x 0 193 , y 0 69 (with image size 193 x 69) dzp 0.023266561 -> scaling image by 1.0216605511853067 mosaic get_fwhm: self.fwhm = 3.85 Found 4 relevant rows in amp-corrections file. Found overlap: image x 853 1189 and amp range 0 2005 and image y 321 654 and amp range 0 2039 Range in image: x 0 337 , y 0 334 (with image size 337 x 334) dzp -0.005856539 -> scaling image by 0.9946204500110131 Faked up EXPNUM 75770145 Faked up EXPNUM 75770145 Faked up EXPNUM 75770145 90prime get_fwhm: self.fwhm = 3.95 Found 4 relevant rows in amp-corrections file. Found overlap: image x 580 772 and amp range 0 2015 and image y 811 1002 and amp range 0 2047 Range in image: x 0 193 , y 0 192 (with image size 193 x 192) dzp -0.03470582 -> scaling image by 0.9685402452207784 Stage tims : Wall: 5.56 s, CPU: 0.79 s, VmPeak: 951 MB, VmSize: 951 MB, VmRSS: 174 MB, VmData: 208 MB, maxrss: 0.178516 MB Stage tims finished: 2020-08-07T13:34:04.111950 Resources for stage tims : 4.347 s worker CPU, 16.872 s worker Wall, Wall: 5.563 s, Cores in use: 0.92, Total efficiency (on 32 cores): 2.9 % Stage tims finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-tims.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-tims.p Running stage refs Running stage refs at 2020-08-07T13:34:04.152416 Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00906.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00743.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00822.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00823.fits Matched 8 large galaxies to Gaia stars. Matched 0 large galaxies to Tycho-2 stars. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/metrics/cus/reference-custom-283288p60683.fits Stage refs : Wall: 4.74 s, CPU: 4.33 s, VmPeak: 1020 MB, VmSize: 1014 MB, VmRSS: 240 MB, VmData: 271 MB, maxrss: 0.245812 MB Stage refs finished: 2020-08-07T13:34:08.893106 Resources for stage refs : 0.000 s worker CPU, 0.001 s worker Wall, Wall: 4.743 s, Cores in use: 0.91, Total efficiency (on 32 cores): 2.9 % Stage refs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-refs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-refs.p Running stage outliers Running stage outliers at 2020-08-07T13:34:08.915685 Stage outliers : Wall: 0.35 s, CPU: 0.24 s, VmPeak: 1020 MB, VmSize: 1014 MB, VmRSS: 241 MB, VmData: 272 MB, maxrss: 0.246752 MB Stage outliers finished: 2020-08-07T13:34:09.266382 Resources for stage outliers : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 0.353 s, Cores in use: 0.68, Total efficiency (on 32 cores): 2.1 % Stage outliers finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-outliers.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-outliers.p Running stage halos Running stage halos at 2020-08-07T13:34:09.289726 Stage halos : Wall: 0.21 s, CPU: 0.20 s, VmPeak: 1020 MB, VmSize: 1014 MB, VmRSS: 245 MB, VmData: 277 MB, maxrss: 0.250996 MB Stage halos finished: 2020-08-07T13:34:09.500730 Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.213 s, Cores in use: 0.95, Total efficiency (on 32 cores): 3.0 % Stage halos finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-halos.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-halos.p Running stage fit_on_coadds Warning: Stellar halo subtraction is only implemented for DECam Running stage fit_on_coadds at 2020-08-07T13:34:09.534140 PSF sigmas in coadd pixels: ['2.01', '2.38', '2.91'] Using average PSF sigma 2.432027803763198 Gaussian PSF norm 0.11599159817884436 vs pixelized 0.095838696 average tim pixel scale / coadd scale: 1.7302539572830646 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-copsf-g.fits.fz PSF sigmas in coadd pixels: ['2.05', '2.08', '2.91'] Using average PSF sigma 2.3476229855258617 Gaussian PSF norm 0.12016188012858874 vs pixelized 0.10435013 average tim pixel scale / coadd scale: 1.7295054778570436 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-copsf-r.fits.fz PSF sigmas in coadd pixels: ['1.57', '2.61', '1.62'] Using average PSF sigma 1.9355495361597093 Gaussian PSF norm 0.14574403108978426 vs pixelized 0.116829835 average tim pixel scale / coadd scale: 0.9909552679297635 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-copsf-z.fits.fz Stage fit_on_coadds : Wall: 0.24 s, CPU: 0.19 s, VmPeak: 1020 MB, VmSize: 1019 MB, VmRSS: 259 MB, VmData: 288 MB, maxrss: 0.265488 MB Stage fit_on_coadds finished: 2020-08-07T13:34:09.771529 Resources for stage fit_on_coadds : 0.286 s worker CPU, 0.445 s worker Wall, Wall: 0.240 s, Cores in use: 1.98, Total efficiency (on 32 cores): 6.2 % Stage fit_on_coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-fit_on_coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-fit_on_coadds.p Running stage srcs Running stage srcs at 2020-08-07T13:34:09.785057 SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])] Found 37 potential peaks Of 37 potential peaks: 15 in veto map, 2 cut by saddle test, 0 cut by aper test, 20 kept SED z : 20 new peaks Found 31 potential peaks Of 31 potential peaks: 30 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED r : 0 new peaks Found 29 potential peaks Of 29 potential peaks: 28 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED g : 0 new peaks Found 36 potential peaks Of 36 potential peaks: 33 in veto map, 2 cut by saddle test, 0 cut by aper test, 1 kept SED Flat : 1 new peaks Found 39 potential peaks Of 39 potential peaks: 36 in veto map, 1 cut by saddle test, 0 cut by aper test, 2 kept SED Red : 2 new peaks Adding 23 new sources Median coadd sky for g : 5.9140417e-05 Median coadd sky for r : 4.192812e-05 Median coadd sky for z : -1.774911e-05 Stage srcs : Wall: 0.63 s, CPU: 0.43 s, VmPeak: 1020 MB, VmSize: 1019 MB, VmRSS: 260 MB, VmData: 288 MB, maxrss: 0.2662 MB Stage srcs finished: 2020-08-07T13:34:10.418283 Resources for stage srcs : 0.143 s worker CPU, 0.206 s worker Wall, Wall: 0.635 s, Cores in use: 0.90, Total efficiency (on 32 cores): 2.8 % Stage srcs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-srcs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-srcs.p Running stage fitblobs Running stage fitblobs at 2020-08-07T13:34:10.433810 Blob 1 of 23, id: 10, sources: 1, size: 126x132, npix 7991, brick X: 102,228, Y: 98,230, one pixel: 200 98 Blob 2 of 23, id: 1, sources: 7, size: 80x137, npix 7535, brick X: 249,329, Y: 0,137, one pixel: 268 0 Blob 3 of 23, id: 6, sources: 4, size: 67x74, npix 3083, brick X: 95,162, Y: 45,119, one pixel: 142 45 Blob 4 of 23, id: 13, sources: 4, size: 72x83, npix 2948, brick X: 0,72, Y: 153,236, one pixel: 42 153 Blob 5 of 23, id: 21, sources: 2, size: 81x48, npix 2631, brick X: 106,187, Y: 281,329, one pixel: 124 281 Blob 6 of 23, id: 2, sources: 2, size: 54x57, npix 2299, brick X: 164,218, Y: 1,58, one pixel: 185 1 Blob 7 of 23, id: 19, sources: 2, size: 63x45, npix 1595, brick X: 139,202, Y: 252,297, one pixel: 153 252 Blob 8 of 23, id: 7, sources: 2, size: 41x46, npix 1200, brick X: 163,204, Y: 50,96, one pixel: 171 50 Blob 9 of 23, id: 4, sources: 2, size: 38x43, npix 908, brick X: 32,70, Y: 38,81, one pixel: 44 38 Blob 10 of 23, id: 17, sources: 1, size: 36x35, npix 871, brick X: 156,192, Y: 220,255, one pixel: 172 220 Blob 11 of 23, id: 11, sources: 1, size: 33x33, npix 801, brick X: 0,33, Y: 119,152, one pixel: 12 119 Blob 12 of 23, id: 15, sources: 1, size: 32x32, npix 701, brick X: 55,87, Y: 212,244, one pixel: 67 212 Blob 13 of 23, id: 0, sources: 1, size: 34x29, npix 667, brick X: 35,69, Y: 0,29, one pixel: 41 0 Blob 14 of 23, id: 9, sources: 1, size: 28x33, npix 627, brick X: 65,93, Y: 91,124, one pixel: 76 91 Blob 15 of 23, id: 12, sources: 1, size: 31x31, npix 624, brick X: 246,277, Y: 139,170, one pixel: 260 139 Blob 16 of 23, id: 16, sources: 1, size: 27x28, npix 511, brick X: 215,242, Y: 215,243, one pixel: 226 215 Blob 17 of 23, id: 20, sources: 1, size: 26x26, npix 426, brick X: 258,284, Y: 276,302, one pixel: 269 276 Blob 18 of 23, id: 8, sources: 1, size: 27x26, npix 420, brick X: 220,247, Y: 72,98, one pixel: 234 72 Blob 19 of 23, id: 14, sources: 1, size: 23x22, npix 316, brick X: 291,314, Y: 175,197, one pixel: 301 175 Blob 20 of 23, id: 3, sources: 1, size: 23x23, npix 307, brick X: 98,121, Y: 6,29, one pixel: 109 6 Blob 21 of 23, id: 22, sources: 1, size: 22x22, npix 281, brick X: 38,60, Y: 293,315, one pixel: 48 293 Blob 22 of 23, id: 18, sources: 1, size: 17x21, npix 233, brick X: 312,329, Y: 229,250, one pixel: 320 229 Blob 23 of 23, id: 5, sources: 2, size: 12x20, npix 153, brick X: 0,12, Y: 43,63, one pixel: 2 43 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (283.31563, 60.67591), pm (-2.4, -1.9), parallax -0.183 with NanoMaggies: g=18.1, r=16.8, z=15.4 Reset brightness to NanoMaggies: g=19.1, r=18.7, z=18.6 Blob 22 finished, total: Wall: 0.29 s, CPU: 0.19 s, VmPeak: 810 MB, VmSize: 725 MB, VmRSS: 142 MB, VmData: 183 MB, maxrss: 0.145 MB Blob 20 finished, total: Wall: 0.39 s, CPU: 0.30 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 90 MB, VmData: 138 MB, maxrss: 0.091824 MB Blob 23 finished, total: Wall: 0.40 s, CPU: 0.28 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.089992 MB Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (283.26563, 60.67036), pm (2.7, -14.3), parallax 0.969 with NanoMaggies: g=16.5, r=15.6, z=15.1 Reset brightness to NanoMaggies: g=20, r=18.6, z=17.4 Blob 17 finished, total: Wall: 0.48 s, CPU: 0.35 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.089884 MB Blob 16 finished, total: Wall: 0.54 s, CPU: 0.44 s, VmPeak: 669 MB, VmSize: 669 MB, VmRSS: 93 MB, VmData: 140 MB, maxrss: 0.094824 MB Blob 10 finished, total: Wall: 0.65 s, CPU: 0.55 s, VmPeak: 812 MB, VmSize: 725 MB, VmRSS: 142 MB, VmData: 183 MB, maxrss: 0.145504 MB Blob 15 finished, total: Wall: 0.64 s, CPU: 0.57 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 138 MB, maxrss: 0.09016 MB Measuring 1 iterative sources Blob 21 finished, total: Wall: 0.70 s, CPU: 0.59 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.0899 MB Measuring 1 iterative sources Blob 19 finished, total: Wall: 0.79 s, CPU: 0.71 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 92 MB, VmData: 139 MB, maxrss: 0.093892 MB Blob 14 finished, total: Wall: 0.83 s, CPU: 0.75 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 93 MB, VmData: 139 MB, maxrss: 0.094992 MB Blob 12 finished, total: Wall: 0.91 s, CPU: 0.83 s, VmPeak: 811 MB, VmSize: 725 MB, VmRSS: 142 MB, VmData: 183 MB, maxrss: 0.145208 MB Blob 18 finished, total: Wall: 0.91 s, CPU: 0.84 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 88 MB, VmData: 137 MB, maxrss: 0.089992 MB Measuring 1 iterative sources Blob 9 finished, total: Wall: 1.09 s, CPU: 1.02 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 89 MB, VmData: 138 MB, maxrss: 0.091072 MB Blob 11 finished, total: Wall: 1.16 s, CPU: 1.07 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 92 MB, VmData: 139 MB, maxrss: 0.093984 MB Blob 13 finished, total: Wall: 1.22 s, CPU: 1.12 s, VmPeak: 813 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.145984 MB Blob 6 finished, total: Wall: 1.24 s, CPU: 1.15 s, VmPeak: 810 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.147356 MB Blob 7 finished, total: Wall: 1.29 s, CPU: 1.19 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 90 MB, VmData: 139 MB, maxrss: 0.092156 MB Blob 8 finished, total: Wall: 1.42 s, CPU: 1.34 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 89 MB, VmData: 138 MB, maxrss: 0.091396 MB Measuring 1 iterative sources Blob 5 finished, total: Wall: 1.94 s, CPU: 1.85 s, VmPeak: 669 MB, VmSize: 669 MB, VmRSS: 92 MB, VmData: 140 MB, maxrss: 0.093844 MB kdtree_internal.c:2152:kdtree_build_2_ddd: Data, N, or D is zero Exception in one_blob: brick custom-283288p60683, iblob 10 Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1641, in _bounce_one_blob result = one_blob(X) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 345, in run ras[not_galaxy], decs[not_galaxy], 1./3600.) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match (kd1,kd2) = _buildtrees(x1, x2) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees kd2 = spherematch_c.KdTree(fx2) SystemError: returned NULL without setting an error multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 121, in worker result = (True, func(*args, **kwds)) File "/usr/local/lib/python/astrometry/util/timingpool.py", line 269, in __call__ R = self.func(*args, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1641, in _bounce_one_blob result = one_blob(X) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 124, in one_blob B = ob.run(B, reoptimize=reoptimize, iterative_detection=iterative) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/oneblob.py", line 345, in run ras[not_galaxy], decs[not_galaxy], 1./3600.) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 81, in match_radec X = match(xyz1, xyz2, r, notself=notself, indexlist=indexlist) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 294, in match (kd1,kd2) = _buildtrees(x1, x2) File "/usr/local/lib/python/astrometry/libkd/spherematch.py", line 176, in _buildtrees kd2 = spherematch_c.KdTree(fx2) SystemError: returned NULL without setting an error """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3775, in sys.exit(main()) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3743, in main run_brick(opt.brick, survey, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3299, in run_brick initial_args=initargs, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage R = stagefunc(stage, **Px) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3288, in mystagefunc R = stagefunc(stage, mp=mp, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__ rtn = super(CallGlobalTime, self).__call__(stage, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__ return func(*self.args, **kwa) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 1135, in stage_fitblobs r = Riter.next(timeout) File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 748, in next raise value SystemError: returned NULL without setting an error Something went wrong; please check the logfile. ERROR: galaxy PGC2605417; please check the logfile. Finished galaxy PGC2605417 in 0.271 minutes. Started working on galaxy PGC2605417 at Fri Aug 7 14:47:43 2020 Bands touching this brick, g r z python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 283.28852889391 60.6835168525139 --width 329 --height 329 --pixscale 0.262 --threads 1 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 runbrick.py starting at 2020-08-07T14:47:44.012477 legacypipe git version: DR9.5.11-watershed-6-g6a2880fc Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '283.28852889391', '60.6835168525139', '--width', '329', '--height', '329', '--pixscale', '0.262', '--threads', '1', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 283.28852889391 60.6835168525139 --width 329 --height 329 --pixscale 0.262 --threads 1 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 Parsed RA,Dec 283.28852889391 60.6835168525139 Starting process 61320 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 641 MB, VmSize: 641 MB, VmRSS: 99 MB, VmData: 128 MB, maxrss: 0.107708 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-srcs.p and forcing stage srcs Runstage fit_on_coadds Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-fit_on_coadds.p and forcing stage fit_on_coadds Runstage halos Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-halos.p and forcing stage halos Runstage outliers Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-outliers.p and forcing stage outliers Runstage refs Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-refs.p and forcing stage refs Runstage tims Ignoring pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-tims.p and forcing stage tims Running stage tims Running stage tims at 2020-08-07T14:47:44.464835 Keeping 9 CCDs: k4m_160607_101036_ooi_zd_ls9-CCD1 z expnum 94073 exptime 80.103 propid 2016A-0453 seeing 0.97 MJD 57546.424 object MzLS_688841_z mosaic/CP/V4.3/CP20160606/k4m_160607_101036_ooi_zd_ls9.fits.fz k4m_170927_053443_ooi_zd_ls9-CCD2 z expnum 141032 exptime 155.103 propid 2016A-0453 seeing 1.61 MJD 58023.232 object MzLS_688221_z mosaic/CP/V4.3/CP20170926/k4m_170927_053443_ooi_zd_ls9.fits.fz k4m_160606_094544_ooi_zd_ls9-CCD3 z expnum 93804 exptime 89.104 propid 2016A-0453 seeing 1.00 MJD 57545.407 object MzLS_689456_z mosaic/CP/V4.3/CP20160605/k4m_160606_094544_ooi_zd_ls9.fits.fz ksb_180519_100225_ooi_g_ls9-CCD2 g expnum 82570189 exptime 112.617 propid NOTBASS seeing 1.24 MJD 58257.418 object 38803 90prime/CP/V2.3/CP20180518/ksb_180519_100225_ooi_g_ls9.fits.fz ksb_170624_103052_ooi_r_ls9-CCD2 r expnum 79280167 exptime 136.0 propid BASS seeing 1.26 MJD 57928.438 object 38803 90prime/CP/V2.3/CP20170623/ksb_170624_103052_ooi_r_ls9.fits.fz ksb_180508_104536_ooi_r_ls9-CCD4 r expnum 82460215 exptime 117.421 propid NOTBASS seeing 1.28 MJD 58246.448 object 38801 90prime/CP/V2.3/CP20180507/ksb_180508_104536_ooi_r_ls9.fits.fz ksb_180517_111237_ooi_g_ls9-CCD4 g expnum 82550159 exptime 141.8 propid NOTBASS seeing 1.46 MJD 58255.467 object 38801 90prime/CP/V2.3/CP20180516/ksb_180517_111237_ooi_g_ls9.fits.fz ksb_160708_104905_ooi_g_ls9-CCD3 g expnum 75770145 exptime 212.6 propid BASS seeing 1.79 MJD 57577.451 object 37852 90prime/CP/V2.3/CP20160707/ksb_160708_104905_ooi_g_ls9.fits.fz ksb_160713_094902_ooi_r_ls9-CCD3 r expnum 75820120 exptime 109.0 propid BASS seeing 1.79 MJD 57582.409 object 37852 90prime/CP/V2.3/CP20160712/ksb_160713_094902_ooi_r_ls9.fits.fz mosaic get_fwhm: self.fwhm = 3.73 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3557 3888 and amp range 2006 4053 and image y 1448 1782 and amp range 0 2039 Range in image: x 0 332 , y 0 335 (with image size 332 x 335) dzp 0.001388291 -> scaling image by 1.0012794811153327 mosaic get_fwhm: self.fwhm = 6.19 Found 4 relevant rows in amp-corrections file. Found overlap: image x 1967 2300 and amp range 0 2047 and image y 2324 2658 and amp range 2040 4078 Range in image: x 0 81 , y 0 335 (with image size 334 x 335) dzp 0.003927584 -> scaling image by 1.0036239891963392 Found overlap: image x 1967 2300 and amp range 2048 4095 and image y 2324 2658 and amp range 2040 4078 Range in image: x 81 334 , y 0 335 (with image size 334 x 335) dzp -0.036442947 -> scaling image by 0.9669918668439786 mosaic get_fwhm: self.fwhm = 3.85 Found 4 relevant rows in amp-corrections file. Found overlap: image x 853 1189 and amp range 0 2005 and image y 321 654 and amp range 0 2039 Range in image: x 0 337 , y 0 334 (with image size 337 x 334) dzp -0.005856539 -> scaling image by 0.9946204500110131 Faked up EXPNUM 82570189 Faked up EXPNUM 82570189 Faked up EXPNUM 82570189 90prime get_fwhm: self.fwhm = 2.73 Found 4 relevant rows in amp-corrections file. Found overlap: image x 335 528 and amp range 0 2015 and image y 2343 2537 and amp range 2048 4095 Range in image: x 0 194 , y 0 195 (with image size 194 x 195) dzp 0.0459271 -> scaling image by 1.0432078344950917 Faked up EXPNUM 79280167 Faked up EXPNUM 79280167 Faked up EXPNUM 79280167 90prime get_fwhm: self.fwhm = 2.79 Found 4 relevant rows in amp-corrections file. Found overlap: image x 357 550 and amp range 0 2015 and image y 2351 2545 and amp range 2048 4095 Range in image: x 0 194 , y 0 195 (with image size 194 x 195) dzp 0.05525464 -> scaling image by 1.0522086198429996 Faked up EXPNUM 82460215 Faked up EXPNUM 82460215 Faked up EXPNUM 82460215 90prime get_fwhm: self.fwhm = 2.83 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2366 2558 and amp range 2016 4031 and image y 15 48 and amp range 0 2047 Range in image: x 0 193 , y 0 34 (with image size 193 x 34) dzp 0.043530058 -> scaling image by 1.0409072255640852 Faked up EXPNUM 82550159 Faked up EXPNUM 82550159 Faked up EXPNUM 82550159 90prime get_fwhm: self.fwhm = 3.23 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2350 2542 and amp range 2016 4031 and image y 15 83 and amp range 0 2047 Range in image: x 0 193 , y 0 69 (with image size 193 x 69) dzp 0.023266561 -> scaling image by 1.0216605511853067 Faked up EXPNUM 75770145 Faked up EXPNUM 75770145 Faked up EXPNUM 75770145 90prime get_fwhm: self.fwhm = 3.95 Found 4 relevant rows in amp-corrections file. Found overlap: image x 580 772 and amp range 0 2015 and image y 811 1002 and amp range 0 2047 Range in image: x 0 193 , y 0 192 (with image size 193 x 192) dzp -0.03470582 -> scaling image by 0.9685402452207784 Faked up EXPNUM 75820120 Faked up EXPNUM 75820120 Faked up EXPNUM 75820120 90prime get_fwhm: self.fwhm = 3.95 Found 4 relevant rows in amp-corrections file. Found overlap: image x 587 779 and amp range 0 2015 and image y 814 1006 and amp range 0 2047 Range in image: x 0 193 , y 0 193 (with image size 193 x 193) dzp -0.024507914 -> scaling image by 0.9776802328037623 Stage tims : Wall: 6.47 s, CPU: 1.81 s, VmPeak: 864 MB, VmSize: 758 MB, VmRSS: 176 MB, VmData: 195 MB, maxrss: 0.180576 MB Stage tims finished: 2020-08-07T14:47:50.939177 Resources for stage tims : Wall: 6.47 s, CPU: 1.81 s Stage tims finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-tims.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-tims.p Running stage refs Running stage refs at 2020-08-07T14:47:52.048061 Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00906.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00743.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00822.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00823.fits Matched 8 large galaxies to Gaia stars. Matched 0 large galaxies to Tycho-2 stars. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/metrics/cus/reference-custom-283288p60683.fits Stage refs : Wall: 4.81 s, CPU: 4.39 s, VmPeak: 864 MB, VmSize: 809 MB, VmRSS: 229 MB, VmData: 247 MB, maxrss: 0.234608 MB Stage refs finished: 2020-08-07T14:47:56.858318 Resources for stage refs : Wall: 4.81 s, CPU: 4.39 s Stage refs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-refs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-refs.p Running stage outliers Running stage outliers at 2020-08-07T14:47:56.887913 Stage outliers : Wall: 0.29 s, CPU: 0.27 s, VmPeak: 864 MB, VmSize: 809 MB, VmRSS: 230 MB, VmData: 247 MB, maxrss: 0.235328 MB Stage outliers finished: 2020-08-07T14:47:57.175508 Resources for stage outliers : Wall: 0.29 s, CPU: 0.27 s Stage outliers finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-outliers.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-outliers.p Running stage halos Running stage halos at 2020-08-07T14:47:57.351245 Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Stage halos : Wall: 0.25 s, CPU: 0.25 s, VmPeak: 864 MB, VmSize: 809 MB, VmRSS: 230 MB, VmData: 247 MB, maxrss: 0.235336 MB Stage halos finished: 2020-08-07T14:47:57.601990 Resources for stage halos : Wall: 0.25 s, CPU: 0.25 s Stage halos finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-halos.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-halos.p Running stage fit_on_coadds Running stage fit_on_coadds at 2020-08-07T14:47:57.622887 PSF sigmas in coadd pixels: ['2.01', '2.38', '2.91'] Using average PSF sigma 2.432027803763198 Gaussian PSF norm 0.11599159817884436 vs pixelized 0.095838696 average tim pixel scale / coadd scale: 1.7302539572830646 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-copsf-g.fits.fz PSF sigmas in coadd pixels: ['2.05', '2.08', '2.91'] Using average PSF sigma 2.3476229855258617 Gaussian PSF norm 0.12016188012858874 vs pixelized 0.10435013 average tim pixel scale / coadd scale: 1.7295054778570436 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-copsf-r.fits.fz PSF sigmas in coadd pixels: ['1.57', '2.61', '1.62'] Using average PSF sigma 1.9355495361597093 Gaussian PSF norm 0.14574403108978426 vs pixelized 0.116829835 average tim pixel scale / coadd scale: 0.9909552679297635 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-copsf-z.fits.fz Stage fit_on_coadds : Wall: 0.32 s, CPU: 0.29 s, VmPeak: 864 MB, VmSize: 809 MB, VmRSS: 230 MB, VmData: 247 MB, maxrss: 0.235604 MB Stage fit_on_coadds finished: 2020-08-07T14:47:57.947467 Resources for stage fit_on_coadds : Wall: 0.32 s, CPU: 0.29 s Stage fit_on_coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-fit_on_coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-fit_on_coadds.p Running stage srcs Running stage srcs at 2020-08-07T14:47:57.956970 SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])] Found 37 potential peaks Of 37 potential peaks: 15 in veto map, 2 cut by saddle test, 0 cut by aper test, 20 kept SED z : 20 new peaks Found 31 potential peaks Of 31 potential peaks: 30 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED r : 0 new peaks Found 29 potential peaks Of 29 potential peaks: 28 in veto map, 1 cut by saddle test, 0 cut by aper test, 0 kept SED g : 0 new peaks Found 36 potential peaks Of 36 potential peaks: 33 in veto map, 2 cut by saddle test, 0 cut by aper test, 1 kept SED Flat : 1 new peaks Found 39 potential peaks Of 39 potential peaks: 36 in veto map, 1 cut by saddle test, 0 cut by aper test, 2 kept SED Red : 2 new peaks Adding 23 new sources Median coadd sky for g : 5.9140417e-05 Median coadd sky for r : 4.192812e-05 Median coadd sky for z : -1.7749027e-05 Stage srcs : Wall: 0.55 s, CPU: 0.54 s, VmPeak: 864 MB, VmSize: 809 MB, VmRSS: 231 MB, VmData: 247 MB, maxrss: 0.236436 MB Stage srcs finished: 2020-08-07T14:47:58.503624 Resources for stage srcs : Wall: 0.55 s, CPU: 0.54 s Stage srcs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-srcs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-srcs.p Running stage fitblobs Running stage fitblobs at 2020-08-07T14:47:58.614411 Blob 1 of 23, id: 10, sources: 1, size: 126x132, npix 7991, brick X: 102,228, Y: 98,230, one pixel: 200 98 Known galaxy. Initial shape: EllipseWithPriors(0.25): log r_e=0.948351, ee1=0.186563, ee2=0.452735 Blob 1 finished, total: Wall: 3.62 s, CPU: 3.62 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241576 MB Blob 2 of 23, id: 1, sources: 7, size: 80x137, npix 7535, brick X: 249,329, Y: 0,137, one pixel: 268 0 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (283.26563, 60.67036), pm (2.7, -14.3), parallax 0.969 with NanoMaggies: g=16.5, r=15.6, z=15.1 Reset brightness to NanoMaggies: g=20, r=18.6, z=17.4 Measuring 2 iterative sources Blob 2 finished, total: Wall: 4.63 s, CPU: 4.63 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 3 of 23, id: 6, sources: 4, size: 67x74, npix 3083, brick X: 95,162, Y: 45,119, one pixel: 142 45 Measuring 1 iterative sources Blob 3 finished, total: Wall: 2.65 s, CPU: 2.65 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 4 of 23, id: 13, sources: 4, size: 72x83, npix 2948, brick X: 0,72, Y: 153,236, one pixel: 42 153 Blob 4 finished, total: Wall: 1.80 s, CPU: 1.80 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 5 of 23, id: 21, sources: 2, size: 81x48, npix 2631, brick X: 106,187, Y: 281,329, one pixel: 124 281 Measuring 1 iterative sources Blob 5 finished, total: Wall: 1.38 s, CPU: 1.38 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 6 of 23, id: 2, sources: 2, size: 54x57, npix 2299, brick X: 164,218, Y: 1,58, one pixel: 185 1 Measuring 1 iterative sources Blob 6 finished, total: Wall: 0.92 s, CPU: 0.92 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 7 of 23, id: 19, sources: 2, size: 63x45, npix 1595, brick X: 139,202, Y: 252,297, one pixel: 153 252 Blob 7 finished, total: Wall: 0.91 s, CPU: 0.91 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 8 of 23, id: 7, sources: 2, size: 41x46, npix 1200, brick X: 163,204, Y: 50,96, one pixel: 171 50 Measuring 1 iterative sources Blob 8 finished, total: Wall: 0.98 s, CPU: 0.98 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 9 of 23, id: 4, sources: 2, size: 38x43, npix 908, brick X: 32,70, Y: 38,81, one pixel: 44 38 Blob 9 finished, total: Wall: 0.72 s, CPU: 0.72 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 10 of 23, id: 17, sources: 1, size: 36x35, npix 871, brick X: 156,192, Y: 220,255, one pixel: 172 220 Blob 10 finished, total: Wall: 0.44 s, CPU: 0.44 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 11 of 23, id: 11, sources: 1, size: 33x33, npix 801, brick X: 0,33, Y: 119,152, one pixel: 12 119 Measuring 1 iterative sources Blob 11 finished, total: Wall: 0.66 s, CPU: 0.66 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 12 of 23, id: 15, sources: 1, size: 32x32, npix 701, brick X: 55,87, Y: 212,244, one pixel: 67 212 Blob 12 finished, total: Wall: 0.47 s, CPU: 0.47 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 13 of 23, id: 0, sources: 1, size: 34x29, npix 667, brick X: 35,69, Y: 0,29, one pixel: 41 0 Blob 13 finished, total: Wall: 0.91 s, CPU: 0.91 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 14 of 23, id: 9, sources: 1, size: 28x33, npix 627, brick X: 65,93, Y: 91,124, one pixel: 76 91 Blob 14 finished, total: Wall: 0.60 s, CPU: 0.60 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 15 of 23, id: 12, sources: 1, size: 31x31, npix 624, brick X: 246,277, Y: 139,170, one pixel: 260 139 Blob 15 finished, total: Wall: 0.44 s, CPU: 0.44 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 16 of 23, id: 16, sources: 1, size: 27x28, npix 511, brick X: 215,242, Y: 215,243, one pixel: 226 215 Blob 16 finished, total: Wall: 0.35 s, CPU: 0.35 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 17 of 23, id: 20, sources: 1, size: 26x26, npix 426, brick X: 258,284, Y: 276,302, one pixel: 269 276 Blob 17 finished, total: Wall: 0.27 s, CPU: 0.27 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 18 of 23, id: 8, sources: 1, size: 27x26, npix 420, brick X: 220,247, Y: 72,98, one pixel: 234 72 Blob 18 finished, total: Wall: 0.49 s, CPU: 0.49 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 19 of 23, id: 14, sources: 1, size: 23x22, npix 316, brick X: 291,314, Y: 175,197, one pixel: 301 175 Blob 19 finished, total: Wall: 0.39 s, CPU: 0.39 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 20 of 23, id: 3, sources: 1, size: 23x23, npix 307, brick X: 98,121, Y: 6,29, one pixel: 109 6 Blob 20 finished, total: Wall: 0.15 s, CPU: 0.15 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 21 of 23, id: 22, sources: 1, size: 22x22, npix 281, brick X: 38,60, Y: 293,315, one pixel: 48 293 Blob 21 finished, total: Wall: 0.30 s, CPU: 0.30 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 22 of 23, id: 18, sources: 1, size: 17x21, npix 233, brick X: 312,329, Y: 229,250, one pixel: 320 229 Blob 22 finished, total: Wall: 0.10 s, CPU: 0.10 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Blob 23 of 23, id: 5, sources: 2, size: 12x20, npix 153, brick X: 0,12, Y: 43,63, one pixel: 2 43 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (283.31563, 60.67591), pm (-2.4, -1.9), parallax -0.183 with NanoMaggies: g=18.1, r=16.8, z=15.4 Reset brightness to NanoMaggies: g=19.1, r=18.7, z=18.6 Blob 23 finished, total: Wall: 0.14 s, CPU: 0.14 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Old catalog: 48 New catalog: 47 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/metrics/cus/blobs-custom-283288p60683.fits.gz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/metrics/cus/all-models-custom-283288p60683.fits Stage fitblobs : Wall: 23.78 s, CPU: 23.76 s, VmPeak: 864 MB, VmSize: 818 MB, VmRSS: 236 MB, VmData: 247 MB, maxrss: 0.241768 MB Stage fitblobs finished: 2020-08-07T14:48:22.391924 Resources for stage fitblobs : Wall: 23.78 s, CPU: 23.76 s Stage fitblobs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-fitblobs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-fitblobs.p Running stage coadds Running stage coadds at 2020-08-07T14:48:22.469951 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-ccds.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-image-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-invvar-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-nexp-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-depth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-galdepth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-psfsize-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-model-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-chi2-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-blobmodel-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-image-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-invvar-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-nexp-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-depth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-galdepth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-psfsize-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-model-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-chi2-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-blobmodel-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-image-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-invvar-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-nexp-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-depth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-galdepth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-psfsize-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-model-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-chi2-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-blobmodel-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-depth.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/tmp-legacysurvey-custom-283288p60683-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/tmp-legacysurvey-custom-283288p60683-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/tmp-legacysurvey-custom-283288p60683-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/tmp-legacysurvey-custom-283288p60683-resid.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-resid.jpg Stage coadds : Wall: 2.12 s, CPU: 1.38 s, VmPeak: 864 MB, VmSize: 829 MB, VmRSS: 241 MB, VmData: 253 MB, maxrss: 0.247152 MB Stage coadds finished: 2020-08-07T14:48:24.591919 Resources for stage coadds : Wall: 2.12 s, CPU: 1.38 s Stage coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-coadds.p Running stage wise_forced Running stage wise_forced at 2020-08-07T14:48:24.726627 Cut to 1 unWISE tiles Reading WISE tile 2820p605 band 1 Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w1-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w1-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w1-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w1-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.07 s, CPU: 0.07 s, VmPeak: 880 MB, VmSize: 877 MB, VmRSS: 235 MB, VmData: 240 MB, maxrss: 0.302004 MB Reading WISE tile 2820p605 band 2 Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w2-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w2-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w2-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w2-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.05 s, CPU: 0.05 s, VmPeak: 935 MB, VmSize: 877 MB, VmRSS: 236 MB, VmData: 240 MB, maxrss: 0.302004 MB Reading WISE tile 2820p605 band 3 Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w3-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w3-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w3-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.06 s, CPU: 0.06 s, VmPeak: 935 MB, VmSize: 877 MB, VmRSS: 236 MB, VmData: 240 MB, maxrss: 0.302004 MB Reading WISE tile 2820p605 band 4 Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w4-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w4-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/282/2820p605/unwise-2820p605-w4-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.07 s, CPU: 0.07 s, VmPeak: 935 MB, VmSize: 925 MB, VmRSS: 284 MB, VmData: 288 MB, maxrss: 0.302004 MB Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-image-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-model-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-invvar-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-image-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-model-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-invvar-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-image-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-model-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-invvar-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-image-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-model-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-invvar-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/tmp-legacysurvey-custom-283288p60683-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/tmp-legacysurvey-custom-283288p60683-wisemodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-wisemodel.jpg Setting WISE apphot Setting WISE apphot Setting WISE apphot Setting WISE apphot Stage wise_forced : Wall: 4.56 s, CPU: 3.37 s, VmPeak: 935 MB, VmSize: 874 MB, VmRSS: 229 MB, VmData: 236 MB, maxrss: 0.302004 MB Stage wise_forced finished: 2020-08-07T14:48:29.285121 Resources for stage wise_forced : Wall: 4.56 s, CPU: 3.37 s Stage wise_forced finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-wise_forced.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-wise_forced.p Running stage writecat Running stage writecat at 2020-08-07T14:48:29.381156 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/coadd/cus/custom-283288p60683/legacysurvey-custom-283288p60683-maskbits.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/tractor-i/cus/tractor-custom-283288p60683.fits Reading SFD maps... END Rejected (1), keyrecords following the END keyrecord were ignored. END Rejected (1), keyrecords following the END keyrecord were ignored. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/tractor/cus/tractor-custom-283288p60683.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/tractor/cus/brick-custom-283288p60683.sha256sum Stage writecat : Wall: 0.75 s, CPU: 0.58 s, VmPeak: 1018 MB, VmSize: 890 MB, VmRSS: 246 MB, VmData: 248 MB, maxrss: 0.382988 MB Stage writecat finished: 2020-08-07T14:48:30.128096 Resources for stage writecat : Wall: 0.75 s, CPU: 0.59 s Stage writecat finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-writecat.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2605417/PGC2605417-largegalaxy-writecat.p All done: Wall: 45.81 s, CPU: 36.78 s Finished galaxy PGC2605417 in 0.792 minutes.