Started working on galaxy 3C390.3 at Sun Jun 7 17:23:14 2020 Bands touching this brick, g r z python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 280.53776550000003 79.7713425 --width 275 --height 275 --pixscale 0.262 --threads 4 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 runbrick.py starting at 2020-06-07T17:23:15.952323 legacypipe git version: DR9.5.2 Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '280.53776550000003', '79.7713425', '--width', '275', '--height', '275', '--pixscale', '0.262', '--threads', '4', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-%%(stage)s.p', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 280.53776550000003 79.7713425 --width 275 --height 275 --pixscale 0.262 --threads 4 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 Parsed RA,Dec 280.53776550000003 79.7713425 Starting process 47487 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 632 MB, VmSize: 632 MB, VmRSS: 56 MB, VmData: 127 MB, maxrss: 0.057304 MB Starting process 47488 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 632 MB, VmSize: 632 MB, VmRSS: 56 MB, VmData: 127 MB, maxrss: 0.057304 MB Starting process 47489 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 632 MB, VmSize: 632 MB, VmRSS: 56 MB, VmData: 127 MB, maxrss: 0.057304 MB Starting process 47490 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 632 MB, VmSize: 632 MB, VmRSS: 56 MB, VmData: 127 MB, maxrss: 0.057308 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage fit_on_coadds Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-06-07T17:23:16.352828 Keeping 12 CCDs: k4m_171221_030001_ooi_zd_ls9.fits.fz k4m_171221_030001_ooi_zd_ls9-CCD3 z exptime 126.103 propid 2016A-0453 seeing 1.20 MJD 58108.125 object MzLS_730082_z k4m_171008_060834_ooi_zd_ls9.fits.fz k4m_171008_060834_ooi_zd_ls9-CCD3 z exptime 200.103 propid 2016A-0453 seeing 1.31 MJD 58034.256 object MzLS_730542_z k4m_171112_035616_ooi_zd_ls9.fits.fz k4m_171112_035616_ooi_zd_ls9-CCD4 z exptime 238.103 propid 2016A-0453 seeing 1.20 MJD 58069.164 object MzLS_730314_z k4m_171125_052741_ooi_zd_ls9.fits.fz k4m_171125_052741_ooi_zd_ls9-CCD4 z exptime 130.104 propid 2016A-0453 seeing 0.77 MJD 58082.228 object MzLS_730766_z ksb_180722_100116_ooi_g_ls9.fits.fz ksb_180722_100116_ooi_g_ls9-CCD3 g exptime 187.334 propid NOTBASS seeing 1.55 MJD 58321.418 object 52423 ksb_180703_103559_ooi_r_ls9.fits.fz ksb_180703_103559_ooi_r_ls9-CCD3 r exptime 250.0 propid NOTBASS seeing 1.34 MJD 58302.442 object 52423 ksb_180707_082623_ooi_r_ls9.fits.fz ksb_180707_082623_ooi_r_ls9-CCD2 r exptime 141.273 propid NOTBASS seeing 1.43 MJD 58306.352 object 52412 ksb_180618_081820_ooi_g_ls9.fits.fz ksb_180618_081820_ooi_g_ls9-CCD2 g exptime 217.51 propid NOTBASS seeing 2.07 MJD 58287.346 object 52412 ksb_160712_104905_ooi_g_ls9.fits.fz ksb_160712_104905_ooi_g_ls9-CCD3 g exptime 234.7 propid BASS seeing 1.83 MJD 57581.451 object 51902 ksb_180703_081020_ooi_r_ls9.fits.fz ksb_180703_081020_ooi_r_ls9-CCD3 r exptime 250.0 propid NOTBASS seeing 1.51 MJD 58302.341 object 51902 ksb_180608_104824_ooi_r_ls9.fits.fz ksb_180608_104824_ooi_r_ls9-CCD4 r exptime 250.0 propid NOTBASS seeing 1.56 MJD 58277.450 object 52421 ksb_180722_102838_ooi_g_ls9.fits.fz ksb_180722_102838_ooi_g_ls9-CCD4 g exptime 187.334 propid NOTBASS seeing 1.37 MJD 58321.437 object 52421 mosaic get_fwhm: self.fwhm = 4.61 Found 4 relevant rows in amp-corrections file. Found overlap: image x 7 293 and amp range 0 2005 and image y 1347 1628 and amp range 0 2039 Range in image: x 0 287 , y 0 282 (with image size 287 x 282) dzp 0.0014252509 -> scaling image by 1.0013135665633692 mosaic get_fwhm: self.fwhm = 4.63 Found 4 relevant rows in amp-corrections file. Found overlap: image x 1137 1425 and amp range 0 2047 and image y 2885 3173 and amp range 2022 4060 Range in image: x 0 289 , y 0 289 (with image size 289 x 289) dzp -0.016192675 -> scaling image by 0.9851966584456431 mosaic get_fwhm: self.fwhm = 5.05 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2722 3008 and amp range 2006 4053 and image y 3041 3332 and amp range 2040 4078 Range in image: x 0 287 , y 0 292 (with image size 287 x 292) dzp 0.0056365696 -> scaling image by 1.005204971477435 mosaic get_fwhm: self.fwhm = 2.97 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3926 4088 and amp range 2048 4095 and image y 2709 2995 and amp range 2022 4060 Range in image: x 0 163 , y 0 287 (with image size 163 x 287) dzp -0.019603321 -> scaling image by 0.9821066946375949 Faked up EXPNUM 83020151 Faked up EXPNUM 83020151 Faked up EXPNUM 83020151 90prime get_fwhm: self.fwhm = 2.95 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3906 4016 and amp range 2016 4031 and image y 670 830 and amp range 0 2047 Range in image: x 0 111 , y 0 161 (with image size 111 x 161) dzp 0.0057031885 -> scaling image by 1.00526665101287 Faked up EXPNUM 83060136 Faked up EXPNUM 83060136 Faked up EXPNUM 83060136 90prime get_fwhm: self.fwhm = 3.15 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3569 3737 and amp range 2016 4031 and image y 1113 1283 and amp range 0 2047 Range in image: x 0 169 , y 0 171 (with image size 169 x 171) dzp 0.014483032 -> scaling image by 1.0134287315188357 Faked up EXPNUM 83210121 Faked up EXPNUM 83210121 Faked up EXPNUM 83210121 90prime get_fwhm: self.fwhm = 3.41 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3892 4016 and amp range 2016 4031 and image y 663 823 and amp range 0 2047 Range in image: x 0 125 , y 0 161 (with image size 125 x 161) dzp -0.0012233759 -> scaling image by 0.9988738637196966 Faked up EXPNUM 82870117 Faked up EXPNUM 82870117 Faked up EXPNUM 82870117 90prime get_fwhm: self.fwhm = 4.57 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3543 3712 and amp range 2016 4031 and image y 1122 1292 and amp range 0 2047 Range in image: x 0 170 , y 0 171 (with image size 170 x 171) dzp 0.009189803 -> scaling image by 1.0085000434534261 Faked up EXPNUM 83210128 Faked up EXPNUM 83210128 Faked up EXPNUM 83210128 90prime get_fwhm: self.fwhm = 3.03 Found 4 relevant rows in amp-corrections file. Found overlap: image x 1689 1851 and amp range 0 2015 and image y 2560 2722 and amp range 2048 4095 Range in image: x 0 163 , y 0 163 (with image size 163 x 163) dzp 0.027550658 -> scaling image by 1.0256997821704772 Faked up EXPNUM 82770144 Faked up EXPNUM 82770144 Faked up EXPNUM 82770144 90prime get_fwhm: self.fwhm = 3.43 WARNING: image median 0.01743126 is more than 1 sigma away from zero! Found 4 relevant rows in amp-corrections file. Found overlap: image x 1698 1859 and amp range 0 2015 and image y 2562 2725 and amp range 2048 4095 Range in image: x 0 162 , y 0 164 (with image size 162 x 164) dzp 0.04491909 -> scaling image by 1.0422397564357018 Stage tims : Wall: 6.68 s, CPU: 0.93 s, VmPeak: 942 MB, VmSize: 942 MB, VmRSS: 172 MB, VmData: 206 MB, maxrss: 0.566544 MB Stage tims finished: 2020-06-07T17:23:23.028223 Faked up EXPNUM 75810159 Faked up EXPNUM 75810159 Faked up EXPNUM 75810159 Faked up EXPNUM 83020121 Faked up EXPNUM 83020121 Faked up EXPNUM 83020121 Resources for stage tims : 3.652 s worker CPU, 15.789 s worker Wall, Wall: 6.677 s, Cores in use: 0.69, Total efficiency (on 4 cores): 17.2 % Stage tims finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-tims.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-tims.p Running stage refs Running stage refs at 2020-06-07T17:23:23.121523 Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00137.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00106.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00105.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00078.fits Matched 7 large galaxies to Gaia stars. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/reference-custom-280537p79771.fits Stage refs : Wall: 4.46 s, CPU: 3.11 s, VmPeak: 980 MB, VmSize: 974 MB, VmRSS: 205 MB, VmData: 238 MB, maxrss: 0.566544 MB Stage refs finished: 2020-06-07T17:23:27.583041 Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 4.463 s, Cores in use: 0.70, Total efficiency (on 4 cores): 17.4 % Stage refs finished Running stage outliers Running stage outliers at 2020-06-07T17:23:27.584626 Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/outlier-mask-custom-280537p79771.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/outliers-pre-custom-280537p79771.jpg Image ksb_180722_102838_ooi_g_ls9-CCD4 g : masking 334 positive outlier pixels and 0 negative outlier pixels Image ksb_180707_082623_ooi_r_ls9-CCD2 r : masking 55 positive outlier pixels and 0 negative outlier pixels Image k4m_171221_030001_ooi_zd_ls9-CCD3 z : masking 78 positive outlier pixels and 0 negative outlier pixels Image k4m_171125_052741_ooi_zd_ls9-CCD4 z : masking 621 positive outlier pixels and 0 negative outlier pixels Image k4m_171008_060834_ooi_zd_ls9-CCD3 z : masking 939 positive outlier pixels and 0 negative outlier pixels Image k4m_171112_035616_ooi_zd_ls9-CCD4 z : masking 677 positive outlier pixels and 0 negative outlier pixels Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/outlier-mask-custom-280537p79771.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/outliers-post-custom-280537p79771.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/outliers-masked-pos-custom-280537p79771.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/outliers-masked-neg-custom-280537p79771.jpg Stage outliers : Wall: 1.23 s, CPU: 0.80 s, VmPeak: 983 MB, VmSize: 983 MB, VmRSS: 219 MB, VmData: 249 MB, maxrss: 0.566544 MB Stage outliers finished: 2020-06-07T17:23:28.814873 Resources for stage outliers : 1.365 s worker CPU, 1.394 s worker Wall, Wall: 1.231 s, Cores in use: 1.76, Total efficiency (on 4 cores): 44.1 % Stage outliers finished Running stage halos Running stage halos at 2020-06-07T17:23:28.816547 Stage halos : Wall: 0.20 s, CPU: 0.19 s, VmPeak: 983 MB, VmSize: 983 MB, VmRSS: 219 MB, VmData: 249 MB, maxrss: 0.566544 MB Stage halos finished: 2020-06-07T17:23:29.020551 Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.205 s, Cores in use: 0.95, Total efficiency (on 4 cores): 23.7 % Stage halos finished Running stage fit_on_coadds Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Running stage fit_on_coadds at 2020-06-07T17:23:29.022190 PSF sigmas in coadd pixels: ['2.51', '3.37', '2.23'] Using average PSF sigma 2.702722108297493 Gaussian PSF norm 0.10437432354137814 vs pixelized 0.08759851 average tim pixel scale / coadd scale: 1.730637261395179 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-copsf-g.fits.fz PSF sigmas in coadd pixels: ['2.17', '2.32', '2.53'] Using average PSF sigma 2.3389166742461818 Gaussian PSF norm 0.12060916700454734 vs pixelized 0.095788084 average tim pixel scale / coadd scale: 1.730248225383631 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-copsf-r.fits.fz PSF sigmas in coadd pixels: ['1.94', '2.13', '1.95', '1.25'] Using average PSF sigma 1.820011506738735 Gaussian PSF norm 0.15499615839207617 vs pixelized 0.13379963 average tim pixel scale / coadd scale: 0.9912280830152986 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-copsf-z.fits.fz Stage fit_on_coadds : Wall: 0.32 s, CPU: 0.26 s, VmPeak: 983 MB, VmSize: 983 MB, VmRSS: 222 MB, VmData: 250 MB, maxrss: 0.566544 MB Stage fit_on_coadds finished: 2020-06-07T17:23:29.347108 Resources for stage fit_on_coadds : 0.441 s worker CPU, 0.452 s worker Wall, Wall: 0.326 s, Cores in use: 2.15, Total efficiency (on 4 cores): 53.8 % Stage fit_on_coadds finished Running stage srcs Running stage srcs at 2020-06-07T17:23:29.349789 SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])] Found 30 potential peaks Of 30 potential peaks: 12 in veto map, 8 cut by saddle test, 0 cut by aper test, 10 kept SED z : 10 new peaks Found 163 potential peaks Of 163 potential peaks: 149 in veto map, 9 cut by saddle test, 0 cut by aper test, 5 kept SED r : 5 new peaks Found 106 potential peaks Of 106 potential peaks: 99 in veto map, 6 cut by saddle test, 0 cut by aper test, 1 kept SED g : 1 new peaks Found 157 potential peaks Of 157 potential peaks: 149 in veto map, 7 cut by saddle test, 0 cut by aper test, 1 kept SED Flat : 1 new peaks Found 177 potential peaks Of 177 potential peaks: 169 in veto map, 8 cut by saddle test, 0 cut by aper test, 0 kept SED Red : 0 new peaks Adding 17 new sources Median coadd sky for g : 0.0007846718 Median coadd sky for r : 0.002032939 Median coadd sky for z : 0.00012593961 Stage srcs : Wall: 0.72 s, CPU: 0.65 s, VmPeak: 983 MB, VmSize: 983 MB, VmRSS: 222 MB, VmData: 250 MB, maxrss: 0.566544 MB Stage srcs finished: 2020-06-07T17:23:30.068583 Resources for stage srcs : 0.148 s worker CPU, 0.148 s worker Wall, Wall: 0.720 s, Cores in use: 1.11, Total efficiency (on 4 cores): 27.8 % Stage srcs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-srcs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-srcs.p Running stage fitblobs Running stage fitblobs at 2020-06-07T17:23:30.083521 Blob 1 of 4, id: 0, sources: 17, size: 240x275, npix 50812, brick X: 0,240, Y: 0,275, one pixel: 0 0 Blob 2 of 4, id: 3, sources: 5, size: 83x42, npix 2180, brick X: 0,83, Y: 233,275, one pixel: 10 233 Blob 3 of 4, id: 2, sources: 2, size: 26x54, npix 1028, brick X: 249,275, Y: 194,248, one pixel: 262 194 Blob 4 of 4, id: 1, sources: 1, size: 22x25, npix 340, brick X: 243,265, Y: 63,88, one pixel: 253 63 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (281.40926, 79.94293), pm (8.9, 64.4), parallax 11.292 with NanoMaggies: g=6.41, r=6.34, z=6.44 Reset brightness to NanoMaggies: g=6.41, r=6.34, z=6.44 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.86293, 79.77963), pm (-5.6, 1.5), parallax 2.398 with NanoMaggies: g=9.87, r=9.76, z=9.84 Reset brightness to NanoMaggies: g=9.87, r=9.76, z=9.84 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.59763, 79.78396), pm (1.4, 1.3), parallax 0.539 with NanoMaggies: g=12.7, r=11.8, z=11.4 Reset brightness to NanoMaggies: g=12.7, r=11.8, z=11.4 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.63666, 79.78732), pm (-15.3, 25.0), parallax 2.196 with NanoMaggies: g=13.7, r=13.1, z=12.9 Reset brightness to NanoMaggies: g=13.7, r=13.1, z=12.9 Blob 2 finished, total: Wall: 0.35 s, CPU: 0.35 s, VmPeak: 804 MB, VmSize: 713 MB, VmRSS: 138 MB, VmData: 178 MB, maxrss: 0.141108 MB Blob 4 finished, total: Wall: 0.49 s, CPU: 0.48 s, VmPeak: 808 MB, VmSize: 713 MB, VmRSS: 138 MB, VmData: 178 MB, maxrss: 0.141276 MB Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.47919, 79.77747), pm (-4.3, -1.9), parallax 0.552 with NanoMaggies: g=17, r=16.4, z=16.6 Reset brightness to NanoMaggies: g=17, r=16.5, z=16.2 Blob 3 finished, total: Wall: 0.95 s, CPU: 0.95 s, VmPeak: 806 MB, VmSize: 713 MB, VmRSS: 138 MB, VmData: 178 MB, maxrss: 0.141424 MB Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.60208, 79.76202), pm (36.5, 67.9), parallax 9.184 with NanoMaggies: g=11.4, r=10.5, z=(flux -754) Reset brightness to NanoMaggies: g=12.1, r=11.3, z=10.9 Known galaxy. Initial shape: EllipseWithPriors(0.25): log r_e=1.15446, ee1=0.0302271, ee2=0.0139867 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.60086, 79.77134), pm (3.1, -15.7), parallax 2.348 with NanoMaggies: g=18.4, r=17.5, z=17.9 Reset brightness to NanoMaggies: g=22, r=20.6, z=19.8 Measuring 5 iterative sources Blob 1 finished, total: Wall: 26.31 s, CPU: 26.31 s, VmPeak: 805 MB, VmSize: 723 MB, VmRSS: 156 MB, VmData: 189 MB, maxrss: 0.171068 MB Old catalog: 28 New catalog: 25 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/blobs-custom-280537p79771.fits.gz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/all-models-custom-280537p79771.fits Stage fitblobs : Wall: 26.82 s, CPU: 0.45 s, VmPeak: 983 MB, VmSize: 968 MB, VmRSS: 209 MB, VmData: 236 MB, maxrss: 0.566544 MB Stage fitblobs finished: 2020-06-07T17:23:56.907473 Resources for stage fitblobs : 28.184 s worker CPU, 28.231 s worker Wall, Wall: 26.825 s, Cores in use: 1.07, Total efficiency (on 4 cores): 26.7 % Stage fitblobs finished Running stage coadds Running stage coadds at 2020-06-07T17:23:56.909433 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-ccds.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-nexp-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-depth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-galdepth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-psfsize-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-chi2-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-blobmodel-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-nexp-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-depth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-galdepth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-psfsize-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-chi2-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-blobmodel-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-nexp-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-depth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-galdepth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-psfsize-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-chi2-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-blobmodel-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-depth.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/tmp-legacysurvey-custom-280537p79771-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/tmp-legacysurvey-custom-280537p79771-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/tmp-legacysurvey-custom-280537p79771-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/tmp-legacysurvey-custom-280537p79771-resid.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-resid.jpg Stage coadds : Wall: 1.39 s, CPU: 0.66 s, VmPeak: 983 MB, VmSize: 968 MB, VmRSS: 209 MB, VmData: 236 MB, maxrss: 0.566544 MB Stage coadds finished: 2020-06-07T17:23:58.295873 Resources for stage coadds : 2.054 s worker CPU, 2.142 s worker Wall, Wall: 1.387 s, Cores in use: 1.96, Total efficiency (on 4 cores): 48.9 % Stage coadds finished Running stage wise_forced Running stage wise_forced at 2020-06-07T17:23:58.297506 Cut to 2 unWISE tiles Reading WISE tile 2762p802 band 1 Reading WISE tile 2762p802 band 4 Reading WISE tile 2762p802 band 2 Reading WISE tile 2762p802 band 3 Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w4-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w4-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w4-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits Reading WISE tile 2846p802 band 4 Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w3-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w3-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w3-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits Reading WISE tile 2846p802 band 3 Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w1-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w1-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w1-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w1-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits Reading WISE tile 2846p802 band 1 Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w2-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w2-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w2-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w2-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits Reading WISE tile 2846p802 band 2 Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w4-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w4-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w4-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.11 s, CPU: 0.11 s, VmPeak: 805 MB, VmSize: 801 MB, VmRSS: 158 MB, VmData: 183 MB, maxrss: 0.201204 MB Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w3-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w3-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w3-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.08 s, CPU: 0.08 s, VmPeak: 808 MB, VmSize: 792 MB, VmRSS: 144 MB, VmData: 174 MB, maxrss: 0.208844 MB Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w2-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w2-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w2-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w2-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.06 s, CPU: 0.06 s, VmPeak: 804 MB, VmSize: 792 MB, VmRSS: 144 MB, VmData: 174 MB, maxrss: 0.208648 MB Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w1-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w1-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w1-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w1-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.06 s, CPU: 0.06 s, VmPeak: 806 MB, VmSize: 792 MB, VmRSS: 144 MB, VmData: 173 MB, maxrss: 0.208948 MB Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/tmp-legacysurvey-custom-280537p79771-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/tmp-legacysurvey-custom-280537p79771-wisemodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-wisemodel.jpg Setting WISE apphot Setting WISE apphot Setting WISE apphot Setting WISE apphot Stage wise_forced : Wall: 3.80 s, CPU: 0.41 s, VmPeak: 983 MB, VmSize: 968 MB, VmRSS: 210 MB, VmData: 236 MB, maxrss: 0.566544 MB Stage wise_forced finished: 2020-06-07T17:24:02.098247 Resources for stage wise_forced : 8.515 s worker CPU, 11.339 s worker Wall, Wall: 3.802 s, Cores in use: 2.35, Total efficiency (on 4 cores): 58.7 % Stage wise_forced finished Running stage writecat Running stage writecat at 2020-06-07T17:24:02.100158 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-maskbits.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/tractor-i/cus/tractor-custom-280537p79771.fits Reading SFD maps... END Rejected (1), keyrecords following the END keyrecord were ignored. END Rejected (1), keyrecords following the END keyrecord were ignored. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/tractor/cus/tractor-custom-280537p79771.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/tractor/cus/brick-custom-280537p79771.sha256sum Stage writecat : Wall: 0.79 s, CPU: 0.44 s, VmPeak: 1100 MB, VmSize: 972 MB, VmRSS: 212 MB, VmData: 236 MB, maxrss: 0.566544 MB Stage writecat finished: 2020-06-07T17:24:02.886227 Resources for stage writecat : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 0.787 s, Cores in use: 0.56, Total efficiency (on 4 cores): 13.9 % Stage writecat finished All done: 44.361 s worker CPU, 59.498 s worker Wall, Wall: 46.540 s, Cores in use: 1.12, Total efficiency (on 4 cores): 28.1 % Finished galaxy 3C390.3 in 0.813 minutes. Started working on galaxy 3C390.3 at Fri Aug 7 09:32:37 2020 Bands touching this brick, g r z python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 280.5374800648159 79.77141875948853 --width 275 --height 275 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 runbrick.py starting at 2020-08-07T09:32:38.760298 legacypipe git version: DR9.5.11-watershed-3-gd509eaaf Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '280.5374800648159', '79.77141875948853', '--width', '275', '--height', '275', '--pixscale', '0.262', '--threads', '16', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 280.5374800648159 79.77141875948853 --width 275 --height 275 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 Parsed RA,Dec 280.5374800648159 79.77141875948853 Starting process 19653 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059228 MB Starting process 19654 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059232 MB Starting process 19655 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059232 MB Starting process 19656 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059232 MB Starting process 19657 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 19658 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059292 MB Starting process 19659 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB Starting process 19660 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 19661 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 19662 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB Starting process 19663 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB Starting process 19664 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 19665 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 19666 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 19667 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 19668 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage fit_on_coadds Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-08-07T09:32:39.231701 Keeping 12 CCDs: k4m_171221_030001_ooi_zd_ls9-CCD3 z expnum 152186 exptime 126.103 propid 2016A-0453 seeing 1.20 MJD 58108.125 object MzLS_730082_z mosaic/CP/V4.3/CP20171220/k4m_171221_030001_ooi_zd_ls9.fits.fz k4m_171008_060834_ooi_zd_ls9-CCD3 z expnum 142849 exptime 200.103 propid 2016A-0453 seeing 1.31 MJD 58034.256 object MzLS_730542_z mosaic/CP/V4.3/CP20171007/k4m_171008_060834_ooi_zd_ls9.fits.fz k4m_171112_035616_ooi_zd_ls9-CCD4 z expnum 146074 exptime 238.103 propid 2016A-0453 seeing 1.20 MJD 58069.164 object MzLS_730314_z mosaic/CP/V4.3/CP20171111/k4m_171112_035616_ooi_zd_ls9.fits.fz k4m_171125_052741_ooi_zd_ls9-CCD4 z expnum 148907 exptime 130.104 propid 2016A-0453 seeing 0.77 MJD 58082.228 object MzLS_730766_z mosaic/CP/V4.3/CP20171124/k4m_171125_052741_ooi_zd_ls9.fits.fz ksb_180722_100116_ooi_g_ls9-CCD3 g expnum 83210121 exptime 187.334 propid NOTBASS seeing 1.55 MJD 58321.418 object 52423 90prime/CP/V2.3/CP20180721/ksb_180722_100116_ooi_g_ls9.fits.fz ksb_180703_103559_ooi_r_ls9-CCD3 r expnum 83020151 exptime 250.0 propid NOTBASS seeing 1.34 MJD 58302.442 object 52423 90prime/CP/V2.3/CP20180702/ksb_180703_103559_ooi_r_ls9.fits.fz ksb_180707_082623_ooi_r_ls9-CCD2 r expnum 83060136 exptime 141.273 propid NOTBASS seeing 1.43 MJD 58306.352 object 52412 90prime/CP/V2.3/CP20180706/ksb_180707_082623_ooi_r_ls9.fits.fz ksb_180618_081820_ooi_g_ls9-CCD2 g expnum 82870117 exptime 217.51 propid NOTBASS seeing 2.07 MJD 58287.346 object 52412 90prime/CP/V2.3/CP20180617/ksb_180618_081820_ooi_g_ls9.fits.fz ksb_160712_104905_ooi_g_ls9-CCD3 g expnum 75810159 exptime 234.7 propid BASS seeing 1.83 MJD 57581.451 object 51902 90prime/CP/V2.3/CP20160711/ksb_160712_104905_ooi_g_ls9.fits.fz ksb_180703_081020_ooi_r_ls9-CCD3 r expnum 83020121 exptime 250.0 propid NOTBASS seeing 1.51 MJD 58302.341 object 51902 90prime/CP/V2.3/CP20180702/ksb_180703_081020_ooi_r_ls9.fits.fz ksb_180608_104824_ooi_r_ls9-CCD4 r expnum 82770144 exptime 250.0 propid NOTBASS seeing 1.56 MJD 58277.450 object 52421 90prime/CP/V2.3/CP20180607/ksb_180608_104824_ooi_r_ls9.fits.fz ksb_180722_102838_ooi_g_ls9-CCD4 g expnum 83210128 exptime 187.334 propid NOTBASS seeing 1.37 MJD 58321.437 object 52421 90prime/CP/V2.3/CP20180721/ksb_180722_102838_ooi_g_ls9.fits.fz mosaic get_fwhm: self.fwhm = 4.61 Found 4 relevant rows in amp-corrections file. Found overlap: image x 6 292 and amp range 0 2005 and image y 1348 1628 and amp range 0 2039 Range in image: x 0 287 , y 0 281 (with image size 287 x 281) dzp 0.0014252509 -> scaling image by 1.0013135665633692 mosaic get_fwhm: self.fwhm = 2.97 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3925 4088 and amp range 2048 4095 and image y 2710 2996 and amp range 2022 4060 Range in image: x 0 164 , y 0 287 (with image size 164 x 287) dzp -0.019603321 -> scaling image by 0.9821066946375949 mosaic get_fwhm: self.fwhm = 5.05 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2723 3009 and amp range 2006 4053 and image y 3041 3332 and amp range 2040 4078 Range in image: x 0 287 , y 0 292 (with image size 287 x 292) dzp 0.0056365696 -> scaling image by 1.005204971477435 Faked up EXPNUM 83060136 Faked up EXPNUM 83060136 Faked up EXPNUM 83060136 90prime get_fwhm: self.fwhm = 3.15 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3568 3737 and amp range 2016 4031 and image y 1114 1283 and amp range 0 2047 Range in image: x 0 170 , y 0 170 (with image size 170 x 170) dzp 0.014483032 -> scaling image by 1.0134287315188357 Faked up EXPNUM 83210121 Faked up EXPNUM 83210121 Faked up EXPNUM 83210121 90prime get_fwhm: self.fwhm = 3.41 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3891 4016 and amp range 2016 4031 and image y 663 823 and amp range 0 2047 Range in image: x 0 126 , y 0 161 (with image size 126 x 161) dzp -0.0012233759 -> scaling image by 0.9988738637196966 Faked up EXPNUM 82870117 Faked up EXPNUM 82870117 Faked up EXPNUM 82870117 90prime get_fwhm: self.fwhm = 4.57 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3543 3711 and amp range 2016 4031 and image y 1123 1292 and amp range 0 2047 Range in image: x 0 169 , y 0 170 (with image size 169 x 170) dzp 0.009189803 -> scaling image by 1.0085000434534261 Faked up EXPNUM 82770144 Faked up EXPNUM 82770144 Faked up EXPNUM 82770144 90prime get_fwhm: self.fwhm = 3.43 WARNING: image median 0.017431121 is more than 1 sigma away from zero! Found 4 relevant rows in amp-corrections file. Found overlap: image x 1697 1858 and amp range 0 2015 and image y 2563 2725 and amp range 2048 4095 Range in image: x 0 162 , y 0 163 (with image size 162 x 163) dzp 0.04491909 -> scaling image by 1.0422397564357018 Faked up EXPNUM 83210128 Faked up EXPNUM 83210128 Faked up EXPNUM 83210128 90prime get_fwhm: self.fwhm = 3.03 Found 4 relevant rows in amp-corrections file. Found overlap: image x 1689 1850 and amp range 0 2015 and image y 2560 2722 and amp range 2048 4095 Range in image: x 0 162 , y 0 163 (with image size 162 x 163) dzp 0.027550658 -> scaling image by 1.0256997821704772 Faked up EXPNUM 83020151 Faked up EXPNUM 83020151 Faked up EXPNUM 83020151 90prime get_fwhm: self.fwhm = 2.95 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3906 4016 and amp range 2016 4031 and image y 670 830 and amp range 0 2047 Range in image: x 0 111 , y 0 161 (with image size 111 x 161) dzp 0.0057031885 -> scaling image by 1.00526665101287 mosaic get_fwhm: self.fwhm = 4.63 Found 4 relevant rows in amp-corrections file. Found overlap: image x 1138 1426 and amp range 0 2047 and image y 2884 3172 and amp range 2022 4060 Range in image: x 0 289 , y 0 289 (with image size 289 x 289) dzp -0.016192675 -> scaling image by 0.9851966584456431 Stage tims : Wall: 219.01 s, CPU: 0.99 s, VmPeak: 951 MB, VmSize: 951 MB, VmRSS: 174 MB, VmData: 208 MB, maxrss: 0.178484 MB Stage tims finished: 2020-08-07T09:36:18.240791 Faked up EXPNUM 75810159 Faked up EXPNUM 75810159 Faked up EXPNUM 75810159 Faked up EXPNUM 83020121 Faked up EXPNUM 83020121 Faked up EXPNUM 83020121 Resources for stage tims : 5.774 s worker CPU, 664.499 s worker Wall, Wall: 219.011 s, Cores in use: 0.03, Total efficiency (on 16 cores): 0.2 % Stage tims finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-tims.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-tims.p Running stage refs Running stage refs at 2020-08-07T09:36:18.361185 Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00137.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00106.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00105.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00078.fits Matched 7 large galaxies to Gaia stars. Matched 0 large galaxies to Tycho-2 stars. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/reference-custom-280537p79771.fits Stage refs : Wall: 3.67 s, CPU: 3.00 s, VmPeak: 989 MB, VmSize: 984 MB, VmRSS: 208 MB, VmData: 241 MB, maxrss: 0.212748 MB Stage refs finished: 2020-08-07T09:36:22.031832 Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 3.672 s, Cores in use: 0.82, Total efficiency (on 16 cores): 5.1 % Stage refs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-refs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-refs.p Running stage outliers Running stage outliers at 2020-08-07T09:36:22.099995 Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/outlier-mask-custom-280537p79771.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/outliers-pre-custom-280537p79771.jpg Image ksb_180722_102838_ooi_g_ls9-CCD4 g : masking 320 positive outlier pixels and 0 negative outlier pixels Image ksb_180707_082623_ooi_r_ls9-CCD2 r : masking 54 positive outlier pixels and 0 negative outlier pixels Image k4m_171125_052741_ooi_zd_ls9-CCD4 z : masking 631 positive outlier pixels and 0 negative outlier pixels Image k4m_171221_030001_ooi_zd_ls9-CCD3 z : masking 71 positive outlier pixels and 0 negative outlier pixels Image k4m_171008_060834_ooi_zd_ls9-CCD3 z : masking 997 positive outlier pixels and 0 negative outlier pixels Image k4m_171112_035616_ooi_zd_ls9-CCD4 z : masking 696 positive outlier pixels and 0 negative outlier pixels Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/outlier-mask-custom-280537p79771.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/outliers-post-custom-280537p79771.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/outliers-masked-pos-custom-280537p79771.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/outliers-masked-neg-custom-280537p79771.jpg Stage outliers : Wall: 1.03 s, CPU: 0.65 s, VmPeak: 993 MB, VmSize: 993 MB, VmRSS: 223 MB, VmData: 252 MB, maxrss: 0.22818 MB Stage outliers finished: 2020-08-07T09:36:23.125411 Resources for stage outliers : 1.179 s worker CPU, 1.290 s worker Wall, Wall: 1.027 s, Cores in use: 1.79, Total efficiency (on 16 cores): 11.2 % Stage outliers finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-outliers.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-outliers.p Running stage halos Running stage halos at 2020-08-07T09:36:23.232875 Stage halos : Wall: 0.46 s, CPU: 0.45 s, VmPeak: 993 MB, VmSize: 993 MB, VmRSS: 223 MB, VmData: 253 MB, maxrss: 0.22876 MB Stage halos finished: 2020-08-07T09:36:23.693253 Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.462 s, Cores in use: 0.98, Total efficiency (on 16 cores): 6.1 % Stage halos finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-halos.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-halos.p Running stage fit_on_coadds Warning: Stellar halo subtraction is only implemented for DECam Running stage fit_on_coadds at 2020-08-07T09:36:23.802815 PSF sigmas in coadd pixels: ['2.51', '3.37', '2.23'] Using average PSF sigma 2.702722108297493 Gaussian PSF norm 0.10437432354137814 vs pixelized 0.08759935 average tim pixel scale / coadd scale: 1.730637261395179 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-copsf-g.fits.fz PSF sigmas in coadd pixels: ['2.17', '2.32', '2.53'] Using average PSF sigma 2.3389166742461818 Gaussian PSF norm 0.12060916700454734 vs pixelized 0.09578845 average tim pixel scale / coadd scale: 1.730248225383631 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-copsf-r.fits.fz PSF sigmas in coadd pixels: ['1.94', '2.13', '1.95', '1.25'] Using average PSF sigma 1.820011506738735 Gaussian PSF norm 0.15499615839207617 vs pixelized 0.13379909 average tim pixel scale / coadd scale: 0.9912280830152986 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-copsf-z.fits.fz Stage fit_on_coadds : Wall: 0.27 s, CPU: 0.17 s, VmPeak: 993 MB, VmSize: 993 MB, VmRSS: 226 MB, VmData: 254 MB, maxrss: 0.231284 MB Stage fit_on_coadds finished: 2020-08-07T09:36:24.068137 Resources for stage fit_on_coadds : 0.354 s worker CPU, 0.354 s worker Wall, Wall: 0.267 s, Cores in use: 1.96, Total efficiency (on 16 cores): 12.2 % Stage fit_on_coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-fit_on_coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-fit_on_coadds.p Running stage srcs Running stage srcs at 2020-08-07T09:36:24.080037 SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])] Found 5959 potential peaks Of 5959 potential peaks: 5940 in veto map, 4 cut by saddle test, 0 cut by aper test, 15 kept SED z : 15 new peaks Found 3032 potential peaks Of 3032 potential peaks: 3021 in veto map, 10 cut by saddle test, 0 cut by aper test, 1 kept SED r : 1 new peaks Found 132 potential peaks Of 132 potential peaks: 126 in veto map, 4 cut by saddle test, 0 cut by aper test, 2 kept SED g : 2 new peaks Found 197 potential peaks Of 197 potential peaks: 188 in veto map, 8 cut by saddle test, 0 cut by aper test, 1 kept SED Flat : 1 new peaks Found 252 potential peaks Of 252 potential peaks: 243 in veto map, 9 cut by saddle test, 0 cut by aper test, 0 kept SED Red : 0 new peaks Adding 19 new sources Median coadd sky for g : 0.0006222781 Median coadd sky for r : 0.0016024343 Median coadd sky for z : 0.0002848954 Stage srcs : Wall: 0.59 s, CPU: 0.54 s, VmPeak: 993 MB, VmSize: 993 MB, VmRSS: 226 MB, VmData: 254 MB, maxrss: 0.231752 MB Stage srcs finished: 2020-08-07T09:36:24.665068 Resources for stage srcs : 0.105 s worker CPU, 0.105 s worker Wall, Wall: 0.586 s, Cores in use: 1.11, Total efficiency (on 16 cores): 6.9 % Stage srcs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-srcs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-srcs.p Running stage fitblobs Running stage fitblobs at 2020-08-07T09:36:24.698401 Blob 1 of 2, id: 0, sources: 29, size: 275x275, npix 65768, brick X: 0,275, Y: 0,275, one pixel: 0 0 Blob 2 of 2, id: 1, sources: 1, size: 23x25, npix 352, brick X: 242,265, Y: 62,87, one pixel: 252 62 Blob 2 finished, total: Wall: 0.33 s, CPU: 0.32 s, VmPeak: 805 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.147012 MB Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (281.40926, 79.94293), pm (8.9, 64.4), parallax 11.292 with NanoMaggies: g=6.41, r=6.34, z=6.44 Reset brightness to NanoMaggies: g=6.41, r=6.34, z=6.44 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.86293, 79.77963), pm (-5.6, 1.5), parallax 2.398 with NanoMaggies: g=9.87, r=9.76, z=9.84 Reset brightness to NanoMaggies: g=9.87, r=9.76, z=9.84 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.60208, 79.76202), pm (36.5, 67.9), parallax 9.184 with NanoMaggies: g=12, r=9.49, z=(flux -1.33e+03) Reset brightness to NanoMaggies: g=12.1, r=11.3, z=10.9 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.59763, 79.78396), pm (1.4, 1.3), parallax 0.539 with NanoMaggies: g=12.7, r=11.8, z=11.4 Reset brightness to NanoMaggies: g=12.7, r=11.8, z=11.4 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.63666, 79.78732), pm (-15.3, 25.0), parallax 2.196 with NanoMaggies: g=13.7, r=13.1, z=12.9 Reset brightness to NanoMaggies: g=13.7, r=13.1, z=12.9 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.50401, 79.79325), pm (0.9, 4.9), parallax 0.904 with NanoMaggies: g=14.6, r=14, z=13.9 Reset brightness to NanoMaggies: g=14.6, r=14, z=13.9 Known galaxy. Initial shape: EllipseWithPriors(0.25): log r_e=1.13893, ee1=-0.0511548, ee2=0.00308452 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.46623, 79.78034), pm (-1.4, -2.9), parallax 0.483 with NanoMaggies: g=16.8, r=16.3, z=16.1 Reset brightness to NanoMaggies: g=16.8, r=16.3, z=16.1 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.60086, 79.77134), pm (3.1, -15.7), parallax 2.348 with NanoMaggies: g=18.1, r=17, z=17.7 Reset brightness to NanoMaggies: g=22, r=20.6, z=19.8 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.47919, 79.77747), pm (-4.3, -1.9), parallax 0.552 with NanoMaggies: g=20.3, r=18.8, z=(flux -572) Reset brightness to NanoMaggies: g=17, r=16.5, z=16.2 Measuring 11 iterative sources Blob 1 finished, total: Wall: 31.71 s, CPU: 31.70 s, VmPeak: 740 MB, VmSize: 736 MB, VmRSS: 170 MB, VmData: 204 MB, maxrss: 0.178076 MB Old catalog: 31 New catalog: 34 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/blobs-custom-280537p79771.fits.gz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/all-models-custom-280537p79771.fits Stage fitblobs : Wall: 37.56 s, CPU: 0.49 s, VmPeak: 993 MB, VmSize: 993 MB, VmRSS: 228 MB, VmData: 255 MB, maxrss: 0.23328 MB Stage fitblobs finished: 2020-08-07T09:37:02.256240 Resources for stage fitblobs : 32.068 s worker CPU, 32.100 s worker Wall, Wall: 37.559 s, Cores in use: 0.87, Total efficiency (on 16 cores): 5.4 % Stage fitblobs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-fitblobs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-fitblobs.p Running stage coadds Running stage coadds at 2020-08-07T09:37:04.118649 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-ccds.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-nexp-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-depth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-galdepth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-psfsize-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-chi2-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-blobmodel-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-nexp-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-depth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-galdepth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-psfsize-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-chi2-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-blobmodel-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-nexp-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-depth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-galdepth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-psfsize-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-chi2-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-blobmodel-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-depth.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/tmp-legacysurvey-custom-280537p79771-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/tmp-legacysurvey-custom-280537p79771-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/tmp-legacysurvey-custom-280537p79771-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/tmp-legacysurvey-custom-280537p79771-resid.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-resid.jpg Stage coadds : Wall: 7.01 s, CPU: 0.60 s, VmPeak: 993 MB, VmSize: 993 MB, VmRSS: 228 MB, VmData: 255 MB, maxrss: 0.233824 MB Stage coadds finished: 2020-08-07T09:37:11.125084 Resources for stage coadds : 9.055 s worker CPU, 9.360 s worker Wall, Wall: 7.008 s, Cores in use: 1.38, Total efficiency (on 16 cores): 8.6 % Stage coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-coadds.p Running stage wise_forced Running stage wise_forced at 2020-08-07T09:37:11.151934 Cut to 2 unWISE tiles Reading WISE tile 2762p802 band 1 Reading WISE tile 2762p802 band 2 Reading WISE tile 2762p802 band 3 Reading WISE tile 2762p802 band 4 Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w4-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w4-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w4-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits Reading WISE tile 2846p802 band 4 Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w2-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w2-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w2-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w2-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits Reading WISE tile 2846p802 band 2 Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w1-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w1-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w1-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w1-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits Reading WISE tile 2846p802 band 1 Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w4-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w4-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w4-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.10 s, CPU: 0.10 s, VmPeak: 809 MB, VmSize: 801 MB, VmRSS: 149 MB, VmData: 179 MB, maxrss: 0.183332 MB Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w2-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w2-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w2-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w2-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.07 s, CPU: 0.07 s, VmPeak: 808 MB, VmSize: 793 MB, VmRSS: 141 MB, VmData: 172 MB, maxrss: 0.199788 MB Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w1-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w1-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w1-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w1-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.07 s, CPU: 0.07 s, VmPeak: 811 MB, VmSize: 796 MB, VmRSS: 141 MB, VmData: 174 MB, maxrss: 0.20116 MB Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w3-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w3-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w3-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits Reading WISE tile 2846p802 band 3 Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w3-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w3-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w3-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.08 s, CPU: 0.08 s, VmPeak: 811 MB, VmSize: 793 MB, VmRSS: 138 MB, VmData: 171 MB, maxrss: 0.199028 MB Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/tmp-legacysurvey-custom-280537p79771-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/tmp-legacysurvey-custom-280537p79771-wisemodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-wisemodel.jpg Setting WISE apphot Setting WISE apphot Setting WISE apphot Setting WISE apphot Stage wise_forced : Wall: 64.06 s, CPU: 0.55 s, VmPeak: 993 MB, VmSize: 993 MB, VmRSS: 229 MB, VmData: 255 MB, maxrss: 0.234592 MB Stage wise_forced finished: 2020-08-07T09:38:15.211496 Resources for stage wise_forced : 5.665 s worker CPU, 90.322 s worker Wall, Wall: 64.061 s, Cores in use: 0.10, Total efficiency (on 16 cores): 0.6 % Stage wise_forced finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-wise_forced.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-wise_forced.p Running stage writecat Running stage writecat at 2020-08-07T09:38:15.261544 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-maskbits.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/tractor-i/cus/tractor-custom-280537p79771.fits Reading SFD maps... END Rejected (1), keyrecords following the END keyrecord were ignored. END Rejected (1), keyrecords following the END keyrecord were ignored. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/tractor/cus/tractor-custom-280537p79771.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/tractor/cus/brick-custom-280537p79771.sha256sum Stage writecat : Wall: 0.85 s, CPU: 0.60 s, VmPeak: 1126 MB, VmSize: 998 MB, VmRSS: 231 MB, VmData: 255 MB, maxrss: 0.367604 MB Stage writecat finished: 2020-08-07T09:38:16.108082 Resources for stage writecat : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 0.848 s, Cores in use: 0.71, Total efficiency (on 16 cores): 4.5 % Stage writecat finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-writecat.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-writecat.p All done: 54.203 s worker CPU, 798.033 s worker Wall, Wall: 336.917 s, Cores in use: 0.19, Total efficiency (on 16 cores): 1.2 % Finished galaxy 3C390.3 in 5.657 minutes. Started working on galaxy 3C390.3 at Thu Aug 13 13:31:13 2020 Bands touching this brick, g r z python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 280.5374800648159 79.77141875948853 --width 275 --height 275 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 runbrick.py starting at 2020-08-13T13:31:14.524372 legacypipe git version: DR9.5.11-watershed-9-g82106209 Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '280.5374800648159', '79.77141875948853', '--width', '275', '--height', '275', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-%%(stage)s.p', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 280.5374800648159 79.77141875948853 --width 275 --height 275 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 Parsed RA,Dec 280.5374800648159 79.77141875948853 Starting process 9232 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059228 MB Starting process 9233 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059228 MB Starting process 9234 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059232 MB Starting process 9235 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059232 MB Starting process 9236 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059528 MB Starting process 9237 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05954 MB Starting process 9238 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05954 MB Starting process 9239 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 9240 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB Starting process 9241 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB Starting process 9242 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB Starting process 9243 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB Starting process 9244 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB Starting process 9245 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 9246 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 9247 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB Starting process 9248 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB Starting process 9249 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB Starting process 9250 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB Starting process 9251 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB Starting process 9252 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB Starting process 9253 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB Starting process 9254 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059276 MB Starting process 9255 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05928 MB Starting process 9256 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 9257 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB Starting process 9258 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB Starting process 9259 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059292 MB Starting process 9260 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB Starting process 9261 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB Starting process 9262 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB Starting process 9264 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage fit_on_coadds Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-08-13T13:31:16.575586 Keeping 12 CCDs: k4m_171221_030001_ooi_zd_ls9-CCD3 z expnum 152186 exptime 126.103 propid 2016A-0453 seeing 1.20 MJD 58108.125 object MzLS_730082_z mosaic/CP/V4.3/CP20171220/k4m_171221_030001_ooi_zd_ls9.fits.fz k4m_171008_060834_ooi_zd_ls9-CCD3 z expnum 142849 exptime 200.103 propid 2016A-0453 seeing 1.31 MJD 58034.256 object MzLS_730542_z mosaic/CP/V4.3/CP20171007/k4m_171008_060834_ooi_zd_ls9.fits.fz k4m_171112_035616_ooi_zd_ls9-CCD4 z expnum 146074 exptime 238.103 propid 2016A-0453 seeing 1.20 MJD 58069.164 object MzLS_730314_z mosaic/CP/V4.3/CP20171111/k4m_171112_035616_ooi_zd_ls9.fits.fz k4m_171125_052741_ooi_zd_ls9-CCD4 z expnum 148907 exptime 130.104 propid 2016A-0453 seeing 0.77 MJD 58082.228 object MzLS_730766_z mosaic/CP/V4.3/CP20171124/k4m_171125_052741_ooi_zd_ls9.fits.fz ksb_180722_100116_ooi_g_ls9-CCD3 g expnum 83210121 exptime 187.334 propid NOTBASS seeing 1.55 MJD 58321.418 object 52423 90prime/CP/V2.3/CP20180721/ksb_180722_100116_ooi_g_ls9.fits.fz ksb_180703_103559_ooi_r_ls9-CCD3 r expnum 83020151 exptime 250.0 propid NOTBASS seeing 1.34 MJD 58302.442 object 52423 90prime/CP/V2.3/CP20180702/ksb_180703_103559_ooi_r_ls9.fits.fz ksb_180707_082623_ooi_r_ls9-CCD2 r expnum 83060136 exptime 141.273 propid NOTBASS seeing 1.43 MJD 58306.352 object 52412 90prime/CP/V2.3/CP20180706/ksb_180707_082623_ooi_r_ls9.fits.fz ksb_180618_081820_ooi_g_ls9-CCD2 g expnum 82870117 exptime 217.51 propid NOTBASS seeing 2.07 MJD 58287.346 object 52412 90prime/CP/V2.3/CP20180617/ksb_180618_081820_ooi_g_ls9.fits.fz ksb_160712_104905_ooi_g_ls9-CCD3 g expnum 75810159 exptime 234.7 propid BASS seeing 1.83 MJD 57581.451 object 51902 90prime/CP/V2.3/CP20160711/ksb_160712_104905_ooi_g_ls9.fits.fz ksb_180703_081020_ooi_r_ls9-CCD3 r expnum 83020121 exptime 250.0 propid NOTBASS seeing 1.51 MJD 58302.341 object 51902 90prime/CP/V2.3/CP20180702/ksb_180703_081020_ooi_r_ls9.fits.fz ksb_180608_104824_ooi_r_ls9-CCD4 r expnum 82770144 exptime 250.0 propid NOTBASS seeing 1.56 MJD 58277.450 object 52421 90prime/CP/V2.3/CP20180607/ksb_180608_104824_ooi_r_ls9.fits.fz ksb_180722_102838_ooi_g_ls9-CCD4 g expnum 83210128 exptime 187.334 propid NOTBASS seeing 1.37 MJD 58321.437 object 52421 90prime/CP/V2.3/CP20180721/ksb_180722_102838_ooi_g_ls9.fits.fz Faked up EXPNUM 83210121 Faked up EXPNUM 83210121 Faked up EXPNUM 83210121 90prime get_fwhm: self.fwhm = 3.41 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3891 4016 and amp range 2016 4031 and image y 663 823 and amp range 0 2047 Range in image: x 0 126 , y 0 161 (with image size 126 x 161) dzp -0.0012233759 -> scaling image by 0.9988738637196966 mosaic get_fwhm: self.fwhm = 5.05 Found 4 relevant rows in amp-corrections file. Found overlap: image x 2723 3009 and amp range 2006 4053 and image y 3041 3332 and amp range 2040 4078 Range in image: x 0 287 , y 0 292 (with image size 287 x 292) dzp 0.0056365696 -> scaling image by 1.005204971477435 mosaic get_fwhm: self.fwhm = 4.63 Found 4 relevant rows in amp-corrections file. Found overlap: image x 1138 1426 and amp range 0 2047 and image y 2884 3172 and amp range 2022 4060 Range in image: x 0 289 , y 0 289 (with image size 289 x 289) dzp -0.016192675 -> scaling image by 0.9851966584456431 mosaic get_fwhm: self.fwhm = 4.61 Found 4 relevant rows in amp-corrections file. Found overlap: image x 6 292 and amp range 0 2005 and image y 1348 1628 and amp range 0 2039 Range in image: x 0 287 , y 0 281 (with image size 287 x 281) dzp 0.0014252509 -> scaling image by 1.0013135665633692 mosaic get_fwhm: self.fwhm = 2.97 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3925 4088 and amp range 2048 4095 and image y 2710 2996 and amp range 2022 4060 Range in image: x 0 164 , y 0 287 (with image size 164 x 287) dzp -0.019603321 -> scaling image by 0.9821066946375949 Faked up EXPNUM 83210128 Faked up EXPNUM 83210128 Faked up EXPNUM 83210128 90prime get_fwhm: self.fwhm = 3.03 Found 4 relevant rows in amp-corrections file. Found overlap: image x 1689 1850 and amp range 0 2015 and image y 2560 2722 and amp range 2048 4095 Range in image: x 0 162 , y 0 163 (with image size 162 x 163) dzp 0.027550658 -> scaling image by 1.0256997821704772 Faked up EXPNUM 82870117 Faked up EXPNUM 82870117 Faked up EXPNUM 82870117 90prime get_fwhm: self.fwhm = 4.57 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3543 3711 and amp range 2016 4031 and image y 1123 1292 and amp range 0 2047 Range in image: x 0 169 , y 0 170 (with image size 169 x 170) dzp 0.009189803 -> scaling image by 1.0085000434534261 Faked up EXPNUM 82770144 Faked up EXPNUM 82770144 Faked up EXPNUM 82770144 90prime get_fwhm: self.fwhm = 3.43 WARNING: image median 0.017431121 is more than 1 sigma away from zero! Found 4 relevant rows in amp-corrections file. Found overlap: image x 1697 1858 and amp range 0 2015 and image y 2563 2725 and amp range 2048 4095 Range in image: x 0 162 , y 0 163 (with image size 162 x 163) dzp 0.04491909 -> scaling image by 1.0422397564357018 Faked up EXPNUM 83020151 Faked up EXPNUM 83020151 Faked up EXPNUM 83020151 90prime get_fwhm: self.fwhm = 2.95 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3906 4016 and amp range 2016 4031 and image y 670 830 and amp range 0 2047 Range in image: x 0 111 , y 0 161 (with image size 111 x 161) dzp 0.0057031885 -> scaling image by 1.00526665101287 Faked up EXPNUM 83060136 Faked up EXPNUM 83060136 Faked up EXPNUM 83060136 90prime get_fwhm: self.fwhm = 3.15 Found 4 relevant rows in amp-corrections file. Found overlap: image x 3568 3737 and amp range 2016 4031 and image y 1114 1283 and amp range 0 2047 Range in image: x 0 170 , y 0 170 (with image size 170 x 170) dzp 0.014483032 -> scaling image by 1.0134287315188357 Stage tims : Wall: 2.41 s, CPU: 0.88 s, VmPeak: 951 MB, VmSize: 951 MB, VmRSS: 174 MB, VmData: 207 MB, maxrss: 0.17788 MB Stage tims finished: 2020-08-13T13:31:18.981047 Faked up EXPNUM 83020121 Faked up EXPNUM 83020121 Faked up EXPNUM 83020121 Resources for stage tims : 5.666 s worker CPU, 12.527 s worker Wall, Wall: 2.408 s, Cores in use: 2.72, Total efficiency (on 32 cores): 8.5 % Stage tims finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-tims.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-tims.p Running stage refs Running stage refs at 2020-08-13T13:31:19.014797 Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00137.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00106.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00105.fits Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00078.fits Matched 7 large galaxies to Gaia stars. Matched 0 large galaxies to Tycho-2 stars. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/reference-custom-280537p79771.fits Stage refs : Wall: 3.25 s, CPU: 2.87 s, VmPeak: 989 MB, VmSize: 984 MB, VmRSS: 207 MB, VmData: 240 MB, maxrss: 0.212052 MB Stage refs finished: 2020-08-13T13:31:22.264105 Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 3.251 s, Cores in use: 0.88, Total efficiency (on 32 cores): 2.8 % Stage refs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-refs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-refs.p Running stage outliers Running stage outliers at 2020-08-13T13:31:22.289206 Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/outlier-mask-custom-280537p79771.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/outliers-pre-custom-280537p79771.jpg Image ksb_180722_102838_ooi_g_ls9-CCD4 g : masking 320 positive outlier pixels and 0 negative outlier pixels Image ksb_180707_082623_ooi_r_ls9-CCD2 r : masking 54 positive outlier pixels and 0 negative outlier pixels Image k4m_171125_052741_ooi_zd_ls9-CCD4 z : masking 631 positive outlier pixels and 0 negative outlier pixels Image k4m_171221_030001_ooi_zd_ls9-CCD3 z : masking 71 positive outlier pixels and 0 negative outlier pixels Image k4m_171112_035616_ooi_zd_ls9-CCD4 z : masking 696 positive outlier pixels and 0 negative outlier pixels Image k4m_171008_060834_ooi_zd_ls9-CCD3 z : masking 997 positive outlier pixels and 0 negative outlier pixels Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/outlier-mask-custom-280537p79771.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/outliers-post-custom-280537p79771.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/outliers-masked-pos-custom-280537p79771.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/outliers-masked-neg-custom-280537p79771.jpg Stage outliers : Wall: 2.65 s, CPU: 2.00 s, VmPeak: 993 MB, VmSize: 993 MB, VmRSS: 224 MB, VmData: 254 MB, maxrss: 0.229504 MB Stage outliers finished: 2020-08-13T13:31:24.937655 Resources for stage outliers : 1.245 s worker CPU, 1.503 s worker Wall, Wall: 2.651 s, Cores in use: 1.23, Total efficiency (on 32 cores): 3.8 % Stage outliers finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-outliers.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-outliers.p Running stage halos Faked up EXPNUM 75810159 Faked up EXPNUM 75810159 Faked up EXPNUM 75810159 Running stage halos at 2020-08-13T13:31:24.962921 Stage halos : Wall: 0.80 s, CPU: 0.79 s, VmPeak: 994 MB, VmSize: 994 MB, VmRSS: 224 MB, VmData: 254 MB, maxrss: 0.229828 MB Stage halos finished: 2020-08-13T13:31:25.761264 Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.800 s, Cores in use: 0.99, Total efficiency (on 32 cores): 3.1 % Stage halos finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-halos.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-halos.p Running stage fit_on_coadds Warning: Stellar halo subtraction is only implemented for DECam Running stage fit_on_coadds at 2020-08-13T13:31:25.788600 PSF sigmas in coadd pixels: ['2.51', '3.37', '2.23'] Using average PSF sigma 2.702722108297493 Gaussian PSF norm 0.10437432354137814 vs pixelized 0.08759935 average tim pixel scale / coadd scale: 1.730637261395179 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-copsf-g.fits.fz PSF sigmas in coadd pixels: ['2.17', '2.32', '2.53'] Using average PSF sigma 2.3389166742461818 Gaussian PSF norm 0.12060916700454734 vs pixelized 0.09578845 average tim pixel scale / coadd scale: 1.730248225383631 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-copsf-r.fits.fz PSF sigmas in coadd pixels: ['1.94', '2.13', '1.95', '1.25'] Using average PSF sigma 1.820011506738735 Gaussian PSF norm 0.15499615839207617 vs pixelized 0.13379909 average tim pixel scale / coadd scale: 0.9912280830152986 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-copsf-z.fits.fz Stage fit_on_coadds : Wall: 0.17 s, CPU: 0.17 s, VmPeak: 994 MB, VmSize: 994 MB, VmRSS: 226 MB, VmData: 254 MB, maxrss: 0.231492 MB Stage fit_on_coadds finished: 2020-08-13T13:31:25.963190 Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Warning: Stellar halo subtraction is only implemented for DECam Resources for stage fit_on_coadds : 0.290 s worker CPU, 0.290 s worker Wall, Wall: 0.177 s, Cores in use: 2.61, Total efficiency (on 32 cores): 8.1 % Stage fit_on_coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-fit_on_coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-fit_on_coadds.p Running stage srcs Running stage srcs at 2020-08-13T13:31:25.977808 SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])] Found 5959 potential peaks Of 5959 potential peaks: 5940 in veto map, 4 cut by saddle test, 0 cut by aper test, 15 kept SED z : 15 new peaks Found 3032 potential peaks Of 3032 potential peaks: 3021 in veto map, 10 cut by saddle test, 0 cut by aper test, 1 kept SED r : 1 new peaks Found 132 potential peaks Of 132 potential peaks: 126 in veto map, 4 cut by saddle test, 0 cut by aper test, 2 kept SED g : 2 new peaks Found 197 potential peaks Of 197 potential peaks: 188 in veto map, 8 cut by saddle test, 0 cut by aper test, 1 kept SED Flat : 1 new peaks Found 252 potential peaks Of 252 potential peaks: 243 in veto map, 9 cut by saddle test, 0 cut by aper test, 0 kept SED Red : 0 new peaks Adding 19 new sources Median coadd sky for g : 0.0006222781 Median coadd sky for r : 0.0016024343 Median coadd sky for z : 0.0002848955 Stage srcs : Wall: 1.65 s, CPU: 1.61 s, VmPeak: 994 MB, VmSize: 994 MB, VmRSS: 226 MB, VmData: 254 MB, maxrss: 0.231848 MB Stage srcs finished: 2020-08-13T13:31:27.630372 Warning: Stellar halo subtraction is only implemented for DECam Resources for stage srcs : 0.115 s worker CPU, 0.115 s worker Wall, Wall: 1.655 s, Cores in use: 1.04, Total efficiency (on 32 cores): 3.3 % Stage srcs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-srcs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-srcs.p Running stage fitblobs Running stage fitblobs at 2020-08-13T13:31:27.646189 Blob 1 of 2, id: 0, sources: 29, size: 275x275, npix 65768, brick X: 0,275, Y: 0,275, one pixel: 0 0 Blob 2 of 2, id: 1, sources: 1, size: 23x25, npix 352, brick X: 242,265, Y: 62,87, one pixel: 252 62 Blob 2 finished, total: Wall: 0.32 s, CPU: 0.32 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 92 MB, VmData: 139 MB, maxrss: 0.09466 MB Reference source: GaiaSource at GaiaPosition: RA, Dec = (281.40926, 79.94293), pm (8.9, 64.4), parallax 11.292 with NanoMaggies: g=6.41, r=6.34, z=6.44 cat entry: tabledata object with 1 rows and 62 columns Reference source: GaiaSource at GaiaPosition: RA, Dec = (280.86293, 79.77963), pm (-5.6, 1.5), parallax 2.398 with NanoMaggies: g=9.87, r=9.76, z=9.84 cat entry: tabledata object with 1 rows and 62 columns Reference source: GaiaSource at GaiaPosition: RA, Dec = (280.60208, 79.76202), pm (36.5, 67.9), parallax 9.184 with NanoMaggies: g=11.3, r=10.3, z=(flux -1.19e+03) cat entry: tabledata object with 1 rows and 62 columns Reference source: GaiaSource at GaiaPosition: RA, Dec = (280.59763, 79.78396), pm (1.4, 1.3), parallax 0.539 with NanoMaggies: g=12.7, r=11.8, z=11.4 cat entry: tabledata object with 1 rows and 62 columns Reference source: GaiaSource at GaiaPosition: RA, Dec = (280.63666, 79.78732), pm (-15.3, 25.0), parallax 2.196 with NanoMaggies: g=13.7, r=13.1, z=12.9 cat entry: tabledata object with 1 rows and 62 columns Reference source: GaiaSource at GaiaPosition: RA, Dec = (280.50401, 79.79325), pm (0.9, 4.9), parallax 0.904 with NanoMaggies: g=14.6, r=14, z=13.9 cat entry: tabledata object with 1 rows and 62 columns Reference source: GaiaSource at GaiaPosition: RA, Dec = (280.46623, 79.78034), pm (-1.4, -2.9), parallax 0.483 with NanoMaggies: g=16.8, r=16.3, z=16.1 cat entry: tabledata object with 1 rows and 62 columns Reference source: GaiaSource at GaiaPosition: RA, Dec = (280.47919, 79.77747), pm (-4.3, -1.9), parallax 0.552 with NanoMaggies: g=18.3, r=17.6, z=17.6 cat entry: tabledata object with 1 rows and 62 columns Reference source: GaiaSource at GaiaPosition: RA, Dec = (280.60086, 79.77134), pm (3.1, -15.7), parallax 2.348 with NanoMaggies: g=18.1, r=17, z=17.7 cat entry: tabledata object with 1 rows and 62 columns Reference source: ExpGalaxy at RaDecPos: RA, Dec = (280.53752, 79.77141) with NanoMaggies: g=16.5, r=15.7, z=15.1 and EllipseWithPriors(0.25): log r_e=1.19426, ee1=-0.0566719, ee2=0.00293207 cat entry: tabledata object with 1 rows and 62 columns cat entry: tabledata object with 2 rows and 62 columns Reference source: ExpGalaxy at RaDecPos: RA, Dec = (280.58875, 79.78753) with NanoMaggies: g=13.9, r=13.1, z=(flux -1.95e+03) and EllipseWithPriors(0.25): log r_e=1.85609, ee1=-7.23821e-10, ee2=-1.18646e-26 cat entry: tabledata object with 1 rows and 62 columns cat entry: tabledata object with 2 rows and 62 columns Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (281.40926, 79.94293), pm (8.9, 64.4), parallax 11.292 with NanoMaggies: g=6.41, r=6.34, z=6.44 Reset brightness to NanoMaggies: g=6.41, r=6.34, z=6.44 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.86293, 79.77963), pm (-5.6, 1.5), parallax 2.398 with NanoMaggies: g=9.87, r=9.76, z=9.84 Reset brightness to NanoMaggies: g=9.87, r=9.76, z=9.84 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.60208, 79.76202), pm (36.5, 67.9), parallax 9.184 with NanoMaggies: g=11.3, r=10.3, z=(flux -1.19e+03) Reset brightness to NanoMaggies: g=12.1, r=11.3, z=10.9 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.59763, 79.78396), pm (1.4, 1.3), parallax 0.539 with NanoMaggies: g=12.7, r=11.8, z=11.4 Reset brightness to NanoMaggies: g=12.7, r=11.8, z=11.4 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.63666, 79.78732), pm (-15.3, 25.0), parallax 2.196 with NanoMaggies: g=13.7, r=13.1, z=12.9 Reset brightness to NanoMaggies: g=13.7, r=13.1, z=12.9 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.50401, 79.79325), pm (0.9, 4.9), parallax 0.904 with NanoMaggies: g=14.6, r=14, z=13.9 Reset brightness to NanoMaggies: g=14.6, r=14, z=13.9 Known galaxy. Initial shape: EllipseWithPriors(0.25): log r_e=1.19426, ee1=-0.0566719, ee2=0.00293207 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.46623, 79.78034), pm (-1.4, -2.9), parallax 0.483 with NanoMaggies: g=16.8, r=16.3, z=16.1 Reset brightness to NanoMaggies: g=16.8, r=16.3, z=16.1 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.60086, 79.77134), pm (3.1, -15.7), parallax 2.348 with NanoMaggies: g=18.1, r=17, z=17.7 Reset brightness to NanoMaggies: g=22, r=20.6, z=19.8 Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (280.47919, 79.77747), pm (-4.3, -1.9), parallax 0.552 with NanoMaggies: g=18.3, r=17.6, z=17.6 Reset brightness to NanoMaggies: g=17, r=16.5, z=16.2 Measuring 13 iterative sources Blob 1 finished, total: Wall: 33.47 s, CPU: 33.46 s, VmPeak: 811 MB, VmSize: 760 MB, VmRSS: 188 MB, VmData: 216 MB, maxrss: 0.198632 MB Old catalog: 31 New catalog: 35 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/blobs-custom-280537p79771.fits.gz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/metrics/cus/all-models-custom-280537p79771.fits Stage fitblobs : Wall: 35.44 s, CPU: 1.69 s, VmPeak: 994 MB, VmSize: 994 MB, VmRSS: 231 MB, VmData: 258 MB, maxrss: 0.23652 MB Stage fitblobs finished: 2020-08-13T13:32:03.082830 Resources for stage fitblobs : 33.826 s worker CPU, 34.110 s worker Wall, Wall: 35.439 s, Cores in use: 1.00, Total efficiency (on 32 cores): 3.1 % Stage fitblobs finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-fitblobs.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-fitblobs.p Running stage coadds Running stage coadds at 2020-08-13T13:32:03.102563 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-ccds.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-nexp-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-depth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-galdepth-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-psfsize-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-chi2-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-blobmodel-g.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-nexp-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-depth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-galdepth-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-psfsize-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-chi2-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-blobmodel-r.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-nexp-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-depth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-galdepth-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-psfsize-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-chi2-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-blobmodel-z.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-depth.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/tmp-legacysurvey-custom-280537p79771-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/tmp-legacysurvey-custom-280537p79771-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/tmp-legacysurvey-custom-280537p79771-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-blobmodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/tmp-legacysurvey-custom-280537p79771-resid.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-resid.jpg Stage coadds : Wall: 2.17 s, CPU: 0.64 s, VmPeak: 994 MB, VmSize: 994 MB, VmRSS: 231 MB, VmData: 258 MB, maxrss: 0.236972 MB Stage coadds finished: 2020-08-13T13:32:05.274273 Resources for stage coadds : 26.010 s worker CPU, 27.062 s worker Wall, Wall: 2.174 s, Cores in use: 12.26, Total efficiency (on 32 cores): 38.3 % Stage coadds finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-coadds.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-coadds.p Running stage wise_forced Running stage wise_forced at 2020-08-13T13:32:05.295404 Cut to 2 unWISE tiles Reading WISE tile 2762p802 band 1 Reading WISE tile 2762p802 band 3 Reading WISE tile 2762p802 band 2 Reading WISE tile 2762p802 band 4 Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w4-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w4-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w4-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits Reading WISE tile 2846p802 band 4 Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w3-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w3-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w3-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits Reading WISE tile 2846p802 band 3 Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w2-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w2-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w2-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w2-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits Reading WISE tile 2846p802 band 2 Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w1-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w1-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w1-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/276/2762p802/unwise-2762p802-w1-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits Reading WISE tile 2846p802 band 1 Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w4-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w4-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w4-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w4-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.10 s, CPU: 0.10 s, VmPeak: 801 MB, VmSize: 801 MB, VmRSS: 146 MB, VmData: 179 MB, maxrss: 0.189564 MB Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w3-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w3-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w3-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w3-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.09 s, CPU: 0.09 s, VmPeak: 808 MB, VmSize: 793 MB, VmRSS: 137 MB, VmData: 171 MB, maxrss: 0.1987 MB Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w2-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w2-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w2-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w2-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.08 s, CPU: 0.08 s, VmPeak: 793 MB, VmSize: 793 MB, VmRSS: 138 MB, VmData: 171 MB, maxrss: 0.195092 MB Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w1-img-m.fits Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w1-invvar-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w1-n-m.fits.gz Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2846p802/unwise-2846p802-w1-n-u.fits.gz Reading /usr/local/lib/python/wise/wise-psf-avg.fits unWISE forced photometry took Wall: 0.08 s, CPU: 0.08 s, VmPeak: 804 MB, VmSize: 796 MB, VmRSS: 143 MB, VmData: 174 MB, maxrss: 0.199964 MB Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-W1.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-W2.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-W3.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-image-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-model-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-invvar-W4.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/tmp-legacysurvey-custom-280537p79771-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-wise.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/tmp-legacysurvey-custom-280537p79771-wisemodel.jpg Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-wisemodel.jpg Setting WISE apphot Setting WISE apphot Setting WISE apphot Setting WISE apphot Stage wise_forced : Wall: 3.49 s, CPU: 0.49 s, VmPeak: 994 MB, VmSize: 994 MB, VmRSS: 232 MB, VmData: 258 MB, maxrss: 0.23774 MB Stage wise_forced finished: 2020-08-13T13:32:08.782402 Resources for stage wise_forced : 6.224 s worker CPU, 8.330 s worker Wall, Wall: 3.489 s, Cores in use: 1.92, Total efficiency (on 32 cores): 6.0 % Stage wise_forced finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-wise_forced.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-wise_forced.p Running stage writecat Running stage writecat at 2020-08-13T13:32:09.031014 Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/coadd/cus/custom-280537p79771/legacysurvey-custom-280537p79771-maskbits.fits.fz Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/tractor-i/cus/tractor-custom-280537p79771.fits Reading SFD maps... END Rejected (1), keyrecords following the END keyrecord were ignored. END Rejected (1), keyrecords following the END keyrecord were ignored. Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/tractor/cus/tractor-custom-280537p79771.fits Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/tractor/cus/brick-custom-280537p79771.sha256sum Stage writecat : Wall: 0.80 s, CPU: 0.59 s, VmPeak: 1126 MB, VmSize: 998 MB, VmRSS: 234 MB, VmData: 258 MB, maxrss: 0.370752 MB Stage writecat finished: 2020-08-13T13:32:09.830580 Resources for stage writecat : 0.000 s worker CPU, 0.001 s worker Wall, Wall: 0.802 s, Cores in use: 0.75, Total efficiency (on 32 cores): 2.3 % Stage writecat finished Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-writecat.p Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/280/3C390.3/3C390.3-largegalaxy-writecat.p All done: 73.382 s worker CPU, 83.944 s worker Wall, Wall: 53.281 s, Cores in use: 1.60, Total efficiency (on 32 cores): 5.0 % Finished galaxy 3C390.3 in 0.941 minutes.