Started working on galaxy PGC2050816 at Wed Aug  5 19:54:19 2020
Bands touching this brick, g r z
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 277.655583 34.5454025 --width 251 --height 251 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 

runbrick.py starting at 2020-08-05T19:54:19.883480
legacypipe git version: DR9.5.10-12-gf22e529f
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '277.655583', '34.5454025', '--width', '251', '--height', '251', '--pixscale', '0.262', '--threads', '16', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-%%(stage)s.p', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 277.655583 34.5454025 --width 251 --height 251 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0

Parsed RA,Dec 277.655583 34.5454025
Starting process 13483 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 13484 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 13485 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 13486 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 13487 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059352 MB
Starting process 13488 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05936 MB
Starting process 13489 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05936 MB
Starting process 13490 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 13491 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 13492 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 13493 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 13494 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 13495 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 13496 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 13497 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 13498 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage fit_on_coadds
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-08-05T19:54:20.360374
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Keeping 8 CCDs:
  k4m_160418_115925_ooi_zd_ls9-CCD1 z expnum 83130 exptime 115.104 propid 2016A-0453 seeing 1.55 MJD 57496.500 object MzLS_577361_z 
    mosaic/CP/V4.3/CP20160417/k4m_160418_115925_ooi_zd_ls9.fits.fz
  k4m_160415_120705_ooi_zd_ls9-CCD4 z expnum 82539 exptime 127.102 propid 2016A-0453 seeing 1.26 MJD 57493.505 object MzLS_576329_z 
    mosaic/CP/V4.3/CP20160414/k4m_160415_120705_ooi_zd_ls9.fits.fz
  k4m_170807_081211_ooi_zd_ls9-CCD4 z expnum 136168 exptime 151.104 propid 2016A-0453 seeing 1.09 MJD 57972.342 object MzLS_576329_z 
    mosaic/CP/V4.3/CP20170806/k4m_170807_081211_ooi_zd_ls9.fits.fz
  k4m_160417_120058_ooi_zd_ls9-CCD4 z expnum 82869 exptime 80.103 propid 2016A-0453 seeing 1.09 MJD 57495.501 object MzLS_575295_z 
    mosaic/CP/V4.3/CP20160416/k4m_160417_120058_ooi_zd_ls9.fits.fz
  ksb_180518_084042_ooi_r_ls9-CCD2 r expnum 82560157 exptime 160.286 propid NOTBASS seeing 1.99 MJD 58256.362 object 60893 
    90prime/CP/V2.3/CP20180517/ksb_180518_084042_ooi_r_ls9.fits.fz
  ksb_160628_070316_ooi_g_ls9-CCD2 g expnum 75670100 exptime 115.2 propid BASS seeing 1.62 MJD 57567.294 object 60893 
    90prime/CP/V2.3/CP20160627/ksb_160628_070316_ooi_g_ls9.fits.fz
  ksb_160602_085337_ooi_g_ls9-CCD1 g expnum 75410139 exptime 126.7 propid BASS seeing 1.35 MJD 57541.371 object 60732 
    90prime/CP/V2.3/CP20160601/ksb_160602_085337_ooi_g_ls9.fits.fz
  ksb_160510_110400_ooi_r_ls9-CCD1 r expnum 75180256 exptime 63.5 propid BASS seeing 1.13 MJD 57518.461 object 60732 
    90prime/CP/V2.3/CP20160509/ksb_160510_110400_ooi_r_ls9.fits.fz
Faked up EXPNUM 82560157
Faked up EXPNUM 82560157
Faked up EXPNUM 82560157
90prime get_fwhm: self.fwhm = 4.39
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 1449 1596 and amp range 0 2015 and image y 2075 2223 and amp range 2048 4095
Range in image: x 0 148 , y 0 149 (with image size 148 x 149)
dzp 0.0503617 -> scaling image by 1.047477444828492
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 75670100
Faked up EXPNUM 75670100
Faked up EXPNUM 75670100
90prime get_fwhm: self.fwhm = 3.57
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 1446 1593 and amp range 0 2015 and image y 2096 2243 and amp range 2048 4095
Range in image: x 0 148 , y 0 148 (with image size 148 x 148)
dzp 0.01693853 -> scaling image by 1.015723292931903
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 75180256
Faked up EXPNUM 75180256
Faked up EXPNUM 75180256
90prime get_fwhm: self.fwhm = 2.49
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 1675 1822 and amp range 0 2015 and image y 3738 3884 and amp range 2048 4095
Range in image: x 0 148 , y 0 147 (with image size 148 x 147)
dzp -0.00015519658 -> scaling image by 0.9998570688798908
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 75410139
Faked up EXPNUM 75410139
Faked up EXPNUM 75410139
90prime get_fwhm: self.fwhm = 2.99
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 1675 1822 and amp range 0 2015 and image y 3723 3870 and amp range 2048 4095
Range in image: x 0 148 , y 0 148 (with image size 148 x 148)
dzp -0.0051191514 -> scaling image by 0.9952961850385206
mosaic get_fwhm: self.fwhm = 5.97
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 1669 1924 and amp range 0 2005 and image y 2955 3208 and amp range 2040 4078
Range in image: x 0 256 , y 0 254 (with image size 256 x 254)
dzp -0.005990375 -> scaling image by 0.994497853444703
mosaic get_fwhm: self.fwhm = 4.21
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 102 356 and amp range 0 2047 and image y 2369 2624 and amp range 2022 4060
Range in image: x 0 255 , y 0 256 (with image size 255 x 256)
dzp -0.016192675 -> scaling image by 0.9851966584456431
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 4.21
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2816 3073 and amp range 2048 4095 and image y 1060 1314 and amp range 0 2021
Range in image: x 0 258 , y 0 255 (with image size 258 x 255)
dzp -0.0006399284 -> scaling image by 0.9994107778325068
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 4.85
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 117 371 and amp range 0 2047 and image y 2392 2646 and amp range 2022 4060
Range in image: x 0 255 , y 0 255 (with image size 255 x 255)
dzp 0.0033053984 -> scaling image by 1.0030490232713904
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Stage tims : Wall: 6.94 s, CPU: 0.90 s, VmPeak: 957 MB, VmSize: 957 MB, VmRSS: 175 MB, VmData: 207 MB, maxrss: 0.179348 MB
Stage tims finished: 2020-08-05T19:54:27.300711
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage tims : 4.109 s worker CPU, 25.084 s worker Wall, Wall: 6.942 s, Cores in use: 0.72, Total efficiency (on 16 cores): 4.5 %
Stage tims finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-tims.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-tims.p
Running stage refs
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage refs at 2020-08-05T19:54:27.322180
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-02850.fits
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-02723.fits
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-02594.fits
Matched 4 large galaxies to Gaia stars.
Matched 0 large galaxies to Tycho-2 stars.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/metrics/cus/reference-custom-277655p34545.fits
Stage refs : Wall: 13.08 s, CPU: 9.66 s, VmPeak: 1070 MB, VmSize: 1064 MB, VmRSS: 286 MB, VmData: 315 MB, maxrss: 0.292792 MB
Stage refs finished: 2020-08-05T19:54:40.406150
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 13.086 s, Cores in use: 0.74, Total efficiency (on 16 cores): 4.6 %
Stage refs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-refs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-refs.p
Running stage outliers
Running stage outliers at 2020-08-05T19:54:40.480165
Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/metrics/cus/outlier-mask-custom-277655p34545.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/metrics/cus/outliers-pre-custom-277655p34545.jpg
Image ksb_160602_085337_ooi_g_ls9-CCD1 g : masking 34 positive outlier pixels and 0 negative outlier pixels
Image ksb_180518_084042_ooi_r_ls9-CCD2 r : masking 118 positive outlier pixels and 0 negative outlier pixels
Image k4m_160415_120705_ooi_zd_ls9-CCD4 z : masking 0 positive outlier pixels and 264 negative outlier pixels
Image k4m_170807_081211_ooi_zd_ls9-CCD4 z : masking 1176 positive outlier pixels and 0 negative outlier pixels
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/metrics/cus/outlier-mask-custom-277655p34545.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/metrics/cus/outliers-post-custom-277655p34545.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/metrics/cus/outliers-masked-pos-custom-277655p34545.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/metrics/cus/outliers-masked-neg-custom-277655p34545.jpg
Stage outliers : Wall: 0.97 s, CPU: 0.59 s, VmPeak: 1074 MB, VmSize: 1074 MB, VmRSS: 298 MB, VmData: 324 MB, maxrss: 0.304736 MB
Stage outliers finished: 2020-08-05T19:54:41.447732
Resources for stage outliers : 0.636 s worker CPU, 0.930 s worker Wall, Wall: 0.969 s, Cores in use: 1.27, Total efficiency (on 16 cores): 7.9 %
Stage outliers finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-outliers.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-outliers.p
Running stage halos
Running stage halos at 2020-08-05T19:54:41.472207
Stage halos : Wall: 0.45 s, CPU: 0.43 s, VmPeak: 1074 MB, VmSize: 1074 MB, VmRSS: 298 MB, VmData: 325 MB, maxrss: 0.305108 MB
Stage halos finished: 2020-08-05T19:54:41.918252
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.448 s, Cores in use: 0.97, Total efficiency (on 16 cores): 6.1 %
Stage halos finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-halos.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-halos.p
Running stage fit_on_coadds
Warning: Stellar halo subtraction is only implemented for DECam
Running stage fit_on_coadds at 2020-08-05T19:54:41.939212
PSF sigmas in coadd pixels: ['2.63', '2.20']
Using average PSF sigma 2.413185491002638
Gaussian PSF norm 0.11689726828942291 vs pixelized 0.09785512
average tim pixel scale / coadd scale: 1.7289383863880945
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-copsf-g.fits.fz
PSF sigmas in coadd pixels: ['3.23', '1.83']
Using average PSF sigma 2.5326713135314485
Gaussian PSF norm 0.11138231410713033 vs pixelized 0.08822795
average tim pixel scale / coadd scale: 1.7300093184822445
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-copsf-r.fits.fz
PSF sigmas in coadd pixels: ['2.52', '2.05', '1.78', '1.78']
Using average PSF sigma 2.028297773669782
Gaussian PSF norm 0.13907957472313667 vs pixelized 0.11229317
average tim pixel scale / coadd scale: 0.9909544014798491
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-copsf-z.fits.fz
Stage fit_on_coadds : Wall: 0.14 s, CPU: 0.13 s, VmPeak: 1074 MB, VmSize: 1074 MB, VmRSS: 299 MB, VmData: 325 MB, maxrss: 0.30658 MB
Stage fit_on_coadds finished: 2020-08-05T19:54:42.082244
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage fit_on_coadds : 0.140 s worker CPU, 0.140 s worker Wall, Wall: 0.145 s, Cores in use: 1.89, Total efficiency (on 16 cores): 11.8 %
Stage fit_on_coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-fit_on_coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-fit_on_coadds.p
Running stage srcs
Running stage srcs at 2020-08-05T19:54:42.092928
SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])]
Found 60 potential peaks
Of 60 potential peaks: 16 in veto map, 11 cut by saddle test, 0 cut by aper test, 33 kept
SED z : 33 new peaks
Found 45 potential peaks
Of 45 potential peaks: 37 in veto map, 7 cut by saddle test, 0 cut by aper test, 1 kept
SED r : 1 new peaks
Found 41 potential peaks
Of 41 potential peaks: 34 in veto map, 6 cut by saddle test, 0 cut by aper test, 1 kept
SED g : 1 new peaks
Found 60 potential peaks
Of 60 potential peaks: 49 in veto map, 11 cut by saddle test, 0 cut by aper test, 0 kept
SED Flat : 0 new peaks
Found 63 potential peaks
Of 63 potential peaks: 54 in veto map, 9 cut by saddle test, 0 cut by aper test, 0 kept
SED Red : 0 new peaks
Adding 35 new sources
Median coadd sky for g : -9.5624746e-05
Median coadd sky for r : -0.00014947305
Median coadd sky for z : -0.00030689043
Stage srcs : Wall: 0.55 s, CPU: 0.51 s, VmPeak: 1074 MB, VmSize: 1074 MB, VmRSS: 300 MB, VmData: 325 MB, maxrss: 0.307084 MB
Stage srcs finished: 2020-08-05T19:54:42.641032
Resources for stage srcs : 0.070 s worker CPU, 0.070 s worker Wall, Wall: 0.550 s, Cores in use: 1.06, Total efficiency (on 16 cores): 6.6 %
Stage srcs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-srcs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-srcs.p
Running stage fitblobs
Running stage fitblobs at 2020-08-05T19:54:42.660097
Blob 1 of 11, id: 1, sources: 30, size: 206x251, npix 24167, brick X: 42,248, Y: 0,251, one pixel: 73 0
Blob 2 of 11, id: 9, sources: 6, size: 93x62, npix 3299, brick X: 6,99, Y: 189,251, one pixel: 66 189
Blob 3 of 11, id: 3, sources: 3, size: 47x61, npix 1499, brick X: 29,76, Y: 104,165, one pixel: 42 104
Blob 4 of 11, id: 8, sources: 2, size: 40x47, npix 1261, brick X: 211,251, Y: 174,221, one pixel: 244 174
Blob 5 of 11, id: 6, sources: 2, size: 43x46, npix 1180, brick X: 0,43, Y: 157,203, one pixel: 4 157
Blob 6 of 11, id: 4, sources: 2, size: 37x37, npix 756, brick X: 214,251, Y: 112,149, one pixel: 227 112
Blob 7 of 11, id: 0, sources: 1, size: 28x36, npix 685, brick X: 15,43, Y: 0,36, one pixel: 25 0
Blob 8 of 11, id: 2, sources: 1, size: 27x25, npix 430, brick X: 29,56, Y: 49,74, one pixel: 39 49
Blob 9 of 11, id: 5, sources: 1, size: 25x24, npix 387, brick X: 218,243, Y: 153,177, one pixel: 228 153
Blob 10 of 11, id: 7, sources: 1, size: 22x22, npix 293, brick X: 73,95, Y: 166,188, one pixel: 82 166
Blob 11 of 11, id: 10, sources: 2, size: 24x16, npix 261, brick X: 183,207, Y: 235,251, one pixel: 195 235
Blob 10 finished, total: Wall: 0.18 s, CPU: 0.18 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 91 MB, VmData: 138 MB, maxrss: 0.092752 MB
Blob 9 finished, total: Wall: 0.19 s, CPU: 0.18 s, VmPeak: 815 MB, VmSize: 731 MB, VmRSS: 146 MB, VmData: 183 MB, maxrss: 0.149892 MB
Blob 8 finished, total: Wall: 0.23 s, CPU: 0.23 s, VmPeak: 812 MB, VmSize: 731 MB, VmRSS: 146 MB, VmData: 183 MB, maxrss: 0.14928 MB
Measuring 1 iterative sources
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (277.64865, 34.56094), pm (-1.2, -6.4), parallax 0.705 with NanoMaggies: g=17.2, r=16.5, z=16.3
Reset brightness to NanoMaggies: g=17.2, r=16.5, z=16.3
Blob 11 finished, total: Wall: 0.56 s, CPU: 0.54 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 92 MB, VmData: 139 MB, maxrss: 0.09428 MB
Blob 7 finished, total: Wall: 0.75 s, CPU: 0.73 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 91 MB, VmData: 138 MB, maxrss: 0.09292 MB
Blob 6 finished, total: Wall: 0.94 s, CPU: 0.93 s, VmPeak: 813 MB, VmSize: 732 MB, VmRSS: 146 MB, VmData: 183 MB, maxrss: 0.14992 MB
Blob 5 finished, total: Wall: 0.97 s, CPU: 0.95 s, VmPeak: 817 MB, VmSize: 732 MB, VmRSS: 146 MB, VmData: 183 MB, maxrss: 0.149928 MB
Measuring 3 iterative sources
Measuring 1 iterative sources
Blob 4 finished, total: Wall: 1.56 s, CPU: 1.55 s, VmPeak: 809 MB, VmSize: 731 MB, VmRSS: 148 MB, VmData: 183 MB, maxrss: 0.151172 MB
Blob 3 finished, total: Wall: 1.97 s, CPU: 1.95 s, VmPeak: 669 MB, VmSize: 669 MB, VmRSS: 93 MB, VmData: 140 MB, maxrss: 0.095548 MB
Measuring 3 iterative sources
Blob 2 finished, total: Wall: 4.19 s, CPU: 4.18 s, VmPeak: 672 MB, VmSize: 671 MB, VmRSS: 97 MB, VmData: 143 MB, maxrss: 0.100016 MB
Known galaxy.  Initial shape: EllipseWithPriors(0.25): log r_e=1.63628, ee1=0.283611, ee2=-0.220582
Measuring 9 iterative sources
Blob 1 finished, total: Wall: 19.22 s, CPU: 19.21 s, VmPeak: 716 MB, VmSize: 692 MB, VmRSS: 124 MB, VmData: 163 MB, maxrss: 0.151312 MB
Old catalog: 59
New catalog: 67
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/metrics/cus/blobs-custom-277655p34545.fits.gz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/metrics/cus/all-models-custom-277655p34545.fits
Stage fitblobs : Wall: 21.31 s, CPU: 1.82 s, VmPeak: 1074 MB, VmSize: 1074 MB, VmRSS: 302 MB, VmData: 327 MB, maxrss: 0.309536 MB
Stage fitblobs finished: 2020-08-05T19:55:03.971711
Resources for stage fitblobs : 30.823 s worker CPU, 32.754 s worker Wall, Wall: 21.313 s, Cores in use: 1.53, Total efficiency (on 16 cores): 9.6 %
Stage fitblobs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-fitblobs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-fitblobs.p
Running stage coadds
Running stage coadds at 2020-08-05T19:55:03.993632
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-ccds.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-nexp-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-depth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-galdepth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-psfsize-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-chi2-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-blobmodel-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-nexp-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-depth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-galdepth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-psfsize-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-chi2-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-blobmodel-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-nexp-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-depth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-galdepth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-psfsize-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-chi2-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-blobmodel-z.fits.fz
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-depth.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/tmp-legacysurvey-custom-277655p34545-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/tmp-legacysurvey-custom-277655p34545-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/tmp-legacysurvey-custom-277655p34545-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/tmp-legacysurvey-custom-277655p34545-resid.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-resid.jpg
Stage coadds : Wall: 4.34 s, CPU: 1.12 s, VmPeak: 1074 MB, VmSize: 1074 MB, VmRSS: 303 MB, VmData: 327 MB, maxrss: 0.30992 MB
Stage coadds finished: 2020-08-05T19:55:08.330047
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage coadds : 14.652 s worker CPU, 24.411 s worker Wall, Wall: 4.338 s, Cores in use: 3.64, Total efficiency (on 16 cores): 22.7 %
Stage coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-coadds.p
Running stage wise_forced
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage wise_forced at 2020-08-05T19:55:08.348173
Cut to 1 unWISE tiles
Reading WISE tile 2777p348 band 1
Reading WISE tile 2777p348 band 2
Reading WISE tile 2777p348 band 4
Reading WISE tile 2777p348 band 3
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w3-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w3-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w3-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.11 s, CPU: 0.28 s, VmPeak: 1006 MB, VmSize: 1006 MB, VmRSS: 146 MB, VmData: 186 MB, maxrss: 0.206108 MB
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w4-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w4-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w4-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.12 s, CPU: 0.32 s, VmPeak: 1014 MB, VmSize: 1014 MB, VmRSS: 153 MB, VmData: 190 MB, maxrss: 0.192268 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w1-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w1-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w1-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w1-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.08 s, CPU: 0.19 s, VmPeak: 1014 MB, VmSize: 1014 MB, VmRSS: 155 MB, VmData: 190 MB, maxrss: 0.219536 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w2-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w2-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w2-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w2-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.09 s, CPU: 0.21 s, VmPeak: 1006 MB, VmSize: 1006 MB, VmRSS: 146 MB, VmData: 186 MB, maxrss: 0.205772 MB
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/tmp-legacysurvey-custom-277655p34545-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/tmp-legacysurvey-custom-277655p34545-wisemodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-wisemodel.jpg
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Stage wise_forced : Wall: 3.06 s, CPU: 1.16 s, VmPeak: 1074 MB, VmSize: 1074 MB, VmRSS: 303 MB, VmData: 327 MB, maxrss: 0.310436 MB
Stage wise_forced finished: 2020-08-05T19:55:11.404768
Resources for stage wise_forced : 4.033 s worker CPU, 5.941 s worker Wall, Wall: 3.058 s, Cores in use: 1.70, Total efficiency (on 16 cores): 10.6 %
Stage wise_forced finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-wise_forced.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-wise_forced.p
Running stage writecat
Running stage writecat at 2020-08-05T19:55:11.423093
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-maskbits.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/tractor-i/cus/tractor-custom-277655p34545.fits
Reading SFD maps...
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/tractor/cus/tractor-custom-277655p34545.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/tractor/cus/brick-custom-277655p34545.sha256sum
Stage writecat : Wall: 6.60 s, CPU: 2.46 s, VmPeak: 1207 MB, VmSize: 1079 MB, VmRSS: 305 MB, VmData: 327 MB, maxrss: 0.443316 MB
Stage writecat finished: 2020-08-05T19:55:18.021466
Resources for stage writecat : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 6.600 s, Cores in use: 0.37, Total efficiency (on 16 cores): 2.3 %
Stage writecat finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-writecat.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-writecat.p
All done: 54.467 s worker CPU, 89.335 s worker Wall, Wall: 57.687 s, Cores in use: 1.27, Total efficiency (on 16 cores): 8.0 %
Finished galaxy PGC2050816 in 0.995 minutes.
Started working on galaxy PGC2050816 at Wed Aug  5 20:31:14 2020
Bands touching this brick, g r z
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 277.655583 34.5454025 --width 251 --height 251 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 

runbrick.py starting at 2020-08-05T20:31:18.146878
legacypipe git version: DR9.5.10-12-gf22e529f
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '277.655583', '34.5454025', '--width', '251', '--height', '251', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-%%(stage)s.p', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 277.655583 34.5454025 --width 251 --height 251 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0

Parsed RA,Dec 277.655583 34.5454025
Starting process 21311 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 21312 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 21313 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 21314 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 21315 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059484 MB
Starting process 21316 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059492 MB
Starting process 21317 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059492 MB
Starting process 21318 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 21319 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 21320 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 21321 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 21322 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 21323 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 21324 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 21325 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 21326 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 21327 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059276 MB
Starting process 21328 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05928 MB
Starting process 21329 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB
Starting process 21330 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059292 MB
Starting process 21331 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059292 MB
Starting process 21332 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB
Starting process 21333 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB
Starting process 21334 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Starting process 21335 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Starting process 21336 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Starting process 21337 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Starting process 21338 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059308 MB
Starting process 21339 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB
Starting process 21340 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB
Starting process 21341 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB
Starting process 21342 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059316 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage fit_on_coadds
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-08-05T20:31:20.264292
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Keeping 8 CCDs:
  k4m_160418_115925_ooi_zd_ls9-CCD1 z expnum 83130 exptime 115.104 propid 2016A-0453 seeing 1.55 MJD 57496.500 object MzLS_577361_z 
    mosaic/CP/V4.3/CP20160417/k4m_160418_115925_ooi_zd_ls9.fits.fz
  k4m_160415_120705_ooi_zd_ls9-CCD4 z expnum 82539 exptime 127.102 propid 2016A-0453 seeing 1.26 MJD 57493.505 object MzLS_576329_z 
    mosaic/CP/V4.3/CP20160414/k4m_160415_120705_ooi_zd_ls9.fits.fz
  k4m_170807_081211_ooi_zd_ls9-CCD4 z expnum 136168 exptime 151.104 propid 2016A-0453 seeing 1.09 MJD 57972.342 object MzLS_576329_z 
    mosaic/CP/V4.3/CP20170806/k4m_170807_081211_ooi_zd_ls9.fits.fz
  k4m_160417_120058_ooi_zd_ls9-CCD4 z expnum 82869 exptime 80.103 propid 2016A-0453 seeing 1.09 MJD 57495.501 object MzLS_575295_z 
    mosaic/CP/V4.3/CP20160416/k4m_160417_120058_ooi_zd_ls9.fits.fz
  ksb_180518_084042_ooi_r_ls9-CCD2 r expnum 82560157 exptime 160.286 propid NOTBASS seeing 1.99 MJD 58256.362 object 60893 
    90prime/CP/V2.3/CP20180517/ksb_180518_084042_ooi_r_ls9.fits.fz
  ksb_160628_070316_ooi_g_ls9-CCD2 g expnum 75670100 exptime 115.2 propid BASS seeing 1.62 MJD 57567.294 object 60893 
    90prime/CP/V2.3/CP20160627/ksb_160628_070316_ooi_g_ls9.fits.fz
  ksb_160602_085337_ooi_g_ls9-CCD1 g expnum 75410139 exptime 126.7 propid BASS seeing 1.35 MJD 57541.371 object 60732 
    90prime/CP/V2.3/CP20160601/ksb_160602_085337_ooi_g_ls9.fits.fz
  ksb_160510_110400_ooi_r_ls9-CCD1 r expnum 75180256 exptime 63.5 propid BASS seeing 1.13 MJD 57518.461 object 60732 
    90prime/CP/V2.3/CP20160509/ksb_160510_110400_ooi_r_ls9.fits.fz
Faked up EXPNUM 75410139
Faked up EXPNUM 75410139
Faked up EXPNUM 75410139
90prime get_fwhm: self.fwhm = 2.99
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 1675 1822 and amp range 0 2015 and image y 3723 3870 and amp range 2048 4095
Range in image: x 0 148 , y 0 148 (with image size 148 x 148)
dzp -0.0051191514 -> scaling image by 0.9952961850385206
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 75180256
Faked up EXPNUM 75180256
Faked up EXPNUM 75180256
90prime get_fwhm: self.fwhm = 2.49
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 1675 1822 and amp range 0 2015 and image y 3738 3884 and amp range 2048 4095
Range in image: x 0 148 , y 0 147 (with image size 148 x 147)
dzp -0.00015519658 -> scaling image by 0.9998570688798908
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 5.97
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 1669 1924 and amp range 0 2005 and image y 2955 3208 and amp range 2040 4078
Range in image: x 0 256 , y 0 254 (with image size 256 x 254)
dzp -0.005990375 -> scaling image by 0.994497853444703
Faked up EXPNUM 82560157
Faked up EXPNUM 82560157
Faked up EXPNUM 82560157
90prime get_fwhm: self.fwhm = 4.39
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 1449 1596 and amp range 0 2015 and image y 2075 2223 and amp range 2048 4095
Range in image: x 0 148 , y 0 149 (with image size 148 x 149)
dzp 0.0503617 -> scaling image by 1.047477444828492
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 75670100
Faked up EXPNUM 75670100
Faked up EXPNUM 75670100
90prime get_fwhm: self.fwhm = 3.57
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 1446 1593 and amp range 0 2015 and image y 2096 2243 and amp range 2048 4095
Range in image: x 0 148 , y 0 148 (with image size 148 x 148)
dzp 0.01693853 -> scaling image by 1.015723292931903
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 4.21
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2816 3073 and amp range 2048 4095 and image y 1060 1314 and amp range 0 2021
Range in image: x 0 258 , y 0 255 (with image size 258 x 255)
dzp -0.0006399284 -> scaling image by 0.9994107778325068
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 4.85
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 117 371 and amp range 0 2047 and image y 2392 2646 and amp range 2022 4060
Range in image: x 0 255 , y 0 255 (with image size 255 x 255)
dzp 0.0033053984 -> scaling image by 1.0030490232713904
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 4.21
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 102 356 and amp range 0 2047 and image y 2369 2624 and amp range 2022 4060
Range in image: x 0 255 , y 0 256 (with image size 255 x 256)
dzp -0.016192675 -> scaling image by 0.9851966584456431
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Stage tims : Wall: 7.27 s, CPU: 1.34 s, VmPeak: 957 MB, VmSize: 957 MB, VmRSS: 174 MB, VmData: 207 MB, maxrss: 0.178672 MB
Stage tims finished: 2020-08-05T20:31:27.538188
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage tims : 4.202 s worker CPU, 24.381 s worker Wall, Wall: 7.276 s, Cores in use: 0.76, Total efficiency (on 32 cores): 2.4 %
Stage tims finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-tims.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-tims.p
Running stage refs
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage refs at 2020-08-05T20:31:27.662327
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-02850.fits
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-02723.fits
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-02594.fits
Matched 4 large galaxies to Gaia stars.
Matched 0 large galaxies to Tycho-2 stars.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/metrics/cus/reference-custom-277655p34545.fits
Stage refs : Wall: 15.74 s, CPU: 9.99 s, VmPeak: 1070 MB, VmSize: 1064 MB, VmRSS: 285 MB, VmData: 314 MB, maxrss: 0.292088 MB
Stage refs finished: 2020-08-05T20:31:43.404617
Resources for stage refs : 0.000 s worker CPU, 0.001 s worker Wall, Wall: 15.745 s, Cores in use: 0.63, Total efficiency (on 32 cores): 2.0 %
Stage refs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-refs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-refs.p
Running stage outliers
Running stage outliers at 2020-08-05T20:31:43.425566
Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/metrics/cus/outlier-mask-custom-277655p34545.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/metrics/cus/outliers-pre-custom-277655p34545.jpg
Image ksb_160602_085337_ooi_g_ls9-CCD1 g : masking 34 positive outlier pixels and 0 negative outlier pixels
Image ksb_180518_084042_ooi_r_ls9-CCD2 r : masking 118 positive outlier pixels and 0 negative outlier pixels
Image k4m_160415_120705_ooi_zd_ls9-CCD4 z : masking 0 positive outlier pixels and 264 negative outlier pixels
Image k4m_170807_081211_ooi_zd_ls9-CCD4 z : masking 1176 positive outlier pixels and 0 negative outlier pixels
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/metrics/cus/outlier-mask-custom-277655p34545.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/metrics/cus/outliers-post-custom-277655p34545.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/metrics/cus/outliers-masked-pos-custom-277655p34545.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/metrics/cus/outliers-masked-neg-custom-277655p34545.jpg
Stage outliers : Wall: 1.16 s, CPU: 0.73 s, VmPeak: 1074 MB, VmSize: 1074 MB, VmRSS: 297 MB, VmData: 324 MB, maxrss: 0.30458 MB
Stage outliers finished: 2020-08-05T20:31:44.580858
Resources for stage outliers : 0.782 s worker CPU, 1.004 s worker Wall, Wall: 1.158 s, Cores in use: 1.31, Total efficiency (on 32 cores): 4.1 %
Stage outliers finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-outliers.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-outliers.p
Running stage halos
Running stage halos at 2020-08-05T20:31:44.607197
Stage halos : Wall: 0.56 s, CPU: 0.54 s, VmPeak: 1074 MB, VmSize: 1074 MB, VmRSS: 298 MB, VmData: 325 MB, maxrss: 0.305064 MB
Stage halos finished: 2020-08-05T20:31:45.163602
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.559 s, Cores in use: 0.98, Total efficiency (on 32 cores): 3.1 %
Stage halos finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-halos.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-halos.p
Running stage fit_on_coadds
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Running stage fit_on_coadds at 2020-08-05T20:31:45.237693
PSF sigmas in coadd pixels: ['2.63', '2.20']
Using average PSF sigma 2.413185491002638
Gaussian PSF norm 0.11689726828942291 vs pixelized 0.09785512
average tim pixel scale / coadd scale: 1.7289383863880945
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-copsf-g.fits.fz
PSF sigmas in coadd pixels: ['3.23', '1.83']
Using average PSF sigma 2.5326713135314485
Gaussian PSF norm 0.11138231410713033 vs pixelized 0.08822795
average tim pixel scale / coadd scale: 1.7300093184822445
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-copsf-r.fits.fz
PSF sigmas in coadd pixels: ['2.52', '2.05', '1.78', '1.78']
Using average PSF sigma 2.028297773669782
Gaussian PSF norm 0.13907957472313667 vs pixelized 0.11229316
average tim pixel scale / coadd scale: 0.9909544014798491
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-copsf-z.fits.fz
Stage fit_on_coadds : Wall: 0.20 s, CPU: 0.13 s, VmPeak: 1074 MB, VmSize: 1074 MB, VmRSS: 299 MB, VmData: 325 MB, maxrss: 0.306544 MB
Stage fit_on_coadds finished: 2020-08-05T20:31:45.433363
Resources for stage fit_on_coadds : 0.128 s worker CPU, 0.128 s worker Wall, Wall: 0.198 s, Cores in use: 1.33, Total efficiency (on 32 cores): 4.2 %
Stage fit_on_coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-fit_on_coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-fit_on_coadds.p
Running stage srcs
Warning: Stellar halo subtraction is only implemented for DECam
Running stage srcs at 2020-08-05T20:31:45.446031
SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])]
Found 60 potential peaks
Of 60 potential peaks: 16 in veto map, 11 cut by saddle test, 0 cut by aper test, 33 kept
SED z : 33 new peaks
Found 45 potential peaks
Of 45 potential peaks: 37 in veto map, 7 cut by saddle test, 0 cut by aper test, 1 kept
SED r : 1 new peaks
Found 41 potential peaks
Of 41 potential peaks: 34 in veto map, 6 cut by saddle test, 0 cut by aper test, 1 kept
SED g : 1 new peaks
Found 60 potential peaks
Of 60 potential peaks: 49 in veto map, 11 cut by saddle test, 0 cut by aper test, 0 kept
SED Flat : 0 new peaks
Found 63 potential peaks
Of 63 potential peaks: 54 in veto map, 9 cut by saddle test, 0 cut by aper test, 0 kept
SED Red : 0 new peaks
Adding 35 new sources
Median coadd sky for g : -9.5624746e-05
Median coadd sky for r : -0.00014947305
Median coadd sky for z : -0.0003068904
Stage srcs : Wall: 0.67 s, CPU: 0.63 s, VmPeak: 1074 MB, VmSize: 1074 MB, VmRSS: 300 MB, VmData: 325 MB, maxrss: 0.307048 MB
Stage srcs finished: 2020-08-05T20:31:46.115516
Resources for stage srcs : 0.070 s worker CPU, 0.070 s worker Wall, Wall: 0.672 s, Cores in use: 1.05, Total efficiency (on 32 cores): 3.3 %
Stage srcs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-srcs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-srcs.p
Running stage fitblobs
Running stage fitblobs at 2020-08-05T20:31:46.130103
Blob 1 of 11, id: 1, sources: 30, size: 206x251, npix 24167, brick X: 42,248, Y: 0,251, one pixel: 73 0
Blob 2 of 11, id: 9, sources: 6, size: 93x62, npix 3299, brick X: 6,99, Y: 189,251, one pixel: 66 189
Blob 3 of 11, id: 3, sources: 3, size: 47x61, npix 1499, brick X: 29,76, Y: 104,165, one pixel: 42 104
Blob 4 of 11, id: 8, sources: 2, size: 40x47, npix 1261, brick X: 211,251, Y: 174,221, one pixel: 244 174
Blob 5 of 11, id: 6, sources: 2, size: 43x46, npix 1180, brick X: 0,43, Y: 157,203, one pixel: 4 157
Blob 6 of 11, id: 4, sources: 2, size: 37x37, npix 756, brick X: 214,251, Y: 112,149, one pixel: 227 112
Blob 7 of 11, id: 0, sources: 1, size: 28x36, npix 685, brick X: 15,43, Y: 0,36, one pixel: 25 0
Blob 8 of 11, id: 2, sources: 1, size: 27x25, npix 430, brick X: 29,56, Y: 49,74, one pixel: 39 49
Blob 9 of 11, id: 5, sources: 1, size: 25x24, npix 387, brick X: 218,243, Y: 153,177, one pixel: 228 153
Blob 10 of 11, id: 7, sources: 1, size: 22x22, npix 293, brick X: 73,95, Y: 166,188, one pixel: 82 166
Blob 11 of 11, id: 10, sources: 2, size: 24x16, npix 261, brick X: 183,207, Y: 235,251, one pixel: 195 235
Blob 8 finished, total: Wall: 0.16 s, CPU: 0.16 s, VmPeak: 815 MB, VmSize: 731 MB, VmRSS: 145 MB, VmData: 183 MB, maxrss: 0.1487 MB
Blob 10 finished, total: Wall: 0.20 s, CPU: 0.20 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 90 MB, VmData: 138 MB, maxrss: 0.09256 MB
Blob 9 finished, total: Wall: 0.22 s, CPU: 0.22 s, VmPeak: 669 MB, VmSize: 669 MB, VmRSS: 92 MB, VmData: 140 MB, maxrss: 0.093952 MB
Measuring 1 iterative sources
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (277.64865, 34.56094), pm (-1.2, -6.4), parallax 0.705 with NanoMaggies: g=17.2, r=16.5, z=16.3
Reset brightness to NanoMaggies: g=17.2, r=16.5, z=16.3
Blob 11 finished, total: Wall: 0.45 s, CPU: 0.45 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 91 MB, VmData: 139 MB, maxrss: 0.093276 MB
Blob 7 finished, total: Wall: 0.80 s, CPU: 0.80 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 91 MB, VmData: 139 MB, maxrss: 0.093584 MB
Blob 6 finished, total: Wall: 0.90 s, CPU: 0.89 s, VmPeak: 670 MB, VmSize: 670 MB, VmRSS: 93 MB, VmData: 141 MB, maxrss: 0.095724 MB
Measuring 3 iterative sources
Blob 5 finished, total: Wall: 0.99 s, CPU: 0.99 s, VmPeak: 813 MB, VmSize: 731 MB, VmRSS: 146 MB, VmData: 183 MB, maxrss: 0.149704 MB
Measuring 1 iterative sources
Blob 4 finished, total: Wall: 1.45 s, CPU: 1.44 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 91 MB, VmData: 138 MB, maxrss: 0.09276 MB
Blob 3 finished, total: Wall: 1.96 s, CPU: 1.96 s, VmPeak: 669 MB, VmSize: 669 MB, VmRSS: 93 MB, VmData: 140 MB, maxrss: 0.09516 MB
Measuring 3 iterative sources
Blob 2 finished, total: Wall: 4.12 s, CPU: 4.12 s, VmPeak: 672 MB, VmSize: 671 MB, VmRSS: 96 MB, VmData: 142 MB, maxrss: 0.09936 MB
Known galaxy.  Initial shape: EllipseWithPriors(0.25): log r_e=1.63628, ee1=0.283612, ee2=-0.220582
Measuring 9 iterative sources
Blob 1 finished, total: Wall: 19.82 s, CPU: 19.81 s, VmPeak: 716 MB, VmSize: 692 MB, VmRSS: 125 MB, VmData: 163 MB, maxrss: 0.151456 MB
Old catalog: 59
New catalog: 67
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/metrics/cus/blobs-custom-277655p34545.fits.gz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/metrics/cus/all-models-custom-277655p34545.fits
Stage fitblobs : Wall: 20.45 s, CPU: 0.60 s, VmPeak: 1074 MB, VmSize: 1074 MB, VmRSS: 302 MB, VmData: 327 MB, maxrss: 0.3095 MB
Stage fitblobs finished: 2020-08-05T20:32:06.584235
Resources for stage fitblobs : 31.209 s worker CPU, 31.350 s worker Wall, Wall: 20.456 s, Cores in use: 1.56, Total efficiency (on 32 cores): 4.9 %
Stage fitblobs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-fitblobs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-fitblobs.p
Running stage coadds
Running stage coadds at 2020-08-05T20:32:06.613596
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-ccds.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-nexp-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-depth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-galdepth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-psfsize-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-chi2-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-blobmodel-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-nexp-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-depth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-galdepth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-psfsize-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-chi2-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-blobmodel-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-nexp-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-depth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-galdepth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-psfsize-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-chi2-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-blobmodel-z.fits.fz
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-depth.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/tmp-legacysurvey-custom-277655p34545-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/tmp-legacysurvey-custom-277655p34545-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/tmp-legacysurvey-custom-277655p34545-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/tmp-legacysurvey-custom-277655p34545-resid.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-resid.jpg
Stage coadds : Wall: 3.64 s, CPU: 0.62 s, VmPeak: 1074 MB, VmSize: 1074 MB, VmRSS: 303 MB, VmData: 327 MB, maxrss: 0.309888 MB
Stage coadds finished: 2020-08-05T20:32:10.249452
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage coadds : 31.314 s worker CPU, 62.783 s worker Wall, Wall: 3.638 s, Cores in use: 8.78, Total efficiency (on 32 cores): 27.4 %
Stage coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-coadds.p
Running stage wise_forced
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage wise_forced at 2020-08-05T20:32:10.273822
Cut to 1 unWISE tiles
Reading WISE tile 2777p348 band 1
Reading WISE tile 2777p348 band 2
Reading WISE tile 2777p348 band 3
Reading WISE tile 2777p348 band 4
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w3-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w3-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w3-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.12 s, CPU: 0.30 s, VmPeak: 1008 MB, VmSize: 1008 MB, VmRSS: 146 MB, VmData: 184 MB, maxrss: 0.207524 MB
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w4-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w4-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w4-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.13 s, CPU: 0.34 s, VmPeak: 1013 MB, VmSize: 1013 MB, VmRSS: 153 MB, VmData: 190 MB, maxrss: 0.19206 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w1-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w1-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w1-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w1-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.10 s, CPU: 0.22 s, VmPeak: 1008 MB, VmSize: 1008 MB, VmRSS: 145 MB, VmData: 185 MB, maxrss: 0.206204 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w2-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w2-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w2-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w2-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.10 s, CPU: 0.21 s, VmPeak: 1003 MB, VmSize: 1003 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.202388 MB
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/tmp-legacysurvey-custom-277655p34545-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/tmp-legacysurvey-custom-277655p34545-wisemodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-wisemodel.jpg
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Stage wise_forced : Wall: 3.92 s, CPU: 1.31 s, VmPeak: 1074 MB, VmSize: 1074 MB, VmRSS: 303 MB, VmData: 327 MB, maxrss: 0.3104 MB
Stage wise_forced finished: 2020-08-05T20:32:14.195405
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage wise_forced : 4.317 s worker CPU, 7.095 s worker Wall, Wall: 3.924 s, Cores in use: 1.43, Total efficiency (on 32 cores): 4.5 %
Stage wise_forced finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-wise_forced.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-wise_forced.p
Running stage writecat
Running stage writecat at 2020-08-05T20:32:14.273078
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-maskbits.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/tractor-i/cus/tractor-custom-277655p34545.fits
Reading SFD maps...
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/tractor/cus/tractor-custom-277655p34545.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/tractor/cus/brick-custom-277655p34545.sha256sum
Stage writecat : Wall: 3.26 s, CPU: 2.48 s, VmPeak: 1207 MB, VmSize: 1079 MB, VmRSS: 305 MB, VmData: 327 MB, maxrss: 0.44328 MB
Stage writecat finished: 2020-08-05T20:32:17.528384
Resources for stage writecat : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 3.258 s, Cores in use: 0.76, Total efficiency (on 32 cores): 2.4 %
Stage writecat finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-writecat.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/277/PGC2050816/PGC2050816-largegalaxy-writecat.p
All done: 72.029 s worker CPU, 126.820 s worker Wall, Wall: 57.298 s, Cores in use: 1.58, Total efficiency (on 32 cores): 4.9 %
Finished galaxy PGC2050816 in 1.063 minutes.
Started working on galaxy PGC2050816 at Fri Aug  7 09:26:57 2020
Bands touching this brick, g r z
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 277.655119540526 34.54534652690173 --width 251 --height 251 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 

runbrick.py starting at 2020-08-07T09:26:58.250605
legacypipe git version: DR9.5.11-watershed-3-gd509eaaf
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '277.655119540526', '34.54534652690173', '--width', '251', '--height', '251', '--pixscale', '0.262', '--threads', '16', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 277.655119540526 34.54534652690173 --width 251 --height 251 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0

Parsed RA,Dec 277.655119540526 34.54534652690173
Starting process 17263 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB
Starting process 17264 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 17265 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 17266 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 17267 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059544 MB
Starting process 17268 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059556 MB
Starting process 17269 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059556 MB
Starting process 17270 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 17271 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 17272 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 17273 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 17274 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 17275 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 17277 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 17278 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 17276 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage fit_on_coadds
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-08-07T09:26:58.805253
Keeping 8 CCDs:
  k4m_160418_115925_ooi_zd_ls9-CCD1 z expnum 83130 exptime 115.104 propid 2016A-0453 seeing 1.55 MJD 57496.500 object MzLS_577361_z 
    mosaic/CP/V4.3/CP20160417/k4m_160418_115925_ooi_zd_ls9.fits.fz
  k4m_160415_120705_ooi_zd_ls9-CCD4 z expnum 82539 exptime 127.102 propid 2016A-0453 seeing 1.26 MJD 57493.505 object MzLS_576329_z 
    mosaic/CP/V4.3/CP20160414/k4m_160415_120705_ooi_zd_ls9.fits.fz
  k4m_170807_081211_ooi_zd_ls9-CCD4 z expnum 136168 exptime 151.104 propid 2016A-0453 seeing 1.09 MJD 57972.342 object MzLS_576329_z 
    mosaic/CP/V4.3/CP20170806/k4m_170807_081211_ooi_zd_ls9.fits.fz
  k4m_160417_120058_ooi_zd_ls9-CCD4 z expnum 82869 exptime 80.103 propid 2016A-0453 seeing 1.09 MJD 57495.501 object MzLS_575295_z 
    mosaic/CP/V4.3/CP20160416/k4m_160417_120058_ooi_zd_ls9.fits.fz
  ksb_180518_084042_ooi_r_ls9-CCD2 r expnum 82560157 exptime 160.286 propid NOTBASS seeing 1.99 MJD 58256.362 object 60893 
    90prime/CP/V2.3/CP20180517/ksb_180518_084042_ooi_r_ls9.fits.fz
  ksb_160628_070316_ooi_g_ls9-CCD2 g expnum 75670100 exptime 115.2 propid BASS seeing 1.62 MJD 57567.294 object 60893 
    90prime/CP/V2.3/CP20160627/ksb_160628_070316_ooi_g_ls9.fits.fz
  ksb_160602_085337_ooi_g_ls9-CCD1 g expnum 75410139 exptime 126.7 propid BASS seeing 1.35 MJD 57541.371 object 60732 
    90prime/CP/V2.3/CP20160601/ksb_160602_085337_ooi_g_ls9.fits.fz
  ksb_160510_110400_ooi_r_ls9-CCD1 r expnum 75180256 exptime 63.5 propid BASS seeing 1.13 MJD 57518.461 object 60732 
    90prime/CP/V2.3/CP20160509/ksb_160510_110400_ooi_r_ls9.fits.fz
mosaic get_fwhm: self.fwhm = 5.97
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 1669 1923 and amp range 0 2005 and image y 2950 3203 and amp range 2040 4078
Range in image: x 0 255 , y 0 254 (with image size 255 x 254)
dzp -0.005990375 -> scaling image by 0.994497853444703
mosaic get_fwhm: self.fwhm = 4.85
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 116 370 and amp range 0 2047 and image y 2386 2641 and amp range 2022 4060
Range in image: x 0 255 , y 0 256 (with image size 255 x 256)
dzp 0.0033053984 -> scaling image by 1.0030490232713904
mosaic get_fwhm: self.fwhm = 4.21
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 101 355 and amp range 0 2047 and image y 2364 2619 and amp range 2022 4060
Range in image: x 0 255 , y 0 256 (with image size 255 x 256)
dzp -0.016192675 -> scaling image by 0.9851966584456431
mosaic get_fwhm: self.fwhm = 4.21
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2815 3072 and amp range 2048 4095 and image y 1055 1308 and amp range 0 2021
Range in image: x 0 258 , y 0 254 (with image size 258 x 254)
dzp -0.0006399284 -> scaling image by 0.9994107778325068
Faked up EXPNUM 75180256
Faked up EXPNUM 75180256
Faked up EXPNUM 75180256
90prime get_fwhm: self.fwhm = 2.49
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 1676 1822 and amp range 0 2015 and image y 3741 3887 and amp range 2048 4095
Range in image: x 0 147 , y 0 147 (with image size 147 x 147)
dzp -0.00015519658 -> scaling image by 0.9998570688798908
Faked up EXPNUM 82560157
Faked up EXPNUM 82560157
Faked up EXPNUM 82560157
90prime get_fwhm: self.fwhm = 4.39
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 1449 1596 and amp range 0 2015 and image y 2078 2226 and amp range 2048 4095
Range in image: x 0 148 , y 0 149 (with image size 148 x 149)
dzp 0.0503617 -> scaling image by 1.047477444828492
Faked up EXPNUM 75410139
Faked up EXPNUM 75410139
Faked up EXPNUM 75410139
90prime get_fwhm: self.fwhm = 2.99
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 1676 1822 and amp range 0 2015 and image y 3726 3873 and amp range 2048 4095
Range in image: x 0 147 , y 0 148 (with image size 147 x 148)
dzp -0.0051191514 -> scaling image by 0.9952961850385206
Faked up EXPNUM 75670100
Faked up EXPNUM 75670100
Faked up EXPNUM 75670100
90prime get_fwhm: self.fwhm = 3.57
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 1447 1594 and amp range 0 2015 and image y 2099 2247 and amp range 2048 4095
Range in image: x 0 148 , y 0 149 (with image size 148 x 149)
dzp 0.01693853 -> scaling image by 1.015723292931903
Stage tims : Wall: 84.31 s, CPU: 0.94 s, VmPeak: 951 MB, VmSize: 951 MB, VmRSS: 172 MB, VmData: 206 MB, maxrss: 0.1763 MB
Stage tims finished: 2020-08-07T09:28:23.119329
Resources for stage tims : 3.661 s worker CPU, 190.018 s worker Wall, Wall: 84.316 s, Cores in use: 0.05, Total efficiency (on 16 cores): 0.3 %
Stage tims finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-tims.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-tims.p
Running stage refs
Running stage refs at 2020-08-07T09:28:26.470284
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-02850.fits
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-02723.fits
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-02594.fits
Matched 4 large galaxies to Gaia stars.
Matched 0 large galaxies to Tycho-2 stars.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/metrics/cus/reference-custom-277655p34545.fits
Stage refs : Wall: 10.34 s, CPU: 8.90 s, VmPeak: 1063 MB, VmSize: 1058 MB, VmRSS: 283 MB, VmData: 313 MB, maxrss: 0.28946 MB
Stage refs finished: 2020-08-07T09:28:36.809507
Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 10.341 s, Cores in use: 0.86, Total efficiency (on 16 cores): 5.4 %
Stage refs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-refs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-refs.p
Running stage outliers
Running stage outliers at 2020-08-07T09:28:36.945801
Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/metrics/cus/outlier-mask-custom-277655p34545.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/metrics/cus/outliers-pre-custom-277655p34545.jpg
Image ksb_160602_085337_ooi_g_ls9-CCD1 g : masking 41 positive outlier pixels and 0 negative outlier pixels
Image ksb_180518_084042_ooi_r_ls9-CCD2 r : masking 56 positive outlier pixels and 0 negative outlier pixels
Image k4m_160418_115925_ooi_zd_ls9-CCD1 z : masking 55 positive outlier pixels and 0 negative outlier pixels
Image k4m_160415_120705_ooi_zd_ls9-CCD4 z : masking 0 positive outlier pixels and 257 negative outlier pixels
Image k4m_170807_081211_ooi_zd_ls9-CCD4 z : masking 1135 positive outlier pixels and 0 negative outlier pixels
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/metrics/cus/outlier-mask-custom-277655p34545.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/metrics/cus/outliers-post-custom-277655p34545.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/metrics/cus/outliers-masked-pos-custom-277655p34545.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/metrics/cus/outliers-masked-neg-custom-277655p34545.jpg
Stage outliers : Wall: 0.85 s, CPU: 0.48 s, VmPeak: 1067 MB, VmSize: 1067 MB, VmRSS: 295 MB, VmData: 324 MB, maxrss: 0.302584 MB
Stage outliers finished: 2020-08-07T09:28:37.794672
Resources for stage outliers : 0.642 s worker CPU, 0.712 s worker Wall, Wall: 0.850 s, Cores in use: 1.32, Total efficiency (on 16 cores): 8.3 %
Stage outliers finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-outliers.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-outliers.p
Running stage halos
Running stage halos at 2020-08-07T09:28:37.862724
Stage halos : Wall: 0.24 s, CPU: 0.22 s, VmPeak: 1067 MB, VmSize: 1067 MB, VmRSS: 296 MB, VmData: 324 MB, maxrss: 0.302832 MB
Stage halos finished: 2020-08-07T09:28:38.098494
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.237 s, Cores in use: 0.96, Total efficiency (on 16 cores): 6.0 %
Stage halos finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-halos.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-halos.p
Running stage fit_on_coadds
Warning: Stellar halo subtraction is only implemented for DECam
Running stage fit_on_coadds at 2020-08-07T09:28:38.218238
PSF sigmas in coadd pixels: ['2.63', '2.20']
Using average PSF sigma 2.413185491002638
Gaussian PSF norm 0.11689726828942291 vs pixelized 0.09785106
average tim pixel scale / coadd scale: 1.7289383863880945
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-copsf-g.fits.fz
PSF sigmas in coadd pixels: ['3.23', '1.83']
Using average PSF sigma 2.5326713135314485
Gaussian PSF norm 0.11138231410713033 vs pixelized 0.08822469
average tim pixel scale / coadd scale: 1.7300093184822445
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-copsf-r.fits.fz
PSF sigmas in coadd pixels: ['2.52', '2.05', '1.78', '1.78']
Using average PSF sigma 2.028297773669782
Gaussian PSF norm 0.13907957472313667 vs pixelized 0.1122955
average tim pixel scale / coadd scale: 0.9909544014798491
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-copsf-z.fits.fz
Stage fit_on_coadds : Wall: 0.32 s, CPU: 0.13 s, VmPeak: 1067 MB, VmSize: 1067 MB, VmRSS: 297 MB, VmData: 324 MB, maxrss: 0.304296 MB
Stage fit_on_coadds finished: 2020-08-07T09:28:38.536505
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage fit_on_coadds : 0.132 s worker CPU, 0.132 s worker Wall, Wall: 0.319 s, Cores in use: 0.82, Total efficiency (on 16 cores): 5.1 %
Stage fit_on_coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-fit_on_coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-fit_on_coadds.p
Running stage srcs
Running stage srcs at 2020-08-07T09:28:38.564091
SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])]
Found 59 potential peaks
Of 59 potential peaks: 16 in veto map, 10 cut by saddle test, 0 cut by aper test, 33 kept
SED z : 33 new peaks
Found 43 potential peaks
Of 43 potential peaks: 34 in veto map, 8 cut by saddle test, 0 cut by aper test, 1 kept
SED r : 1 new peaks
Found 40 potential peaks
Of 40 potential peaks: 33 in veto map, 6 cut by saddle test, 0 cut by aper test, 1 kept
SED g : 1 new peaks
Found 62 potential peaks
Of 62 potential peaks: 52 in veto map, 10 cut by saddle test, 0 cut by aper test, 0 kept
SED Flat : 0 new peaks
Found 65 potential peaks
Of 65 potential peaks: 55 in veto map, 10 cut by saddle test, 0 cut by aper test, 0 kept
SED Red : 0 new peaks
Adding 35 new sources
Median coadd sky for g : -0.00010773729
Median coadd sky for r : -0.0001734971
Median coadd sky for z : -0.0003555459
Stage srcs : Wall: 0.55 s, CPU: 0.51 s, VmPeak: 1067 MB, VmSize: 1067 MB, VmRSS: 298 MB, VmData: 324 MB, maxrss: 0.304784 MB
Stage srcs finished: 2020-08-07T09:28:39.109709
Resources for stage srcs : 0.086 s worker CPU, 0.086 s worker Wall, Wall: 0.547 s, Cores in use: 1.09, Total efficiency (on 16 cores): 6.8 %
Stage srcs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-srcs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-srcs.p
Running stage fitblobs
Running stage fitblobs at 2020-08-07T09:28:39.227026
Blob 1 of 12, id: 1, sources: 28, size: 206x251, npix 24249, brick X: 37,243, Y: 0,251, one pixel: 68 0
Blob 2 of 12, id: 10, sources: 6, size: 92x61, npix 3266, brick X: 1,93, Y: 190,251, one pixel: 60 190
Blob 3 of 12, id: 3, sources: 3, size: 46x62, npix 1501, brick X: 24,70, Y: 104,166, one pixel: 38 104
Blob 4 of 12, id: 9, sources: 3, size: 45x49, npix 1492, brick X: 206,251, Y: 174,223, one pixel: 242 174
Blob 5 of 12, id: 7, sources: 2, size: 37x44, npix 1019, brick X: 0,37, Y: 159,203, one pixel: 1 159
Blob 6 of 12, id: 5, sources: 2, size: 36x37, npix 757, brick X: 209,245, Y: 113,150, one pixel: 220 113
Blob 7 of 12, id: 0, sources: 1, size: 27x37, npix 678, brick X: 10,37, Y: 0,37, one pixel: 21 0
Blob 8 of 12, id: 2, sources: 1, size: 27x25, npix 429, brick X: 24,51, Y: 50,75, one pixel: 33 50
Blob 9 of 12, id: 6, sources: 1, size: 26x24, npix 389, brick X: 212,238, Y: 154,178, one pixel: 222 154
Blob 10 of 12, id: 8, sources: 1, size: 22x22, npix 292, brick X: 68,90, Y: 167,189, one pixel: 76 167
Blob 11 of 12, id: 4, sources: 2, size: 16x24, npix 271, brick X: 235,251, Y: 106,130, one pixel: 244 106
Blob 12 of 12, id: 11, sources: 2, size: 22x15, npix 235, brick X: 179,201, Y: 236,251, one pixel: 189 236
Blob 10 finished, total: Wall: 0.19 s, CPU: 0.19 s, VmPeak: 670 MB, VmSize: 670 MB, VmRSS: 94 MB, VmData: 141 MB, maxrss: 0.09634 MB
Blob 8 finished, total: Wall: 0.26 s, CPU: 0.25 s, VmPeak: 669 MB, VmSize: 669 MB, VmRSS: 93 MB, VmData: 140 MB, maxrss: 0.095288 MB
Blob 9 finished, total: Wall: 0.29 s, CPU: 0.29 s, VmPeak: 804 MB, VmSize: 724 MB, VmRSS: 142 MB, VmData: 182 MB, maxrss: 0.145516 MB
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (277.64228, 34.54483), pm (-3.0, -0.9), parallax 0.452 with NanoMaggies: g=17, r=16, z=15.3
Reset brightness to NanoMaggies: g=19.9, r=19.6, z=19.5
Blob 11 finished, total: Wall: 0.49 s, CPU: 0.48 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 90 MB, VmData: 137 MB, maxrss: 0.092032 MB
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (277.64865, 34.56094), pm (-1.2, -6.4), parallax 0.705 with NanoMaggies: g=17.2, r=16.5, z=16.3
Reset brightness to NanoMaggies: g=17.2, r=16.5, z=16.3
Blob 12 finished, total: Wall: 0.52 s, CPU: 0.52 s, VmPeak: 809 MB, VmSize: 724 MB, VmRSS: 142 MB, VmData: 182 MB, maxrss: 0.145872 MB
Measuring 1 iterative sources
Blob 5 finished, total: Wall: 0.97 s, CPU: 0.96 s, VmPeak: 671 MB, VmSize: 671 MB, VmRSS: 95 MB, VmData: 142 MB, maxrss: 0.097052 MB
Blob 6 finished, total: Wall: 1.02 s, CPU: 1.02 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 90 MB, VmData: 138 MB, maxrss: 0.092256 MB
Blob 7 finished, total: Wall: 1.04 s, CPU: 1.04 s, VmPeak: 813 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.14604 MB
Measuring 2 iterative sources
Measuring 1 iterative sources
Blob 4 finished, total: Wall: 1.67 s, CPU: 1.66 s, VmPeak: 815 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.146696 MB
Blob 3 finished, total: Wall: 2.05 s, CPU: 2.05 s, VmPeak: 812 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.146256 MB
Measuring 3 iterative sources
Blob 2 finished, total: Wall: 3.95 s, CPU: 3.95 s, VmPeak: 673 MB, VmSize: 672 MB, VmRSS: 98 MB, VmData: 144 MB, maxrss: 0.101292 MB
Known galaxy.  Initial shape: EllipseWithPriors(0.25): log r_e=1.03795, ee1=0.235268, ee2=-0.262511
Measuring 9 iterative sources
Blob 1 finished, total: Wall: 16.92 s, CPU: 16.91 s, VmPeak: 814 MB, VmSize: 738 MB, VmRSS: 162 MB, VmData: 193 MB, maxrss: 0.165748 MB
Old catalog: 59
New catalog: 68
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/metrics/cus/blobs-custom-277655p34545.fits.gz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/metrics/cus/all-models-custom-277655p34545.fits
Stage fitblobs : Wall: 17.35 s, CPU: 0.36 s, VmPeak: 1067 MB, VmSize: 1067 MB, VmRSS: 300 MB, VmData: 326 MB, maxrss: 0.307248 MB
Stage fitblobs finished: 2020-08-07T09:28:56.579462
Resources for stage fitblobs : 29.527 s worker CPU, 29.665 s worker Wall, Wall: 17.354 s, Cores in use: 1.72, Total efficiency (on 16 cores): 10.8 %
Stage fitblobs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-fitblobs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-fitblobs.p
Running stage coadds
Running stage coadds at 2020-08-07T09:28:56.619385
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-ccds.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-nexp-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-depth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-galdepth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-psfsize-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-chi2-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-blobmodel-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-nexp-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-depth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-galdepth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-psfsize-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-chi2-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-blobmodel-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-nexp-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-depth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-galdepth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-psfsize-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-chi2-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-blobmodel-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-depth.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/tmp-legacysurvey-custom-277655p34545-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/tmp-legacysurvey-custom-277655p34545-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/tmp-legacysurvey-custom-277655p34545-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/tmp-legacysurvey-custom-277655p34545-resid.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-resid.jpg
Stage coadds : Wall: 18.57 s, CPU: 0.58 s, VmPeak: 1067 MB, VmSize: 1067 MB, VmRSS: 300 MB, VmData: 326 MB, maxrss: 0.307572 MB
Stage coadds finished: 2020-08-07T09:29:15.193382
Resources for stage coadds : 10.904 s worker CPU, 11.401 s worker Wall, Wall: 18.575 s, Cores in use: 0.62, Total efficiency (on 16 cores): 3.9 %
Stage coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-coadds.p
Running stage wise_forced
Running stage wise_forced at 2020-08-07T09:29:15.232033
Cut to 1 unWISE tiles
Reading WISE tile 2777p348 band 1
Reading WISE tile 2777p348 band 2
Reading WISE tile 2777p348 band 3
Reading WISE tile 2777p348 band 4
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w1-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w1-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w1-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w1-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.10 s, CPU: 0.25 s, VmPeak: 998 MB, VmSize: 998 MB, VmRSS: 137 MB, VmData: 184 MB, maxrss: 0.195924 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w2-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w2-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w2-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w2-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.10 s, CPU: 0.23 s, VmPeak: 1000 MB, VmSize: 1000 MB, VmRSS: 143 MB, VmData: 186 MB, maxrss: 0.202796 MB
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w3-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w3-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w3-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.10 s, CPU: 0.27 s, VmPeak: 1011 MB, VmSize: 1011 MB, VmRSS: 156 MB, VmData: 190 MB, maxrss: 0.218344 MB
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w4-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w4-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/277/2777p348/unwise-2777p348-w4-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.12 s, CPU: 0.33 s, VmPeak: 1007 MB, VmSize: 1007 MB, VmRSS: 149 MB, VmData: 189 MB, maxrss: 0.1923 MB
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-image-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-model-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-invvar-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/tmp-legacysurvey-custom-277655p34545-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/tmp-legacysurvey-custom-277655p34545-wisemodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-wisemodel.jpg
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Stage wise_forced : Wall: 16.25 s, CPU: 0.61 s, VmPeak: 1067 MB, VmSize: 1067 MB, VmRSS: 301 MB, VmData: 326 MB, maxrss: 0.308084 MB
Stage wise_forced finished: 2020-08-07T09:29:31.479967
Resources for stage wise_forced : 3.889 s worker CPU, 32.701 s worker Wall, Wall: 16.249 s, Cores in use: 0.28, Total efficiency (on 16 cores): 1.7 %
Stage wise_forced finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-wise_forced.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-wise_forced.p
Running stage writecat
Running stage writecat at 2020-08-07T09:29:31.518714
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/coadd/cus/custom-277655p34545/legacysurvey-custom-277655p34545-maskbits.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/tractor-i/cus/tractor-custom-277655p34545.fits
Reading SFD maps...
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/tractor/cus/tractor-custom-277655p34545.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/tractor/cus/brick-custom-277655p34545.sha256sum
Stage writecat : Wall: 5.92 s, CPU: 0.67 s, VmPeak: 1200 MB, VmSize: 1072 MB, VmRSS: 303 MB, VmData: 326 MB, maxrss: 0.440968 MB
Stage writecat finished: 2020-08-07T09:29:37.440784
Resources for stage writecat : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 5.924 s, Cores in use: 0.11, Total efficiency (on 16 cores): 0.7 %
Stage writecat finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-writecat.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/277/PGC2050816/PGC2050816-largegalaxy-writecat.p
All done: 48.845 s worker CPU, 264.720 s worker Wall, Wall: 163.031 s, Cores in use: 0.38, Total efficiency (on 16 cores): 2.4 %
Finished galaxy PGC2050816 in 2.854 minutes.