Started working on galaxy PGC2403824 at Wed Aug  5 20:30:22 2020
Bands touching this brick, g r z
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 272.1239655 51.9695091 --width 275 --height 275 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 

runbrick.py starting at 2020-08-05T20:30:23.530898
legacypipe git version: DR9.5.10-12-gf22e529f
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '272.1239655', '51.9695091', '--width', '275', '--height', '275', '--pixscale', '0.262', '--threads', '16', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-%%(stage)s.p', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 272.1239655 51.9695091 --width 275 --height 275 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0

Parsed RA,Dec 272.1239655 51.9695091
Starting process 33397 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 33398 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 33399 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 33400 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 33401 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059544 MB
Starting process 33402 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059556 MB
Starting process 33403 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059556 MB
Starting process 33404 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 33405 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 33406 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 33407 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 33408 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 33410 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 33411 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 33409 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 33412 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage fit_on_coadds
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-08-05T20:30:24.472815
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Keeping 9 CCDs:
  k4m_170901_054117_ooi_zd_ls9-CCD2 z expnum 137611 exptime 144.104 propid 2016A-0453 seeing 0.83 MJD 57997.237 object MzLS_656661_z 
    mosaic/CP/V4.3/CP20170831/k4m_170901_054117_ooi_zd_ls9.fits.fz
  k4m_160527_092207_ooi_zd_ls9-CCD4 z expnum 91872 exptime 80.102 propid 2016A-0453 seeing 0.78 MJD 57535.390 object MzLS_657438_z 
    mosaic/CP/V4.3/CP20160526/k4m_160527_092207_ooi_zd_ls9.fits.fz
  k4m_160526_091422_ooi_zd_ls9-CCD3 z expnum 91634 exptime 140.105 propid 2016A-0453 seeing 1.42 MJD 57534.385 object MzLS_658211_z 
    mosaic/CP/V4.3/CP20160525/k4m_160526_091422_ooi_zd_ls9.fits.fz
  ksb_170405_112630_ooi_r_ls9-CCD2 r expnum 78480176 exptime 59.967 propid BASS seeing 1.11 MJD 57848.477 object 29451 
    90prime/CP/V2.3/CP20170404/ksb_170405_112630_ooi_r_ls9.fits.fz
  ksb_170527_090558_ooi_g_ls9-CCD2 g expnum 79000142 exptime 114.3 propid 2015A-0801 seeing 1.44 MJD 57900.379 object 29451 
    90prime/CP/V2.3/CP20170526/ksb_170527_090558_ooi_g_ls9.fits.fz
  ksb_160707_091929_ooi_g_ls9-CCD2 g expnum 75760094 exptime 229.0 propid BASS seeing 2.13 MJD 57576.389 object 29452 
    90prime/CP/V2.3/CP20160706/ksb_160707_091929_ooi_g_ls9.fits.fz
  ksb_160712_064128_ooi_r_ls9-CCD2 r expnum 75810082 exptime 138.0 propid BASS seeing 1.64 MJD 57581.279 object 29452 
    90prime/CP/V2.3/CP20160711/ksb_160712_064128_ooi_r_ls9.fits.fz
  ksb_170624_062822_ooi_r_ls9-CCD3 r expnum 79280073 exptime 76.476 propid BASS seeing 1.13 MJD 57928.270 object 29463 
    90prime/CP/V2.3/CP20170623/ksb_170624_062822_ooi_r_ls9.fits.fz
  ksb_170618_061845_ooi_g_ls9-CCD3 g expnum 79220074 exptime 121.071 propid BASS seeing 1.42 MJD 57922.263 object 29463 
    90prime/CP/V2.3/CP20170617/ksb_170618_061845_ooi_g_ls9.fits.fz
Faked up EXPNUM 79280073
Faked up EXPNUM 79280073
Faked up EXPNUM 79280073
90prime get_fwhm: self.fwhm = 2.49
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2294 2454 and amp range 2016 4031 and image y 2710 2870 and amp range 2048 4095
Range in image: x 0 161 , y 0 161 (with image size 161 x 161)
dzp -0.035697103 -> scaling image by 0.9676563679910417
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 79220074
Faked up EXPNUM 79220074
Faked up EXPNUM 79220074
90prime get_fwhm: self.fwhm = 3.13
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2302 2462 and amp range 2016 4031 and image y 2712 2872 and amp range 2048 4095
Range in image: x 0 161 , y 0 161 (with image size 161 x 161)
dzp -0.02530836 -> scaling image by 0.9769597156943128
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 3.21
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3157 3443 and amp range 2048 4095 and image y 632 918 and amp range 0 2039
Range in image: x 0 287 , y 0 287 (with image size 287 x 287)
dzp -0.012875672 -> scaling image by 0.9882111074411305
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 3.01
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 458 735 and amp range 0 2047 and image y 1753 2031 and amp range 0 2021
Range in image: x 0 278 , y 0 269 (with image size 278 x 279)
dzp 0.005139143 -> scaling image by 1.0047445456970268
Found overlap: image x 458 735 and amp range 0 2047 and image y 1753 2031 and amp range 2022 4060
Range in image: x 0 278 , y 269 279 (with image size 278 x 279)
dzp 0.0033053984 -> scaling image by 1.0030490232713904
Faked up EXPNUM 79000142
Faked up EXPNUM 79000142
Faked up EXPNUM 79000142
90prime get_fwhm: self.fwhm = 3.17
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 52 214 and amp range 0 2015 and image y 1149 1311 and amp range 0 2047
Range in image: x 0 163 , y 0 163 (with image size 163 x 163)
dzp 0.0003063402 -> scaling image by 1.0002821895558327
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 75760094
Faked up EXPNUM 75760094
Faked up EXPNUM 75760094
90prime get_fwhm: self.fwhm = 4.71
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2063 2223 and amp range 2016 4031 and image y 3127 3288 and amp range 2048 4095
Range in image: x 0 161 , y 0 162 (with image size 161 x 162)
dzp 0.0026774607 -> scaling image by 1.002469075566243
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 78480176
Faked up EXPNUM 78480176
Faked up EXPNUM 78480176
90prime get_fwhm: self.fwhm = 2.45
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 39 200 and amp range 0 2015 and image y 1155 1317 and amp range 0 2047
Range in image: x 0 162 , y 0 163 (with image size 162 x 163)
dzp 0.005335051 -> scaling image by 1.0049258559887364
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 5.47
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2029 2308 and amp range 2006 4053 and image y 3233 3519 and amp range 2040 4078
Range in image: x 0 280 , y 0 287 (with image size 280 x 287)
dzp -0.0026360443 -> scaling image by 0.99757505842151
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 75810082
Faked up EXPNUM 75810082
Faked up EXPNUM 75810082
90prime get_fwhm: self.fwhm = 3.63
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2062 2223 and amp range 2016 4031 and image y 3141 3302 and amp range 2048 4095
Range in image: x 0 162 , y 0 162 (with image size 162 x 162)
dzp 0.0002007609 -> scaling image by 1.0001849247187462
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Stage tims : Wall: 6.81 s, CPU: 1.06 s, VmPeak: 957 MB, VmSize: 957 MB, VmRSS: 175 MB, VmData: 207 MB, maxrss: 0.179116 MB
Stage tims finished: 2020-08-05T20:30:31.286047
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage tims : 4.682 s worker CPU, 32.298 s worker Wall, Wall: 6.815 s, Cores in use: 0.84, Total efficiency (on 16 cores): 5.3 %
Stage tims finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-tims.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-tims.p
Running stage refs
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage refs at 2020-08-05T20:30:31.413696
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-01378.fits
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-01275.fits
Matched 4 large galaxies to Gaia stars.
Matched 0 large galaxies to Tycho-2 stars.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/metrics/cus/reference-custom-272123p51969.fits
Stage refs : Wall: 18.24 s, CPU: 4.22 s, VmPeak: 1005 MB, VmSize: 1000 MB, VmRSS: 220 MB, VmData: 250 MB, maxrss: 0.225132 MB
Stage refs finished: 2020-08-05T20:30:49.649481
Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 18.237 s, Cores in use: 0.23, Total efficiency (on 16 cores): 1.4 %
Stage refs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-refs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-refs.p
Running stage outliers
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage outliers at 2020-08-05T20:30:49.672715
Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/metrics/cus/outlier-mask-custom-272123p51969.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/metrics/cus/outliers-pre-custom-272123p51969.jpg
Image ksb_170527_090558_ooi_g_ls9-CCD2 g : masking 85 positive outlier pixels and 0 negative outlier pixels
Image ksb_170624_062822_ooi_r_ls9-CCD3 r : masking 35 positive outlier pixels and 0 negative outlier pixels
Image k4m_160526_091422_ooi_zd_ls9-CCD3 z : masking 98 positive outlier pixels and 0 negative outlier pixels
Image k4m_170901_054117_ooi_zd_ls9-CCD2 z : masking 1338 positive outlier pixels and 0 negative outlier pixels
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/metrics/cus/outlier-mask-custom-272123p51969.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/metrics/cus/outliers-post-custom-272123p51969.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/metrics/cus/outliers-masked-pos-custom-272123p51969.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/metrics/cus/outliers-masked-neg-custom-272123p51969.jpg
Stage outliers : Wall: 1.72 s, CPU: 1.36 s, VmPeak: 1009 MB, VmSize: 1009 MB, VmRSS: 235 MB, VmData: 262 MB, maxrss: 0.240996 MB
Stage outliers finished: 2020-08-05T20:30:51.391411
Resources for stage outliers : 0.785 s worker CPU, 0.847 s worker Wall, Wall: 1.720 s, Cores in use: 1.25, Total efficiency (on 16 cores): 7.8 %
Stage outliers finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-outliers.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-outliers.p
Running stage halos
Running stage halos at 2020-08-05T20:30:51.414740
Stage halos : Wall: 0.72 s, CPU: 0.70 s, VmPeak: 1009 MB, VmSize: 1009 MB, VmRSS: 235 MB, VmData: 262 MB, maxrss: 0.24108 MB
Stage halos finished: 2020-08-05T20:30:52.132985
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.720 s, Cores in use: 0.97, Total efficiency (on 16 cores): 6.1 %
Stage halos finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-halos.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-halos.p
Running stage fit_on_coadds
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Running stage fit_on_coadds at 2020-08-05T20:30:52.294144
PSF sigmas in coadd pixels: ['2.33', '3.47', '2.30']
Using average PSF sigma 2.7007499559400343
Gaussian PSF norm 0.10445054017438318 vs pixelized 0.07524477
average tim pixel scale / coadd scale: 1.7293451846228869
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-copsf-g.fits.fz
PSF sigmas in coadd pixels: ['1.80', '2.67', '1.83']
Using average PSF sigma 2.102160112067544
Gaussian PSF norm 0.1341928191646775 vs pixelized 0.113868475
average tim pixel scale / coadd scale: 1.7293235900250212
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-copsf-r.fits.fz
PSF sigmas in coadd pixels: ['1.35', '1.27', '2.31']
Using average PSF sigma 1.6430887332092283
Gaussian PSF norm 0.17168567106105076 vs pixelized 0.14150712
average tim pixel scale / coadd scale: 0.9909119121090487
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-copsf-z.fits.fz
Stage fit_on_coadds : Wall: 0.18 s, CPU: 0.16 s, VmPeak: 1009 MB, VmSize: 1009 MB, VmRSS: 238 MB, VmData: 263 MB, maxrss: 0.243628 MB
Stage fit_on_coadds finished: 2020-08-05T20:30:52.471729
Resources for stage fit_on_coadds : 0.183 s worker CPU, 0.183 s worker Wall, Wall: 0.179 s, Cores in use: 1.90, Total efficiency (on 16 cores): 11.9 %
Stage fit_on_coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-fit_on_coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-fit_on_coadds.p
Running stage srcs
Running stage srcs at 2020-08-05T20:30:52.485608
SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])]
Found 41 potential peaks
Of 41 potential peaks: 9 in veto map, 6 cut by saddle test, 0 cut by aper test, 26 kept
SED z : 26 new peaks
Found 45 potential peaks
Of 45 potential peaks: 39 in veto map, 5 cut by saddle test, 0 cut by aper test, 1 kept
SED r : 1 new peaks
Found 30 potential peaks
Of 30 potential peaks: 25 in veto map, 4 cut by saddle test, 0 cut by aper test, 1 kept
SED g : 1 new peaks
Found 51 potential peaks
Of 51 potential peaks: 41 in veto map, 8 cut by saddle test, 0 cut by aper test, 2 kept
SED Flat : 2 new peaks
Found 69 potential peaks
Of 69 potential peaks: 60 in veto map, 9 cut by saddle test, 0 cut by aper test, 0 kept
SED Red : 0 new peaks
Adding 30 new sources
Median coadd sky for g : 7.758239e-05
Median coadd sky for r : 0.000172188
Median coadd sky for z : 0.00014046526
Stage srcs : Wall: 1.04 s, CPU: 0.99 s, VmPeak: 1009 MB, VmSize: 1009 MB, VmRSS: 238 MB, VmData: 263 MB, maxrss: 0.244148 MB
Stage srcs finished: 2020-08-05T20:30:53.529383
Resources for stage srcs : 0.080 s worker CPU, 0.080 s worker Wall, Wall: 1.045 s, Cores in use: 1.03, Total efficiency (on 16 cores): 6.4 %
Stage srcs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-srcs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-srcs.p
Running stage fitblobs
Running stage fitblobs at 2020-08-05T20:30:53.544250
Blob 1 of 12, id: 2, sources: 21, size: 203x253, npix 25446, brick X: 42,245, Y: 5,258, one pixel: 143 5
Blob 2 of 12, id: 4, sources: 2, size: 56x45, npix 1406, brick X: 160,216, Y: 33,78, one pixel: 176 33
Blob 3 of 12, id: 1, sources: 3, size: 53x48, npix 1128, brick X: 184,237, Y: 0,48, one pixel: 190 0
Blob 4 of 12, id: 9, sources: 1, size: 38x29, npix 713, brick X: 15,53, Y: 207,236, one pixel: 30 207
Blob 5 of 12, id: 3, sources: 2, size: 25x39, npix 676, brick X: 0,25, Y: 10,49, one pixel: 5 10
Blob 6 of 12, id: 0, sources: 1, size: 38x23, npix 532, brick X: 14,52, Y: 0,23, one pixel: 19 0
Blob 7 of 12, id: 8, sources: 2, size: 20x33, npix 492, brick X: 255,275, Y: 198,231, one pixel: 268 198
Blob 8 of 12, id: 11, sources: 3, size: 32x21, npix 462, brick X: 243,275, Y: 254,275, one pixel: 253 254
Blob 9 of 12, id: 6, sources: 1, size: 26x24, npix 377, brick X: 232,258, Y: 142,166, one pixel: 244 142
Blob 10 of 12, id: 5, sources: 1, size: 25x24, npix 375, brick X: 248,273, Y: 77,101, one pixel: 259 77
Blob 11 of 12, id: 7, sources: 1, size: 23x24, npix 349, brick X: 73,96, Y: 145,169, one pixel: 83 145
Blob 12 of 12, id: 10, sources: 1, size: 19x19, npix 217, brick X: 65,84, Y: 208,227, one pixel: 73 208
Blob 12 finished, total: Wall: 0.10 s, CPU: 0.08 s, VmPeak: 809 MB, VmSize: 731 MB, VmRSS: 147 MB, VmData: 183 MB, maxrss: 0.150916 MB
Blob 11 finished, total: Wall: 0.27 s, CPU: 0.24 s, VmPeak: 807 MB, VmSize: 731 MB, VmRSS: 147 MB, VmData: 183 MB, maxrss: 0.150652 MB
Blob 10 finished, total: Wall: 0.39 s, CPU: 0.38 s, VmPeak: 805 MB, VmSize: 731 MB, VmRSS: 147 MB, VmData: 183 MB, maxrss: 0.150652 MB
Blob 9 finished, total: Wall: 0.42 s, CPU: 0.42 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 92 MB, VmData: 139 MB, maxrss: 0.094568 MB
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (272.10748, 51.97514), pm (-0.1, -6.8), parallax 0.686 with NanoMaggies: g=20.8, r=19.3, z=18.1
Reset brightness to NanoMaggies: g=20.5, r=19.1, z=18.2
Blob 7 finished, total: Wall: 0.55 s, CPU: 0.54 s, VmPeak: 809 MB, VmSize: 732 MB, VmRSS: 148 MB, VmData: 183 MB, maxrss: 0.151372 MB
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (271.93958, 52.02776), pm (-0.5, 18.7), parallax 3.568 with NanoMaggies: g=8.35, r=7.58, z=7.24
Reset brightness to NanoMaggies: g=8.35, r=7.58, z=7.24
Blob 8 finished, total: Wall: 0.84 s, CPU: 0.84 s, VmPeak: 813 MB, VmSize: 732 MB, VmRSS: 147 MB, VmData: 183 MB, maxrss: 0.151012 MB
Blob 5 finished, total: Wall: 0.87 s, CPU: 0.85 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 93 MB, VmData: 140 MB, maxrss: 0.095628 MB
Measuring 1 iterative sources
Measuring 1 iterative sources
Blob 4 finished, total: Wall: 1.05 s, CPU: 1.03 s, VmPeak: 670 MB, VmSize: 670 MB, VmRSS: 96 MB, VmData: 141 MB, maxrss: 0.097952 MB
Blob 6 finished, total: Wall: 1.13 s, CPU: 1.11 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 92 MB, VmData: 138 MB, maxrss: 0.094576 MB
Measuring 2 iterative sources
Blob 3 finished, total: Wall: 1.48 s, CPU: 1.46 s, VmPeak: 805 MB, VmSize: 731 MB, VmRSS: 147 MB, VmData: 183 MB, maxrss: 0.150944 MB
Blob 2 finished, total: Wall: 1.56 s, CPU: 1.54 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 93 MB, VmData: 139 MB, maxrss: 0.094992 MB
Known galaxy.  Initial shape: EllipseWithPriors(0.25): log r_e=1.19653, ee1=0.120464, ee2=-0.397401
Measuring 14 iterative sources
Blob 1 finished, total: Wall: 23.08 s, CPU: 23.06 s, VmPeak: 709 MB, VmSize: 699 MB, VmRSS: 131 MB, VmData: 170 MB, maxrss: 0.14098 MB
Old catalog: 45
New catalog: 53
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/metrics/cus/blobs-custom-272123p51969.fits.gz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/metrics/cus/all-models-custom-272123p51969.fits
Stage fitblobs : Wall: 24.06 s, CPU: 0.86 s, VmPeak: 1009 MB, VmSize: 1009 MB, VmRSS: 239 MB, VmData: 263 MB, maxrss: 0.24456 MB
Stage fitblobs finished: 2020-08-05T20:31:17.605834
Resources for stage fitblobs : 31.736 s worker CPU, 32.417 s worker Wall, Wall: 24.063 s, Cores in use: 1.35, Total efficiency (on 16 cores): 8.5 %
Stage fitblobs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-fitblobs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-fitblobs.p
Running stage coadds
Running stage coadds at 2020-08-05T20:31:17.628672
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-ccds.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-nexp-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-depth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-galdepth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-psfsize-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-chi2-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-blobmodel-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-nexp-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-depth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-galdepth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-psfsize-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-chi2-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-blobmodel-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-nexp-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-depth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-galdepth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-psfsize-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-chi2-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-blobmodel-z.fits.fz
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-depth.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/tmp-legacysurvey-custom-272123p51969-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/tmp-legacysurvey-custom-272123p51969-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/tmp-legacysurvey-custom-272123p51969-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/tmp-legacysurvey-custom-272123p51969-resid.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-resid.jpg
Stage coadds : Wall: 4.93 s, CPU: 2.24 s, VmPeak: 1009 MB, VmSize: 1009 MB, VmRSS: 239 MB, VmData: 263 MB, maxrss: 0.244948 MB
Stage coadds finished: 2020-08-05T20:31:22.559121
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage coadds : 9.952 s worker CPU, 16.045 s worker Wall, Wall: 4.932 s, Cores in use: 2.47, Total efficiency (on 16 cores): 15.5 %
Stage coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-coadds.p
Running stage wise_forced
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage wise_forced at 2020-08-05T20:31:22.577677
Cut to 1 unWISE tiles
Reading WISE tile 2717p514 band 2
Reading WISE tile 2717p514 band 3
Reading WISE tile 2717p514 band 4
Reading WISE tile 2717p514 band 1
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w4-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w4-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w4-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.09 s, CPU: 0.09 s, VmPeak: 809 MB, VmSize: 809 MB, VmRSS: 151 MB, VmData: 177 MB, maxrss: 0.19202 MB
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w3-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w3-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w3-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.07 s, CPU: 0.07 s, VmPeak: 807 MB, VmSize: 804 MB, VmRSS: 145 MB, VmData: 172 MB, maxrss: 0.209728 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w1-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w1-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w1-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w1-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.07 s, CPU: 0.07 s, VmPeak: 808 MB, VmSize: 805 MB, VmRSS: 145 MB, VmData: 173 MB, maxrss: 0.208308 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w2-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w2-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w2-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w2-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.07 s, CPU: 0.07 s, VmPeak: 805 MB, VmSize: 803 MB, VmRSS: 142 MB, VmData: 171 MB, maxrss: 0.204844 MB
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/tmp-legacysurvey-custom-272123p51969-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/tmp-legacysurvey-custom-272123p51969-wisemodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-wisemodel.jpg
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Stage wise_forced : Wall: 4.66 s, CPU: 1.60 s, VmPeak: 1009 MB, VmSize: 1009 MB, VmRSS: 240 MB, VmData: 263 MB, maxrss: 0.24546 MB
Stage wise_forced finished: 2020-08-05T20:31:27.237381
Resources for stage wise_forced : 3.098 s worker CPU, 6.590 s worker Wall, Wall: 4.661 s, Cores in use: 1.01, Total efficiency (on 16 cores): 6.3 %
Stage wise_forced finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-wise_forced.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-wise_forced.p
Running stage writecat
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage writecat at 2020-08-05T20:31:27.254184
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-maskbits.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/tractor-i/cus/tractor-custom-272123p51969.fits
Reading SFD maps...
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/tractor/cus/tractor-custom-272123p51969.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/tractor/cus/brick-custom-272123p51969.sha256sum
Stage writecat : Wall: 2.34 s, CPU: 1.38 s, VmPeak: 1142 MB, VmSize: 1014 MB, VmRSS: 242 MB, VmData: 264 MB, maxrss: 0.378388 MB
Stage writecat finished: 2020-08-05T20:31:29.590691
Resources for stage writecat : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 2.338 s, Cores in use: 0.59, Total efficiency (on 16 cores): 3.7 %
Stage writecat finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-writecat.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-writecat.p
All done: 50.521 s worker CPU, 88.465 s worker Wall, Wall: 65.212 s, Cores in use: 1.00, Total efficiency (on 16 cores): 6.3 %
Finished galaxy PGC2403824 in 1.133 minutes.
Started working on galaxy PGC2403824 at Wed Aug  5 22:09:27 2020
Bands touching this brick, g r z
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 272.1239655 51.9695091 --width 275 --height 275 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 

runbrick.py starting at 2020-08-05T22:09:28.959089
legacypipe git version: DR9.5.10-12-gf22e529f
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '272.1239655', '51.9695091', '--width', '275', '--height', '275', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-%%(stage)s.p', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 272.1239655 51.9695091 --width 275 --height 275 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0

Parsed RA,Dec 272.1239655 51.9695091
Starting process 53475 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 53476 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 53477 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 53478 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 53479 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB
Starting process 53480 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 53481 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 53482 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 53483 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 53484 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 53485 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 53486 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 53487 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 53488 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 53489 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 53490 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 53491 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059276 MB
Starting process 53492 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05928 MB
Starting process 53493 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB
Starting process 53494 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB
Starting process 53495 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB
Starting process 53496 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB
Starting process 53497 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB
Starting process 53498 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB
Starting process 53499 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Starting process 53500 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Starting process 53501 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Starting process 53502 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059308 MB
Starting process 53503 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB
Starting process 53504 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB
Starting process 53505 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059312 MB
Starting process 53506 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059316 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage fit_on_coadds
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-08-05T22:09:29.493083
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Keeping 9 CCDs:
  k4m_170901_054117_ooi_zd_ls9-CCD2 z expnum 137611 exptime 144.104 propid 2016A-0453 seeing 0.83 MJD 57997.237 object MzLS_656661_z 
    mosaic/CP/V4.3/CP20170831/k4m_170901_054117_ooi_zd_ls9.fits.fz
  k4m_160527_092207_ooi_zd_ls9-CCD4 z expnum 91872 exptime 80.102 propid 2016A-0453 seeing 0.78 MJD 57535.390 object MzLS_657438_z 
    mosaic/CP/V4.3/CP20160526/k4m_160527_092207_ooi_zd_ls9.fits.fz
  k4m_160526_091422_ooi_zd_ls9-CCD3 z expnum 91634 exptime 140.105 propid 2016A-0453 seeing 1.42 MJD 57534.385 object MzLS_658211_z 
    mosaic/CP/V4.3/CP20160525/k4m_160526_091422_ooi_zd_ls9.fits.fz
  ksb_170405_112630_ooi_r_ls9-CCD2 r expnum 78480176 exptime 59.967 propid BASS seeing 1.11 MJD 57848.477 object 29451 
    90prime/CP/V2.3/CP20170404/ksb_170405_112630_ooi_r_ls9.fits.fz
  ksb_170527_090558_ooi_g_ls9-CCD2 g expnum 79000142 exptime 114.3 propid 2015A-0801 seeing 1.44 MJD 57900.379 object 29451 
    90prime/CP/V2.3/CP20170526/ksb_170527_090558_ooi_g_ls9.fits.fz
  ksb_160707_091929_ooi_g_ls9-CCD2 g expnum 75760094 exptime 229.0 propid BASS seeing 2.13 MJD 57576.389 object 29452 
    90prime/CP/V2.3/CP20160706/ksb_160707_091929_ooi_g_ls9.fits.fz
  ksb_160712_064128_ooi_r_ls9-CCD2 r expnum 75810082 exptime 138.0 propid BASS seeing 1.64 MJD 57581.279 object 29452 
    90prime/CP/V2.3/CP20160711/ksb_160712_064128_ooi_r_ls9.fits.fz
  ksb_170624_062822_ooi_r_ls9-CCD3 r expnum 79280073 exptime 76.476 propid BASS seeing 1.13 MJD 57928.270 object 29463 
    90prime/CP/V2.3/CP20170623/ksb_170624_062822_ooi_r_ls9.fits.fz
  ksb_170618_061845_ooi_g_ls9-CCD3 g expnum 79220074 exptime 121.071 propid BASS seeing 1.42 MJD 57922.263 object 29463 
    90prime/CP/V2.3/CP20170617/ksb_170618_061845_ooi_g_ls9.fits.fz
Faked up EXPNUM 79280073
Faked up EXPNUM 79280073
Faked up EXPNUM 79280073
90prime get_fwhm: self.fwhm = 2.49
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2294 2454 and amp range 2016 4031 and image y 2710 2870 and amp range 2048 4095
Range in image: x 0 161 , y 0 161 (with image size 161 x 161)
dzp -0.035697103 -> scaling image by 0.9676563679910417
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 79220074
Faked up EXPNUM 79220074
Faked up EXPNUM 79220074
90prime get_fwhm: self.fwhm = 3.13
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2302 2462 and amp range 2016 4031 and image y 2712 2872 and amp range 2048 4095
Range in image: x 0 161 , y 0 161 (with image size 161 x 161)
dzp -0.02530836 -> scaling image by 0.9769597156943128
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 75760094
Faked up EXPNUM 75760094
Faked up EXPNUM 75760094
90prime get_fwhm: self.fwhm = 4.71
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2063 2223 and amp range 2016 4031 and image y 3127 3288 and amp range 2048 4095
Range in image: x 0 161 , y 0 162 (with image size 161 x 162)
dzp 0.0026774607 -> scaling image by 1.002469075566243
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 78480176
Faked up EXPNUM 78480176
Faked up EXPNUM 78480176
90prime get_fwhm: self.fwhm = 2.45
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 39 200 and amp range 0 2015 and image y 1155 1317 and amp range 0 2047
Range in image: x 0 162 , y 0 163 (with image size 162 x 163)
dzp 0.005335051 -> scaling image by 1.0049258559887364
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 5.47
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2029 2308 and amp range 2006 4053 and image y 3233 3519 and amp range 2040 4078
Range in image: x 0 280 , y 0 287 (with image size 280 x 287)
dzp -0.0026360443 -> scaling image by 0.99757505842151
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 3.21
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3157 3443 and amp range 2048 4095 and image y 632 918 and amp range 0 2039
Range in image: x 0 287 , y 0 287 (with image size 287 x 287)
dzp -0.012875672 -> scaling image by 0.9882111074411305
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 3.01
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 458 735 and amp range 0 2047 and image y 1753 2031 and amp range 0 2021
Range in image: x 0 278 , y 0 269 (with image size 278 x 279)
dzp 0.005139143 -> scaling image by 1.0047445456970268
Found overlap: image x 458 735 and amp range 0 2047 and image y 1753 2031 and amp range 2022 4060
Range in image: x 0 278 , y 269 279 (with image size 278 x 279)
dzp 0.0033053984 -> scaling image by 1.0030490232713904
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 75810082
Faked up EXPNUM 75810082
Faked up EXPNUM 75810082
90prime get_fwhm: self.fwhm = 3.63
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2062 2223 and amp range 2016 4031 and image y 3141 3302 and amp range 2048 4095
Range in image: x 0 162 , y 0 162 (with image size 162 x 162)
dzp 0.0002007609 -> scaling image by 1.0001849247187462
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 79000142
Faked up EXPNUM 79000142
Faked up EXPNUM 79000142
90prime get_fwhm: self.fwhm = 3.17
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 52 214 and amp range 0 2015 and image y 1149 1311 and amp range 0 2047
Range in image: x 0 163 , y 0 163 (with image size 163 x 163)
dzp 0.0003063402 -> scaling image by 1.0002821895558327
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Stage tims : Wall: 5.61 s, CPU: 1.03 s, VmPeak: 957 MB, VmSize: 957 MB, VmRSS: 175 MB, VmData: 207 MB, maxrss: 0.179044 MB
Stage tims finished: 2020-08-05T22:09:35.105746
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage tims : 5.587 s worker CPU, 25.292 s worker Wall, Wall: 5.615 s, Cores in use: 1.18, Total efficiency (on 32 cores): 3.7 %
Stage tims finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-tims.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-tims.p
Running stage refs
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage refs at 2020-08-05T22:09:39.043834
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-01378.fits
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-01275.fits
Matched 4 large galaxies to Gaia stars.
Matched 0 large galaxies to Tycho-2 stars.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/metrics/cus/reference-custom-272123p51969.fits
Stage refs : Wall: 13.97 s, CPU: 4.14 s, VmPeak: 1006 MB, VmSize: 1000 MB, VmRSS: 220 MB, VmData: 250 MB, maxrss: 0.224892 MB
Stage refs finished: 2020-08-05T22:09:53.010450
Resources for stage refs : 0.000 s worker CPU, 0.001 s worker Wall, Wall: 13.969 s, Cores in use: 0.30, Total efficiency (on 32 cores): 0.9 %
Stage refs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-refs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-refs.p
Running stage outliers
Running stage outliers at 2020-08-05T22:09:53.066764
Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/metrics/cus/outlier-mask-custom-272123p51969.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/metrics/cus/outliers-pre-custom-272123p51969.jpg
Image ksb_170527_090558_ooi_g_ls9-CCD2 g : masking 85 positive outlier pixels and 0 negative outlier pixels
Image ksb_170624_062822_ooi_r_ls9-CCD3 r : masking 35 positive outlier pixels and 0 negative outlier pixels
Image k4m_170901_054117_ooi_zd_ls9-CCD2 z : masking 1338 positive outlier pixels and 0 negative outlier pixels
Image k4m_160526_091422_ooi_zd_ls9-CCD3 z : masking 98 positive outlier pixels and 0 negative outlier pixels
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/metrics/cus/outlier-mask-custom-272123p51969.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/metrics/cus/outliers-post-custom-272123p51969.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/metrics/cus/outliers-masked-pos-custom-272123p51969.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/metrics/cus/outliers-masked-neg-custom-272123p51969.jpg
Stage outliers : Wall: 1.46 s, CPU: 0.95 s, VmPeak: 1009 MB, VmSize: 1009 MB, VmRSS: 236 MB, VmData: 264 MB, maxrss: 0.242104 MB
Stage outliers finished: 2020-08-05T22:09:54.527866
Resources for stage outliers : 0.958 s worker CPU, 1.356 s worker Wall, Wall: 1.463 s, Cores in use: 1.31, Total efficiency (on 32 cores): 4.1 %
Stage outliers finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-outliers.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-outliers.p
Running stage halos
Running stage halos at 2020-08-05T22:09:54.725956
Stage halos : Wall: 0.36 s, CPU: 0.35 s, VmPeak: 1009 MB, VmSize: 1009 MB, VmRSS: 236 MB, VmData: 264 MB, maxrss: 0.242164 MB
Stage halos finished: 2020-08-05T22:09:55.081989
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.358 s, Cores in use: 0.97, Total efficiency (on 32 cores): 3.0 %
Stage halos finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-halos.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-halos.p
Running stage fit_on_coadds
Warning: Stellar halo subtraction is only implemented for DECam
Running stage fit_on_coadds at 2020-08-05T22:09:55.105234
PSF sigmas in coadd pixels: ['2.33', '3.47', '2.30']
Using average PSF sigma 2.7007499559400343
Gaussian PSF norm 0.10445054017438318 vs pixelized 0.07524477
average tim pixel scale / coadd scale: 1.7293451846228869
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-copsf-g.fits.fz
PSF sigmas in coadd pixels: ['1.80', '2.67', '1.83']
Using average PSF sigma 2.102160112067544
Gaussian PSF norm 0.1341928191646775 vs pixelized 0.113868475
average tim pixel scale / coadd scale: 1.7293235900250212
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-copsf-r.fits.fz
PSF sigmas in coadd pixels: ['1.35', '1.27', '2.31']
Using average PSF sigma 1.6430887332092283
Gaussian PSF norm 0.17168567106105076 vs pixelized 0.14150712
average tim pixel scale / coadd scale: 0.9909119121090487
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-copsf-z.fits.fz
Stage fit_on_coadds : Wall: 0.70 s, CPU: 0.16 s, VmPeak: 1009 MB, VmSize: 1009 MB, VmRSS: 238 MB, VmData: 264 MB, maxrss: 0.243724 MB
Stage fit_on_coadds finished: 2020-08-05T22:09:55.801497
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage fit_on_coadds : 0.178 s worker CPU, 0.178 s worker Wall, Wall: 0.698 s, Cores in use: 0.48, Total efficiency (on 32 cores): 1.5 %
Stage fit_on_coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-fit_on_coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-fit_on_coadds.p
Running stage srcs
Running stage srcs at 2020-08-05T22:09:55.813598
SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])]
Found 41 potential peaks
Of 41 potential peaks: 9 in veto map, 6 cut by saddle test, 0 cut by aper test, 26 kept
SED z : 26 new peaks
Found 45 potential peaks
Of 45 potential peaks: 39 in veto map, 5 cut by saddle test, 0 cut by aper test, 1 kept
SED r : 1 new peaks
Found 30 potential peaks
Of 30 potential peaks: 25 in veto map, 4 cut by saddle test, 0 cut by aper test, 1 kept
SED g : 1 new peaks
Found 51 potential peaks
Of 51 potential peaks: 41 in veto map, 8 cut by saddle test, 0 cut by aper test, 2 kept
SED Flat : 2 new peaks
Found 69 potential peaks
Of 69 potential peaks: 60 in veto map, 9 cut by saddle test, 0 cut by aper test, 0 kept
SED Red : 0 new peaks
Adding 30 new sources
Median coadd sky for g : 7.758239e-05
Median coadd sky for r : 0.000172188
Median coadd sky for z : 0.0001404654
Stage srcs : Wall: 0.62 s, CPU: 0.59 s, VmPeak: 1009 MB, VmSize: 1009 MB, VmRSS: 239 MB, VmData: 264 MB, maxrss: 0.244232 MB
Stage srcs finished: 2020-08-05T22:09:56.435775
Resources for stage srcs : 0.085 s worker CPU, 0.085 s worker Wall, Wall: 0.624 s, Cores in use: 1.08, Total efficiency (on 32 cores): 3.4 %
Stage srcs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-srcs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-srcs.p
Running stage fitblobs
Running stage fitblobs at 2020-08-05T22:09:56.450697
Blob 1 of 12, id: 2, sources: 21, size: 203x253, npix 25446, brick X: 42,245, Y: 5,258, one pixel: 143 5
Blob 2 of 12, id: 4, sources: 2, size: 56x45, npix 1406, brick X: 160,216, Y: 33,78, one pixel: 176 33
Blob 3 of 12, id: 1, sources: 3, size: 53x48, npix 1128, brick X: 184,237, Y: 0,48, one pixel: 190 0
Blob 4 of 12, id: 9, sources: 1, size: 38x29, npix 713, brick X: 15,53, Y: 207,236, one pixel: 30 207
Blob 5 of 12, id: 3, sources: 2, size: 25x39, npix 676, brick X: 0,25, Y: 10,49, one pixel: 5 10
Blob 6 of 12, id: 0, sources: 1, size: 38x23, npix 532, brick X: 14,52, Y: 0,23, one pixel: 19 0
Blob 7 of 12, id: 8, sources: 2, size: 20x33, npix 492, brick X: 255,275, Y: 198,231, one pixel: 268 198
Blob 8 of 12, id: 11, sources: 3, size: 32x21, npix 462, brick X: 243,275, Y: 254,275, one pixel: 253 254
Blob 9 of 12, id: 6, sources: 1, size: 26x24, npix 377, brick X: 232,258, Y: 142,166, one pixel: 244 142
Blob 10 of 12, id: 5, sources: 1, size: 25x24, npix 375, brick X: 248,273, Y: 77,101, one pixel: 259 77
Blob 11 of 12, id: 7, sources: 1, size: 23x24, npix 349, brick X: 73,96, Y: 145,169, one pixel: 83 145
Blob 12 of 12, id: 10, sources: 1, size: 19x19, npix 217, brick X: 65,84, Y: 208,227, one pixel: 73 208
Blob 12 finished, total: Wall: 0.05 s, CPU: 0.05 s, VmPeak: 805 MB, VmSize: 731 MB, VmRSS: 147 MB, VmData: 183 MB, maxrss: 0.150088 MB
Blob 11 finished, total: Wall: 0.17 s, CPU: 0.17 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 91 MB, VmData: 139 MB, maxrss: 0.093488 MB
Blob 10 finished, total: Wall: 0.35 s, CPU: 0.35 s, VmPeak: 812 MB, VmSize: 731 MB, VmRSS: 146 MB, VmData: 183 MB, maxrss: 0.149284 MB
Blob 9 finished, total: Wall: 0.41 s, CPU: 0.41 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 90 MB, VmData: 138 MB, maxrss: 0.092004 MB
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (272.10748, 51.97514), pm (-0.1, -6.8), parallax 0.686 with NanoMaggies: g=20.8, r=19.3, z=18.1
Reset brightness to NanoMaggies: g=20.5, r=19.1, z=18.2
Blob 7 finished, total: Wall: 0.52 s, CPU: 0.51 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 90 MB, VmData: 138 MB, maxrss: 0.092052 MB
Blob 5 finished, total: Wall: 0.71 s, CPU: 0.70 s, VmPeak: 807 MB, VmSize: 731 MB, VmRSS: 147 MB, VmData: 183 MB, maxrss: 0.150708 MB
Measuring 1 iterative sources
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (271.93958, 52.02776), pm (-0.5, 18.7), parallax 3.568 with NanoMaggies: g=8.35, r=7.58, z=7.24
Reset brightness to NanoMaggies: g=8.35, r=7.58, z=7.24
Blob 8 finished, total: Wall: 0.77 s, CPU: 0.77 s, VmPeak: 670 MB, VmSize: 670 MB, VmRSS: 95 MB, VmData: 141 MB, maxrss: 0.096828 MB
Measuring 1 iterative sources
Blob 4 finished, total: Wall: 0.89 s, CPU: 0.89 s, VmPeak: 810 MB, VmSize: 732 MB, VmRSS: 149 MB, VmData: 183 MB, maxrss: 0.152488 MB
Blob 6 finished, total: Wall: 0.97 s, CPU: 0.97 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 93 MB, VmData: 138 MB, maxrss: 0.094736 MB
Measuring 2 iterative sources
Blob 3 finished, total: Wall: 1.26 s, CPU: 1.26 s, VmPeak: 809 MB, VmSize: 731 MB, VmRSS: 146 MB, VmData: 183 MB, maxrss: 0.14972 MB
Blob 2 finished, total: Wall: 1.59 s, CPU: 1.59 s, VmPeak: 670 MB, VmSize: 670 MB, VmRSS: 95 MB, VmData: 141 MB, maxrss: 0.096804 MB
Known galaxy.  Initial shape: EllipseWithPriors(0.25): log r_e=1.19653, ee1=0.120464, ee2=-0.397401
Measuring 14 iterative sources
Blob 1 finished, total: Wall: 23.55 s, CPU: 23.55 s, VmPeak: 703 MB, VmSize: 700 MB, VmRSS: 132 MB, VmData: 171 MB, maxrss: 0.138524 MB
Old catalog: 45
New catalog: 53
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/metrics/cus/blobs-custom-272123p51969.fits.gz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/metrics/cus/all-models-custom-272123p51969.fits
Stage fitblobs : Wall: 26.08 s, CPU: 0.37 s, VmPeak: 1009 MB, VmSize: 1009 MB, VmRSS: 239 MB, VmData: 264 MB, maxrss: 0.244644 MB
Stage fitblobs finished: 2020-08-05T22:10:22.534564
Resources for stage fitblobs : 31.396 s worker CPU, 31.458 s worker Wall, Wall: 26.086 s, Cores in use: 1.22, Total efficiency (on 32 cores): 3.8 %
Stage fitblobs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-fitblobs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-fitblobs.p
Running stage coadds
Running stage coadds at 2020-08-05T22:10:22.560182
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-ccds.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-nexp-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-depth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-galdepth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-psfsize-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-chi2-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-blobmodel-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-nexp-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-depth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-galdepth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-psfsize-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-chi2-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-blobmodel-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-nexp-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-depth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-galdepth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-psfsize-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-chi2-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-blobmodel-z.fits.fz
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-depth.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/tmp-legacysurvey-custom-272123p51969-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/tmp-legacysurvey-custom-272123p51969-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/tmp-legacysurvey-custom-272123p51969-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/tmp-legacysurvey-custom-272123p51969-resid.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-resid.jpg
Stage coadds : Wall: 13.48 s, CPU: 2.10 s, VmPeak: 1009 MB, VmSize: 1009 MB, VmRSS: 239 MB, VmData: 264 MB, maxrss: 0.245032 MB
Stage coadds finished: 2020-08-05T22:10:36.043836
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage coadds : 27.053 s worker CPU, 57.695 s worker Wall, Wall: 13.486 s, Cores in use: 2.16, Total efficiency (on 32 cores): 6.8 %
Stage coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-coadds.p
Running stage wise_forced
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage wise_forced at 2020-08-05T22:10:36.060278
Cut to 1 unWISE tiles
Reading WISE tile 2717p514 band 1
Reading WISE tile 2717p514 band 2
Reading WISE tile 2717p514 band 3
Reading WISE tile 2717p514 band 4
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w4-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w4-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w4-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.10 s, CPU: 0.10 s, VmPeak: 812 MB, VmSize: 810 MB, VmRSS: 151 MB, VmData: 178 MB, maxrss: 0.191868 MB
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w3-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w3-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w3-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.07 s, CPU: 0.07 s, VmPeak: 805 MB, VmSize: 799 MB, VmRSS: 140 MB, VmData: 171 MB, maxrss: 0.202508 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w1-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w1-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w1-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w1-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.07 s, CPU: 0.07 s, VmPeak: 813 MB, VmSize: 801 MB, VmRSS: 142 MB, VmData: 173 MB, maxrss: 0.204352 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w2-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w2-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w2-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w2-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.07 s, CPU: 0.07 s, VmPeak: 810 MB, VmSize: 799 MB, VmRSS: 141 MB, VmData: 171 MB, maxrss: 0.202904 MB
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/tmp-legacysurvey-custom-272123p51969-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/tmp-legacysurvey-custom-272123p51969-wisemodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-wisemodel.jpg
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Stage wise_forced : Wall: 4.80 s, CPU: 0.52 s, VmPeak: 1009 MB, VmSize: 1009 MB, VmRSS: 240 MB, VmData: 264 MB, maxrss: 0.245544 MB
Stage wise_forced finished: 2020-08-05T22:10:40.862655
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage wise_forced : 3.290 s worker CPU, 6.981 s worker Wall, Wall: 4.805 s, Cores in use: 0.79, Total efficiency (on 32 cores): 2.5 %
Stage wise_forced finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-wise_forced.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-wise_forced.p
Running stage writecat
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage writecat at 2020-08-05T22:10:40.879201
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-maskbits.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/tractor-i/cus/tractor-custom-272123p51969.fits
Reading SFD maps...
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/tractor/cus/tractor-custom-272123p51969.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/tractor/cus/brick-custom-272123p51969.sha256sum
Stage writecat : Wall: 0.77 s, CPU: 0.57 s, VmPeak: 1142 MB, VmSize: 1014 MB, VmRSS: 242 MB, VmData: 264 MB, maxrss: 0.378536 MB
Stage writecat finished: 2020-08-05T22:10:41.653628
Resources for stage writecat : 0.000 s worker CPU, 0.001 s worker Wall, Wall: 0.777 s, Cores in use: 0.74, Total efficiency (on 32 cores): 2.3 %
Stage writecat finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-writecat.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/272/PGC2403824/PGC2403824-largegalaxy-writecat.p
All done: 68.555 s worker CPU, 123.053 s worker Wall, Wall: 72.184 s, Cores in use: 1.10, Total efficiency (on 32 cores): 3.4 %
Finished galaxy PGC2403824 in 1.241 minutes.
Started working on galaxy PGC2403824 at Fri Aug  7 09:46:39 2020
Bands touching this brick, g r z
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 272.12335372349764 51.969246095098725 --width 275 --height 275 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 

runbrick.py starting at 2020-08-07T09:46:40.772772
legacypipe git version: DR9.5.11-watershed-3-gd509eaaf
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '272.12335372349764', '51.969246095098725', '--width', '275', '--height', '275', '--pixscale', '0.262', '--threads', '16', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 272.12335372349764 51.969246095098725 --width 275 --height 275 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0

Parsed RA,Dec 272.12335372349764 51.969246095098725
Starting process 45272 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059228 MB
Starting process 45273 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059236 MB
Starting process 45274 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059236 MB
Starting process 45275 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059236 MB
Starting process 45276 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059404 MB
Starting process 45277 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059412 MB
Starting process 45278 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059412 MB
Starting process 45279 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 45281 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 45282 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB
Starting process 45283 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 45284 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 45285 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 45286 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 45287 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 45288 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage fit_on_coadds
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-08-07T09:46:41.250232
Keeping 9 CCDs:
  k4m_170901_054117_ooi_zd_ls9-CCD2 z expnum 137611 exptime 144.104 propid 2016A-0453 seeing 0.83 MJD 57997.237 object MzLS_656661_z 
    mosaic/CP/V4.3/CP20170831/k4m_170901_054117_ooi_zd_ls9.fits.fz
  k4m_160527_092207_ooi_zd_ls9-CCD4 z expnum 91872 exptime 80.102 propid 2016A-0453 seeing 0.78 MJD 57535.390 object MzLS_657438_z 
    mosaic/CP/V4.3/CP20160526/k4m_160527_092207_ooi_zd_ls9.fits.fz
  k4m_160526_091422_ooi_zd_ls9-CCD3 z expnum 91634 exptime 140.105 propid 2016A-0453 seeing 1.42 MJD 57534.385 object MzLS_658211_z 
    mosaic/CP/V4.3/CP20160525/k4m_160526_091422_ooi_zd_ls9.fits.fz
  ksb_170405_112630_ooi_r_ls9-CCD2 r expnum 78480176 exptime 59.967 propid BASS seeing 1.11 MJD 57848.477 object 29451 
    90prime/CP/V2.3/CP20170404/ksb_170405_112630_ooi_r_ls9.fits.fz
  ksb_170527_090558_ooi_g_ls9-CCD2 g expnum 79000142 exptime 114.3 propid 2015A-0801 seeing 1.44 MJD 57900.379 object 29451 
    90prime/CP/V2.3/CP20170526/ksb_170527_090558_ooi_g_ls9.fits.fz
  ksb_160707_091929_ooi_g_ls9-CCD2 g expnum 75760094 exptime 229.0 propid BASS seeing 2.13 MJD 57576.389 object 29452 
    90prime/CP/V2.3/CP20160706/ksb_160707_091929_ooi_g_ls9.fits.fz
  ksb_160712_064128_ooi_r_ls9-CCD2 r expnum 75810082 exptime 138.0 propid BASS seeing 1.64 MJD 57581.279 object 29452 
    90prime/CP/V2.3/CP20160711/ksb_160712_064128_ooi_r_ls9.fits.fz
  ksb_170624_062822_ooi_r_ls9-CCD3 r expnum 79280073 exptime 76.476 propid BASS seeing 1.13 MJD 57928.270 object 29463 
    90prime/CP/V2.3/CP20170623/ksb_170624_062822_ooi_r_ls9.fits.fz
  ksb_170618_061845_ooi_g_ls9-CCD3 g expnum 79220074 exptime 121.071 propid BASS seeing 1.42 MJD 57922.263 object 29463 
    90prime/CP/V2.3/CP20170617/ksb_170618_061845_ooi_g_ls9.fits.fz
Faked up EXPNUM 75760094
Faked up EXPNUM 75760094
Faked up EXPNUM 75760094
90prime get_fwhm: self.fwhm = 4.71
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2065 2226 and amp range 2016 4031 and image y 3130 3291 and amp range 2048 4095
Range in image: x 0 162 , y 0 162 (with image size 162 x 162)
dzp 0.0026774607 -> scaling image by 1.002469075566243
Faked up EXPNUM 79220074
Faked up EXPNUM 79220074
Faked up EXPNUM 79220074
90prime get_fwhm: self.fwhm = 3.13
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2304 2464 and amp range 2016 4031 and image y 2715 2875 and amp range 2048 4095
Range in image: x 0 161 , y 0 161 (with image size 161 x 161)
dzp -0.02530836 -> scaling image by 0.9769597156943128
Faked up EXPNUM 78480176
Faked up EXPNUM 78480176
Faked up EXPNUM 78480176
90prime get_fwhm: self.fwhm = 2.45
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 41 202 and amp range 0 2015 and image y 1158 1320 and amp range 0 2047
Range in image: x 0 162 , y 0 163 (with image size 162 x 163)
dzp 0.005335051 -> scaling image by 1.0049258559887364
Faked up EXPNUM 79280073
Faked up EXPNUM 79280073
Faked up EXPNUM 79280073
90prime get_fwhm: self.fwhm = 2.49
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2296 2456 and amp range 2016 4031 and image y 2713 2873 and amp range 2048 4095
Range in image: x 0 161 , y 0 161 (with image size 161 x 161)
dzp -0.035697103 -> scaling image by 0.9676563679910417
mosaic get_fwhm: self.fwhm = 3.21
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 3153 3439 and amp range 2048 4095 and image y 627 913 and amp range 0 2039
Range in image: x 0 287 , y 0 287 (with image size 287 x 287)
dzp -0.012875672 -> scaling image by 0.9882111074411305
mosaic get_fwhm: self.fwhm = 3.01
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 454 732 and amp range 0 2047 and image y 1748 2026 and amp range 0 2021
Range in image: x 0 279 , y 0 274 (with image size 279 x 279)
dzp 0.005139143 -> scaling image by 1.0047445456970268
Found overlap: image x 454 732 and amp range 0 2047 and image y 1748 2026 and amp range 2022 4060
Range in image: x 0 279 , y 274 279 (with image size 279 x 279)
dzp 0.0033053984 -> scaling image by 1.0030490232713904
mosaic get_fwhm: self.fwhm = 5.47
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2025 2304 and amp range 2006 4053 and image y 3228 3513 and amp range 2040 4078
Range in image: x 0 280 , y 0 286 (with image size 280 x 286)
dzp -0.0026360443 -> scaling image by 0.99757505842151
Faked up EXPNUM 75810082
Faked up EXPNUM 75810082
Faked up EXPNUM 75810082
90prime get_fwhm: self.fwhm = 3.63
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2064 2225 and amp range 2016 4031 and image y 3144 3305 and amp range 2048 4095
Range in image: x 0 162 , y 0 162 (with image size 162 x 162)
dzp 0.0002007609 -> scaling image by 1.0001849247187462
Faked up EXPNUM 79000142
Faked up EXPNUM 79000142
Faked up EXPNUM 79000142
90prime get_fwhm: self.fwhm = 3.17
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 54 216 and amp range 0 2015 and image y 1152 1314 and amp range 0 2047
Range in image: x 0 163 , y 0 163 (with image size 163 x 163)
dzp 0.0003063402 -> scaling image by 1.0002821895558327
Stage tims : Wall: 26.94 s, CPU: 0.88 s, VmPeak: 951 MB, VmSize: 951 MB, VmRSS: 173 MB, VmData: 207 MB, maxrss: 0.17706 MB
Stage tims finished: 2020-08-07T09:47:08.186896
Resources for stage tims : 4.313 s worker CPU, 90.943 s worker Wall, Wall: 26.938 s, Cores in use: 0.19, Total efficiency (on 16 cores): 1.2 %
Stage tims finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-tims.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-tims.p
Running stage refs
Running stage refs at 2020-08-07T09:47:08.228817
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-01378.fits
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-01275.fits
Matched 4 large galaxies to Gaia stars.
Matched 0 large galaxies to Tycho-2 stars.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/metrics/cus/reference-custom-272123p51969.fits
Stage refs : Wall: 29.85 s, CPU: 4.05 s, VmPeak: 999 MB, VmSize: 993 MB, VmRSS: 218 MB, VmData: 249 MB, maxrss: 0.223356 MB
Stage refs finished: 2020-08-07T09:47:38.081119
Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 29.854 s, Cores in use: 0.14, Total efficiency (on 16 cores): 0.8 %
Stage refs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-refs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-refs.p
Running stage outliers
Running stage outliers at 2020-08-07T09:47:38.125788
Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/metrics/cus/outlier-mask-custom-272123p51969.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/metrics/cus/outliers-pre-custom-272123p51969.jpg
Image ksb_170527_090558_ooi_g_ls9-CCD2 g : masking 93 positive outlier pixels and 0 negative outlier pixels
Image ksb_170624_062822_ooi_r_ls9-CCD3 r : masking 45 positive outlier pixels and 0 negative outlier pixels
Image k4m_170901_054117_ooi_zd_ls9-CCD2 z : masking 801 positive outlier pixels and 0 negative outlier pixels
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/metrics/cus/outlier-mask-custom-272123p51969.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/metrics/cus/outliers-post-custom-272123p51969.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/metrics/cus/outliers-masked-pos-custom-272123p51969.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/metrics/cus/outliers-masked-neg-custom-272123p51969.jpg
Stage outliers : Wall: 1.01 s, CPU: 0.61 s, VmPeak: 1003 MB, VmSize: 1003 MB, VmRSS: 234 MB, VmData: 262 MB, maxrss: 0.239192 MB
Stage outliers finished: 2020-08-07T09:47:39.131227
Resources for stage outliers : 0.775 s worker CPU, 0.831 s worker Wall, Wall: 1.007 s, Cores in use: 1.38, Total efficiency (on 16 cores): 8.6 %
Stage outliers finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-outliers.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-outliers.p
Running stage halos
Running stage halos at 2020-08-07T09:47:39.163089
Stage halos : Wall: 0.34 s, CPU: 0.33 s, VmPeak: 1003 MB, VmSize: 1003 MB, VmRSS: 234 MB, VmData: 262 MB, maxrss: 0.23948 MB
Stage halos finished: 2020-08-07T09:47:39.503386
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.342 s, Cores in use: 0.96, Total efficiency (on 16 cores): 6.0 %
Stage halos finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-halos.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-halos.p
Running stage fit_on_coadds
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Running stage fit_on_coadds at 2020-08-07T09:47:39.585483
PSF sigmas in coadd pixels: ['2.33', '3.47', '2.30']
Using average PSF sigma 2.7007499559400343
Gaussian PSF norm 0.10445054017438318 vs pixelized 0.075242214
average tim pixel scale / coadd scale: 1.7293451846228869
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-copsf-g.fits.fz
PSF sigmas in coadd pixels: ['1.80', '2.67', '1.83']
Using average PSF sigma 2.102160112067544
Gaussian PSF norm 0.1341928191646775 vs pixelized 0.11385945
average tim pixel scale / coadd scale: 1.7293235900250212
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-copsf-r.fits.fz
PSF sigmas in coadd pixels: ['1.35', '1.27', '2.31']
Using average PSF sigma 1.6430887332092283
Gaussian PSF norm 0.17168567106105076 vs pixelized 0.14150959
average tim pixel scale / coadd scale: 0.9909119121090487
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-copsf-z.fits.fz
Stage fit_on_coadds : Wall: 0.22 s, CPU: 0.15 s, VmPeak: 1003 MB, VmSize: 1003 MB, VmRSS: 236 MB, VmData: 263 MB, maxrss: 0.241848 MB
Stage fit_on_coadds finished: 2020-08-07T09:47:39.801603
Resources for stage fit_on_coadds : 0.170 s worker CPU, 0.170 s worker Wall, Wall: 0.218 s, Cores in use: 1.50, Total efficiency (on 16 cores): 9.3 %
Stage fit_on_coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-fit_on_coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-fit_on_coadds.p
Running stage srcs
Running stage srcs at 2020-08-07T09:47:39.826264
SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])]
Found 40 potential peaks
Of 40 potential peaks: 10 in veto map, 6 cut by saddle test, 0 cut by aper test, 24 kept
SED z : 24 new peaks
Found 43 potential peaks
Of 43 potential peaks: 37 in veto map, 5 cut by saddle test, 0 cut by aper test, 1 kept
SED r : 1 new peaks
Found 28 potential peaks
Of 28 potential peaks: 25 in veto map, 2 cut by saddle test, 0 cut by aper test, 1 kept
SED g : 1 new peaks
Found 49 potential peaks
Of 49 potential peaks: 42 in veto map, 6 cut by saddle test, 0 cut by aper test, 1 kept
SED Flat : 1 new peaks
Found 70 potential peaks
Of 70 potential peaks: 61 in veto map, 8 cut by saddle test, 0 cut by aper test, 1 kept
SED Red : 1 new peaks
Adding 28 new sources
Median coadd sky for g : 7.0232454e-05
Median coadd sky for r : 0.00014778305
Median coadd sky for z : 0.00017386492
Stage srcs : Wall: 0.60 s, CPU: 0.56 s, VmPeak: 1003 MB, VmSize: 1003 MB, VmRSS: 237 MB, VmData: 263 MB, maxrss: 0.242468 MB
Stage srcs finished: 2020-08-07T09:47:40.428875
Resources for stage srcs : 0.080 s worker CPU, 0.080 s worker Wall, Wall: 0.604 s, Cores in use: 1.06, Total efficiency (on 16 cores): 6.6 %
Stage srcs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-srcs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-srcs.p
Running stage fitblobs
Running stage fitblobs at 2020-08-07T09:47:40.442857
Blob 1 of 13, id: 2, sources: 20, size: 203x254, npix 25457, brick X: 37,240, Y: 8,262, one pixel: 140 8
Blob 2 of 13, id: 4, sources: 2, size: 56x44, npix 1407, brick X: 155,211, Y: 37,81, one pixel: 170 37
Blob 3 of 13, id: 1, sources: 3, size: 54x52, npix 1162, brick X: 178,232, Y: 0,52, one pixel: 188 0
Blob 4 of 13, id: 10, sources: 1, size: 38x29, npix 711, brick X: 10,48, Y: 211,240, one pixel: 24 211
Blob 5 of 13, id: 9, sources: 1, size: 25x37, npix 695, brick X: 250,275, Y: 200,237, one pixel: 266 200
Blob 6 of 13, id: 0, sources: 2, size: 41x27, npix 681, brick X: 8,49, Y: 0,27, one pixel: 14 0
Blob 7 of 13, id: 5, sources: 1, size: 25x23, npix 373, brick X: 243,268, Y: 81,104, one pixel: 253 81
Blob 8 of 13, id: 6, sources: 1, size: 26x24, npix 369, brick X: 227,253, Y: 146,170, one pixel: 236 146
Blob 9 of 13, id: 7, sources: 1, size: 24x24, npix 341, brick X: 67,91, Y: 148,172, one pixel: 79 148
Blob 10 of 13, id: 3, sources: 1, size: 20x24, npix 316, brick X: 0,20, Y: 16,40, one pixel: 5 16
Blob 11 of 13, id: 12, sources: 1, size: 21x18, npix 256, brick X: 239,260, Y: 257,275, one pixel: 249 257
Blob 12 of 13, id: 11, sources: 1, size: 19x20, npix 218, brick X: 60,79, Y: 211,231, one pixel: 69 211
Blob 13 of 13, id: 8, sources: 1, size: 17x19, npix 195, brick X: 258,275, Y: 155,174, one pixel: 266 155
Blob 12 finished, total: Wall: 0.09 s, CPU: 0.09 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 87 MB, VmData: 138 MB, maxrss: 0.08944 MB
Blob 11 finished, total: Wall: 0.16 s, CPU: 0.16 s, VmPeak: 810 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.147584 MB
Blob 9 finished, total: Wall: 0.20 s, CPU: 0.20 s, VmPeak: 809 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.147052 MB
Blob 7 finished, total: Wall: 0.34 s, CPU: 0.34 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 93 MB, VmData: 139 MB, maxrss: 0.094932 MB
Blob 13 finished, total: Wall: 0.48 s, CPU: 0.47 s, VmPeak: 805 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.146408 MB
Blob 8 finished, total: Wall: 0.50 s, CPU: 0.50 s, VmPeak: 809 MB, VmSize: 725 MB, VmRSS: 144 MB, VmData: 183 MB, maxrss: 0.147316 MB
Blob 10 finished, total: Wall: 0.59 s, CPU: 0.59 s, VmPeak: 670 MB, VmSize: 670 MB, VmRSS: 95 MB, VmData: 141 MB, maxrss: 0.09722 MB
Blob 5 finished, total: Wall: 0.62 s, CPU: 0.61 s, VmPeak: 813 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.146888 MB
Measuring 1 iterative sources
Blob 6 finished, total: Wall: 0.97 s, CPU: 0.97 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 93 MB, VmData: 139 MB, maxrss: 0.095144 MB
Blob 4 finished, total: Wall: 1.03 s, CPU: 1.03 s, VmPeak: 812 MB, VmSize: 725 MB, VmRSS: 145 MB, VmData: 183 MB, maxrss: 0.147984 MB
Blob 3 finished, total: Wall: 1.19 s, CPU: 1.19 s, VmPeak: 670 MB, VmSize: 670 MB, VmRSS: 95 MB, VmData: 141 MB, maxrss: 0.097488 MB
Measuring 2 iterative sources
Blob 2 finished, total: Wall: 1.66 s, CPU: 1.66 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 93 MB, VmData: 139 MB, maxrss: 0.095668 MB
Known galaxy.  Initial shape: EllipseWithPriors(0.25): log r_e=0.652163, ee1=0.109471, ee2=-0.334819
Measuring 16 iterative sources
Blob 1 finished, total: Wall: 22.45 s, CPU: 22.45 s, VmPeak: 709 MB, VmSize: 696 MB, VmRSS: 127 MB, VmData: 164 MB, maxrss: 0.14096 MB
Old catalog: 43
New catalog: 51
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/metrics/cus/blobs-custom-272123p51969.fits.gz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/metrics/cus/all-models-custom-272123p51969.fits
Stage fitblobs : Wall: 23.01 s, CPU: 0.42 s, VmPeak: 1003 MB, VmSize: 1003 MB, VmRSS: 237 MB, VmData: 263 MB, maxrss: 0.242892 MB
Stage fitblobs finished: 2020-08-07T09:48:03.452672
Resources for stage fitblobs : 30.476 s worker CPU, 30.779 s worker Wall, Wall: 23.011 s, Cores in use: 1.34, Total efficiency (on 16 cores): 8.4 %
Stage fitblobs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-fitblobs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-fitblobs.p
Running stage coadds
Running stage coadds at 2020-08-07T09:48:03.499845
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-ccds.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-nexp-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-depth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-galdepth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-psfsize-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-chi2-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-blobmodel-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-nexp-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-depth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-galdepth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-psfsize-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-chi2-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-blobmodel-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-nexp-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-depth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-galdepth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-psfsize-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-chi2-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-blobmodel-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-depth.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/tmp-legacysurvey-custom-272123p51969-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/tmp-legacysurvey-custom-272123p51969-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/tmp-legacysurvey-custom-272123p51969-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/tmp-legacysurvey-custom-272123p51969-resid.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-resid.jpg
Stage coadds : Wall: 2.42 s, CPU: 0.66 s, VmPeak: 1003 MB, VmSize: 1003 MB, VmRSS: 238 MB, VmData: 263 MB, maxrss: 0.243216 MB
Stage coadds finished: 2020-08-07T09:48:05.924535
Resources for stage coadds : 9.601 s worker CPU, 10.647 s worker Wall, Wall: 2.426 s, Cores in use: 4.23, Total efficiency (on 16 cores): 26.4 %
Stage coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-coadds.p
Running stage wise_forced
Running stage wise_forced at 2020-08-07T09:48:05.954990
Cut to 1 unWISE tiles
Reading WISE tile 2717p514 band 1
Reading WISE tile 2717p514 band 2
Reading WISE tile 2717p514 band 3
Reading WISE tile 2717p514 band 4
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w4-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w4-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w4-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.09 s, CPU: 0.09 s, VmPeak: 813 MB, VmSize: 804 MB, VmRSS: 150 MB, VmData: 178 MB, maxrss: 0.189544 MB
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w3-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w3-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w3-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.07 s, CPU: 0.07 s, VmPeak: 805 MB, VmSize: 793 MB, VmRSS: 140 MB, VmData: 171 MB, maxrss: 0.201468 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w1-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w1-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w1-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w1-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.07 s, CPU: 0.07 s, VmPeak: 809 MB, VmSize: 799 MB, VmRSS: 145 MB, VmData: 173 MB, maxrss: 0.206196 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w2-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w2-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w2-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/271/2717p514/unwise-2717p514-w2-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.07 s, CPU: 0.07 s, VmPeak: 807 MB, VmSize: 792 MB, VmRSS: 139 MB, VmData: 171 MB, maxrss: 0.198964 MB
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-image-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-model-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-invvar-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/tmp-legacysurvey-custom-272123p51969-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/tmp-legacysurvey-custom-272123p51969-wisemodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-wisemodel.jpg
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Stage wise_forced : Wall: 10.85 s, CPU: 0.63 s, VmPeak: 1003 MB, VmSize: 1003 MB, VmRSS: 238 MB, VmData: 263 MB, maxrss: 0.243728 MB
Stage wise_forced finished: 2020-08-07T09:48:16.800705
Resources for stage wise_forced : 3.108 s worker CPU, 12.151 s worker Wall, Wall: 10.847 s, Cores in use: 0.34, Total efficiency (on 16 cores): 2.2 %
Stage wise_forced finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-wise_forced.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-wise_forced.p
Running stage writecat
Running stage writecat at 2020-08-07T09:48:16.917554
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/coadd/cus/custom-272123p51969/legacysurvey-custom-272123p51969-maskbits.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/tractor-i/cus/tractor-custom-272123p51969.fits
Reading SFD maps...
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/tractor/cus/tractor-custom-272123p51969.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/tractor/cus/brick-custom-272123p51969.sha256sum
Stage writecat : Wall: 0.72 s, CPU: 0.54 s, VmPeak: 1135 MB, VmSize: 1007 MB, VmRSS: 240 MB, VmData: 263 MB, maxrss: 0.37674 MB
Stage writecat finished: 2020-08-07T09:48:17.639018
Resources for stage writecat : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 0.723 s, Cores in use: 0.75, Total efficiency (on 16 cores): 4.7 %
Stage writecat finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-writecat.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/272/PGC2403824/PGC2403824-largegalaxy-writecat.p
All done: 48.526 s worker CPU, 145.605 s worker Wall, Wall: 96.437 s, Cores in use: 0.60, Total efficiency (on 16 cores): 3.7 %
Finished galaxy PGC2403824 in 1.651 minutes.