Started working on galaxy PGC2659520 at Wed Aug  5 19:43:04 2020
Bands touching this brick, g r z
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 283.968 63.679264100000005 --width 245 --height 245 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 

runbrick.py starting at 2020-08-05T19:43:06.163722
legacypipe git version: DR9.5.10-12-gf22e529f
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '283.968', '63.679264100000005', '--width', '245', '--height', '245', '--pixscale', '0.262', '--threads', '16', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-%%(stage)s.p', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 283.968 63.679264100000005 --width 245 --height 245 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0

Parsed RA,Dec 283.968 63.679264100000005
Starting process 59218 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059232 MB
Starting process 59219 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059236 MB
Starting process 59220 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059236 MB
Starting process 59221 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB
Starting process 59222 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 59223 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 59224 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 59225 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB
Starting process 59226 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB
Starting process 59227 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 59228 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 59229 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 59230 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 59231 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 59232 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 59233 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage fit_on_coadds
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-08-05T19:43:07.451880
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Keeping 9 CCDs:
  k4m_171029_050735_ooi_zd_ls9-CCD2 z expnum 143937 exptime 125.106 propid 2016A-0453 seeing 0.90 MJD 58055.214 object MzLS_697671_z 
    mosaic/CP/V4.3/CP20171028/k4m_171029_050735_ooi_zd_ls9.fits.fz
  k4m_171002_054418_ooi_zd_ls9-CCD2 z expnum 141841 exptime 153.104 propid 2016A-0453 seeing 1.19 MJD 58028.239 object MzLS_697109_z 
    mosaic/CP/V4.3/CP20171001/k4m_171002_054418_ooi_zd_ls9.fits.fz
  k4m_180206_125911_ooi_zd_ls9-CCD3 z expnum 160867 exptime 84.104 propid 2016A-0453 seeing 0.86 MJD 58155.541 object MzLS_698229_z 
    mosaic/CP/V4.3/CP20180205/k4m_180206_125911_ooi_zd_ls9.fits.fz
  ksb_170622_075117_ooi_g_ls9-CCD2 g expnum 79260099 exptime 176.652 propid 2015A-0801 seeing 1.50 MJD 57926.327 object 41523 
    90prime/CP/V2.3/CP20170621/ksb_170622_075117_ooi_g_ls9.fits.fz
  ksb_170624_105727_ooi_r_ls9-CCD2 r expnum 79280177 exptime 150.605 propid BASS seeing 1.24 MJD 57928.457 object 41523 
    90prime/CP/V2.3/CP20170623/ksb_170624_105727_ooi_r_ls9.fits.fz
  ksb_160711_103513_ooi_g_ls9-CCD1 g expnum 75800144 exptime 148.2 propid BASS seeing 1.54 MJD 57580.441 object 40632 
    90prime/CP/V2.3/CP20160710/ksb_160711_103513_ooi_g_ls9.fits.fz
  ksb_180519_084839_ooi_r_ls9-CCD1 r expnum 82570157 exptime 71.439 propid NOTBASS seeing 1.10 MJD 58257.367 object 40632 
    90prime/CP/V2.3/CP20180518/ksb_180519_084839_ooi_r_ls9.fits.fz
  ksb_180519_101208_ooi_g_ls9-CCD4 g expnum 82570193 exptime 102.439 propid NOTBASS seeing 1.25 MJD 58257.425 object 41521 
    90prime/CP/V2.3/CP20180518/ksb_180519_101208_ooi_g_ls9.fits.fz
  ksb_180508_104811_ooi_r_ls9-CCD4 r expnum 82460216 exptime 116.385 propid NOTBASS seeing 1.27 MJD 58246.450 object 41521 
    90prime/CP/V2.3/CP20180507/ksb_180508_104811_ooi_r_ls9.fits.fz
mosaic get_fwhm: self.fwhm = 3.31
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 1784 2033 and amp range 0 2005 and image y 828 1076 and amp range 0 2039
Range in image: x 0 222 , y 0 249 (with image size 250 x 249)
dzp 0.0014252509 -> scaling image by 1.0013135665633692
Found overlap: image x 1784 2033 and amp range 2006 4053 and image y 828 1076 and amp range 0 2039
Range in image: x 222 250 , y 0 249 (with image size 250 x 249)
dzp -0.006757846 -> scaling image by 0.993795123967763
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 79280177
Faked up EXPNUM 79280177
Faked up EXPNUM 79280177
90prime get_fwhm: self.fwhm = 2.73
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 409 552 and amp range 0 2015 and image y 3381 3524 and amp range 2048 4095
Range in image: x 0 144 , y 0 144 (with image size 144 x 144)
dzp 0.05525464 -> scaling image by 1.0522086198429996
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 4.59
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2902 3155 and amp range 2048 4095 and image y 1417 1668 and amp range 0 2039
Range in image: x 0 254 , y 0 252 (with image size 254 x 252)
dzp -0.012875672 -> scaling image by 0.9882111074411305
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 75800144
Faked up EXPNUM 75800144
Faked up EXPNUM 75800144
90prime get_fwhm: self.fwhm = 3.41
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 614 758 and amp range 0 2015 and image y 3556 3700 and amp range 2048 4095
Range in image: x 0 145 , y 0 145 (with image size 145 x 145)
dzp -0.0051191514 -> scaling image by 0.9952961850385206
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 82460216
Faked up EXPNUM 82460216
Faked up EXPNUM 82460216
90prime get_fwhm: self.fwhm = 2.81
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2422 2565 and amp range 2016 4031 and image y 884 1026 and amp range 0 2047
Range in image: x 0 144 , y 0 143 (with image size 144 x 143)
dzp 0.043530058 -> scaling image by 1.0409072255640852
mosaic get_fwhm: self.fwhm = 3.47
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 211 459 and amp range 0 2047 and image y 544 795 and amp range 0 2039
Range in image: x 0 249 , y 0 252 (with image size 249 x 252)
dzp 0.007023884 -> scaling image by 1.0064902069278665
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 82570157
Faked up EXPNUM 82570157
Faked up EXPNUM 82570157
90prime get_fwhm: self.fwhm = 2.43
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 603 748 and amp range 0 2015 and image y 3571 3715 and amp range 2048 4095
Range in image: x 0 146 , y 0 145 (with image size 146 x 145)
dzp -0.05213645 -> scaling image by 0.953115252660837
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 79260099
Faked up EXPNUM 79260099
Faked up EXPNUM 79260099
90prime get_fwhm: self.fwhm = 3.31
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 412 555 and amp range 0 2015 and image y 3373 3516 and amp range 2048 4095
Range in image: x 0 144 , y 0 144 (with image size 144 x 144)
dzp 0.051889952 -> scaling image by 1.0489528829333021
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 82570193
Faked up EXPNUM 82570193
Faked up EXPNUM 82570193
90prime get_fwhm: self.fwhm = 2.75
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2434 2577 and amp range 2016 4031 and image y 882 1025 and amp range 0 2047
Range in image: x 0 144 , y 0 144 (with image size 144 x 144)
dzp 0.023266561 -> scaling image by 1.0216605511853067
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Stage tims : Wall: 8.08 s, CPU: 1.50 s, VmPeak: 957 MB, VmSize: 957 MB, VmRSS: 174 MB, VmData: 206 MB, maxrss: 0.178132 MB
Stage tims finished: 2020-08-05T19:43:15.535568
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage tims : 4.867 s worker CPU, 30.685 s worker Wall, Wall: 8.086 s, Cores in use: 0.79, Total efficiency (on 16 cores): 4.9 %
Stage tims finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-tims.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-tims.p
Running stage refs
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage refs at 2020-08-05T19:43:16.160836
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00744.fits
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00597.fits
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00668.fits
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00669.fits
Matched 4 large galaxies to Gaia stars.
Matched 0 large galaxies to Tycho-2 stars.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/metrics/cus/reference-custom-283968p63679.fits
Stage refs : Wall: 10.25 s, CPU: 4.86 s, VmPeak: 1020 MB, VmSize: 1014 MB, VmRSS: 233 MB, VmData: 263 MB, maxrss: 0.23884 MB
Stage refs finished: 2020-08-05T19:43:26.415073
Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 10.256 s, Cores in use: 0.47, Total efficiency (on 16 cores): 3.0 %
Stage refs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-refs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-refs.p
Running stage outliers
Running stage outliers at 2020-08-05T19:43:26.477141
Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/metrics/cus/outlier-mask-custom-283968p63679.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/metrics/cus/outliers-pre-custom-283968p63679.jpg
Image ksb_160711_103513_ooi_g_ls9-CCD1 g : masking 147 positive outlier pixels and 0 negative outlier pixels
Image ksb_180519_084839_ooi_r_ls9-CCD1 r : masking 66 positive outlier pixels and 0 negative outlier pixels
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/metrics/cus/outlier-mask-custom-283968p63679.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/metrics/cus/outliers-post-custom-283968p63679.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/metrics/cus/outliers-masked-pos-custom-283968p63679.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/metrics/cus/outliers-masked-neg-custom-283968p63679.jpg
Stage outliers : Wall: 2.05 s, CPU: 1.66 s, VmPeak: 1024 MB, VmSize: 1024 MB, VmRSS: 247 MB, VmData: 273 MB, maxrss: 0.252524 MB
Stage outliers finished: 2020-08-05T19:43:28.528196
Resources for stage outliers : 0.595 s worker CPU, 0.780 s worker Wall, Wall: 2.052 s, Cores in use: 1.10, Total efficiency (on 16 cores): 6.9 %
Stage outliers finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-outliers.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-outliers.p
Running stage halos
Running stage halos at 2020-08-05T19:43:28.599084
Stage halos : Wall: 0.98 s, CPU: 0.95 s, VmPeak: 1024 MB, VmSize: 1024 MB, VmRSS: 247 MB, VmData: 274 MB, maxrss: 0.25278 MB
Stage halos finished: 2020-08-05T19:43:29.579030
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.981 s, Cores in use: 0.97, Total efficiency (on 16 cores): 6.1 %
Stage halos finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-halos.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-halos.p
Running stage fit_on_coadds
Warning: Stellar halo subtraction is only implemented for DECam
Running stage fit_on_coadds at 2020-08-05T19:43:29.604476
PSF sigmas in coadd pixels: ['2.44', '2.51', '2.03']
Using average PSF sigma 2.3231764864877995
Gaussian PSF norm 0.12142632874196818 vs pixelized 0.102694176
average tim pixel scale / coadd scale: 1.7296033866973664
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-copsf-g.fits.fz
PSF sigmas in coadd pixels: ['2.01', '1.79', '2.07']
Using average PSF sigma 1.9556303441132592
Gaussian PSF norm 0.14424750189780283 vs pixelized 0.12053512
average tim pixel scale / coadd scale: 1.7298821836198104
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-copsf-r.fits.fz
PSF sigmas in coadd pixels: ['1.46', '1.94', '1.40']
Using average PSF sigma 1.5983717892361387
Gaussian PSF norm 0.1764888455073967 vs pixelized 0.13782366
average tim pixel scale / coadd scale: 0.9910767375427575
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-copsf-z.fits.fz
Stage fit_on_coadds : Wall: 0.16 s, CPU: 0.13 s, VmPeak: 1024 MB, VmSize: 1024 MB, VmRSS: 249 MB, VmData: 274 MB, maxrss: 0.254892 MB
Stage fit_on_coadds finished: 2020-08-05T19:43:29.761645
Resources for stage fit_on_coadds : 0.142 s worker CPU, 0.142 s worker Wall, Wall: 0.158 s, Cores in use: 1.74, Total efficiency (on 16 cores): 10.9 %
Stage fit_on_coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-fit_on_coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-fit_on_coadds.p
Running stage srcs
Warning: Stellar halo subtraction is only implemented for DECam
Running stage srcs at 2020-08-05T19:43:29.774650
SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])]
Found 19 potential peaks
Of 19 potential peaks: 6 in veto map, 2 cut by saddle test, 0 cut by aper test, 11 kept
SED z : 11 new peaks
Found 19 potential peaks
Of 19 potential peaks: 15 in veto map, 2 cut by saddle test, 0 cut by aper test, 2 kept
SED r : 2 new peaks
Found 17 potential peaks
Of 17 potential peaks: 16 in veto map, 0 cut by saddle test, 0 cut by aper test, 1 kept
SED g : 1 new peaks
Found 27 potential peaks
Of 27 potential peaks: 24 in veto map, 3 cut by saddle test, 0 cut by aper test, 0 kept
SED Flat : 0 new peaks
Found 24 potential peaks
Of 24 potential peaks: 20 in veto map, 4 cut by saddle test, 0 cut by aper test, 0 kept
SED Red : 0 new peaks
Adding 14 new sources
Median coadd sky for g : 5.360128e-05
Median coadd sky for r : 0.00038072618
Median coadd sky for z : 0.00010552217
Stage srcs : Wall: 1.03 s, CPU: 0.99 s, VmPeak: 1024 MB, VmSize: 1024 MB, VmRSS: 250 MB, VmData: 274 MB, maxrss: 0.255572 MB
Stage srcs finished: 2020-08-05T19:43:30.804238
Resources for stage srcs : 0.082 s worker CPU, 0.082 s worker Wall, Wall: 1.031 s, Cores in use: 1.04, Total efficiency (on 16 cores): 6.5 %
Stage srcs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-srcs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-srcs.p
Running stage fitblobs
Running stage fitblobs at 2020-08-05T19:43:30.815075
Blob 1 of 10, id: 5, sources: 7, size: 106x150, npix 10431, brick X: 77,183, Y: 46,196, one pixel: 167 46
Blob 2 of 10, id: 9, sources: 2, size: 54x70, npix 2275, brick X: 188,242, Y: 147,217, one pixel: 214 147
Blob 3 of 10, id: 3, sources: 2, size: 44x54, npix 1415, brick X: 177,221, Y: 17,71, one pixel: 192 17
Blob 4 of 10, id: 4, sources: 2, size: 52x36, npix 1119, brick X: 19,71, Y: 33,69, one pixel: 49 33
Blob 5 of 10, id: 8, sources: 2, size: 41x40, npix 1063, brick X: 42,83, Y: 119,159, one pixel: 53 119
Blob 6 of 10, id: 6, sources: 1, size: 24x42, npix 735, brick X: 0,24, Y: 54,96, one pixel: 5 54
Blob 7 of 10, id: 0, sources: 1, size: 33x22, npix 530, brick X: 51,84, Y: 0,22, one pixel: 56 0
Blob 8 of 10, id: 1, sources: 1, size: 20x25, npix 335, brick X: 225,245, Y: 4,29, one pixel: 236 4
Blob 9 of 10, id: 2, sources: 1, size: 19x24, npix 313, brick X: 0,19, Y: 13,37, one pixel: 5 13
Blob 10 of 10, id: 7, sources: 1, size: 20x23, npix 294, brick X: 225,245, Y: 91,114, one pixel: 235 91
Blob 9 finished, total: Wall: 0.15 s, CPU: 0.15 s, VmPeak: 805 MB, VmSize: 731 MB, VmRSS: 146 MB, VmData: 183 MB, maxrss: 0.149472 MB
Blob 10 finished, total: Wall: 0.18 s, CPU: 0.18 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 91 MB, VmData: 138 MB, maxrss: 0.093176 MB
Blob 2 finished, total: Wall: 0.35 s, CPU: 0.35 s, VmPeak: 808 MB, VmSize: 732 MB, VmRSS: 146 MB, VmData: 183 MB, maxrss: 0.149572 MB
Blob 8 finished, total: Wall: 0.40 s, CPU: 0.40 s, VmPeak: 810 MB, VmSize: 732 MB, VmRSS: 146 MB, VmData: 183 MB, maxrss: 0.149964 MB
Blob 7 finished, total: Wall: 0.80 s, CPU: 0.80 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 92 MB, VmData: 138 MB, maxrss: 0.093928 MB
Measuring 2 iterative sources
Blob 5 finished, total: Wall: 1.14 s, CPU: 1.14 s, VmPeak: 810 MB, VmSize: 732 MB, VmRSS: 146 MB, VmData: 183 MB, maxrss: 0.1499 MB
Blob 3 finished, total: Wall: 1.29 s, CPU: 1.29 s, VmPeak: 810 MB, VmSize: 732 MB, VmRSS: 146 MB, VmData: 183 MB, maxrss: 0.149824 MB
Measuring 1 iterative sources
Blob 6 finished, total: Wall: 1.70 s, CPU: 1.69 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 91 MB, VmData: 139 MB, maxrss: 0.093496 MB
Blob 4 finished, total: Wall: 1.72 s, CPU: 1.72 s, VmPeak: 810 MB, VmSize: 732 MB, VmRSS: 148 MB, VmData: 184 MB, maxrss: 0.151324 MB
Known galaxy.  Initial shape: EllipseWithPriors(0.25): log r_e=1.09874, ee1=0.285089, ee2=-0.076153
Measuring 1 iterative sources
Blob 1 finished, total: Wall: 8.20 s, CPU: 8.19 s, VmPeak: 683 MB, VmSize: 682 MB, VmRSS: 109 MB, VmData: 153 MB, maxrss: 0.112396 MB
Old catalog: 23
New catalog: 24
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/metrics/cus/blobs-custom-283968p63679.fits.gz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/metrics/cus/all-models-custom-283968p63679.fits
Stage fitblobs : Wall: 8.59 s, CPU: 0.36 s, VmPeak: 1024 MB, VmSize: 1024 MB, VmRSS: 250 MB, VmData: 274 MB, maxrss: 0.255736 MB
Stage fitblobs finished: 2020-08-05T19:43:39.408979
Resources for stage fitblobs : 16.059 s worker CPU, 16.120 s worker Wall, Wall: 8.595 s, Cores in use: 1.91, Total efficiency (on 16 cores): 11.9 %
Stage fitblobs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-fitblobs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-fitblobs.p
Running stage coadds
Running stage coadds at 2020-08-05T19:43:39.427399
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-ccds.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-image-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-invvar-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-nexp-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-depth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-galdepth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-psfsize-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-model-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-chi2-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-blobmodel-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-image-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-invvar-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-nexp-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-depth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-galdepth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-psfsize-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-model-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-chi2-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-blobmodel-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-image-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-invvar-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-nexp-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-depth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-galdepth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-psfsize-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-model-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-chi2-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-blobmodel-z.fits.fz
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-depth.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/tmp-legacysurvey-custom-283968p63679-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/tmp-legacysurvey-custom-283968p63679-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/tmp-legacysurvey-custom-283968p63679-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/tmp-legacysurvey-custom-283968p63679-resid.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-resid.jpg
Stage coadds : Wall: 4.33 s, CPU: 2.04 s, VmPeak: 1024 MB, VmSize: 1024 MB, VmRSS: 250 MB, VmData: 274 MB, maxrss: 0.25606 MB
Stage coadds finished: 2020-08-05T19:43:43.755026
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage coadds : 9.486 s worker CPU, 15.698 s worker Wall, Wall: 4.329 s, Cores in use: 2.66, Total efficiency (on 16 cores): 16.6 %
Stage coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-coadds.p
Running stage wise_forced
Running stage wise_forced at 2020-08-05T19:43:43.767649
Cut to 1 unWISE tiles
Reading WISE tile 2840p636 band 1
Reading WISE tile 2840p636 band 3
Reading WISE tile 2840p636 band 2
Reading WISE tile 2840p636 band 4
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w3-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w3-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w3-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.04 s, CPU: 0.04 s, VmPeak: 807 MB, VmSize: 802 MB, VmRSS: 144 MB, VmData: 174 MB, maxrss: 0.207496 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w1-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w1-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w1-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w1-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.04 s, CPU: 0.04 s, VmPeak: 812 MB, VmSize: 812 MB, VmRSS: 153 MB, VmData: 180 MB, maxrss: 0.21044 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w2-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w2-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w2-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w2-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.04 s, CPU: 0.04 s, VmPeak: 810 MB, VmSize: 803 MB, VmRSS: 144 MB, VmData: 171 MB, maxrss: 0.207156 MB
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w4-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w4-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w4-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.04 s, CPU: 0.04 s, VmPeak: 811 MB, VmSize: 804 MB, VmRSS: 146 MB, VmData: 176 MB, maxrss: 0.186712 MB
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-image-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-model-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-invvar-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-image-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-model-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-invvar-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-image-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-model-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-invvar-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-image-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-model-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-invvar-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/tmp-legacysurvey-custom-283968p63679-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/tmp-legacysurvey-custom-283968p63679-wisemodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-wisemodel.jpg
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Stage wise_forced : Wall: 4.63 s, CPU: 0.55 s, VmPeak: 1024 MB, VmSize: 1024 MB, VmRSS: 251 MB, VmData: 274 MB, maxrss: 0.256572 MB
Stage wise_forced finished: 2020-08-05T19:43:48.398456
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage wise_forced : 3.096 s worker CPU, 7.899 s worker Wall, Wall: 4.632 s, Cores in use: 0.79, Total efficiency (on 16 cores): 4.9 %
Stage wise_forced finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-wise_forced.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-wise_forced.p
Running stage writecat
Running stage writecat at 2020-08-05T19:43:48.412951
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-maskbits.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/tractor-i/cus/tractor-custom-283968p63679.fits
Reading SFD maps...
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/tractor/cus/tractor-custom-283968p63679.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/tractor/cus/brick-custom-283968p63679.sha256sum
Stage writecat : Wall: 1.04 s, CPU: 0.74 s, VmPeak: 1156 MB, VmSize: 1028 MB, VmRSS: 252 MB, VmData: 274 MB, maxrss: 0.389452 MB
Stage writecat finished: 2020-08-05T19:43:49.456000
Resources for stage writecat : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 1.045 s, Cores in use: 0.71, Total efficiency (on 16 cores): 4.4 %
Stage writecat finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-writecat.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-writecat.p
All done: 34.332 s worker CPU, 71.411 s worker Wall, Wall: 42.028 s, Cores in use: 1.15, Total efficiency (on 16 cores): 7.2 %
Finished galaxy PGC2659520 in 0.765 minutes.
Started working on galaxy PGC2659520 at Wed Aug  5 20:11:51 2020
Bands touching this brick, g r z
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 283.968 63.679264100000005 --width 245 --height 245 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 

runbrick.py starting at 2020-08-05T20:11:54.595309
legacypipe git version: DR9.5.10-12-gf22e529f
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '283.968', '63.679264100000005', '--width', '245', '--height', '245', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-%%(stage)s.p', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 283.968 63.679264100000005 --width 245 --height 245 --pixscale 0.262 --threads 32 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-%%(stage)s.p --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0

Parsed RA,Dec 283.968 63.679264100000005
Starting process 694 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059236 MB
Starting process 695 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059236 MB
Starting process 696 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB
Starting process 697 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05924 MB
Starting process 698 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 699 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 700 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 701 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB
Starting process 702 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059252 MB
Starting process 703 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 704 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059256 MB
Starting process 705 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 706 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 707 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 708 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 709 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 710 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 711 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 712 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05928 MB
Starting process 713 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB
Starting process 714 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB
Starting process 715 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059284 MB
Starting process 716 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059288 MB
Starting process 717 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059292 MB
Starting process 718 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059292 MB
Starting process 719 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB
Starting process 720 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059296 MB
Starting process 721 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.0593 MB
Starting process 722 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Starting process 723 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Starting process 724 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059304 MB
Starting process 725 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059308 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage fit_on_coadds
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-08-05T20:11:55.570063
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Keeping 9 CCDs:
  k4m_171029_050735_ooi_zd_ls9-CCD2 z expnum 143937 exptime 125.106 propid 2016A-0453 seeing 0.90 MJD 58055.214 object MzLS_697671_z 
    mosaic/CP/V4.3/CP20171028/k4m_171029_050735_ooi_zd_ls9.fits.fz
  k4m_171002_054418_ooi_zd_ls9-CCD2 z expnum 141841 exptime 153.104 propid 2016A-0453 seeing 1.19 MJD 58028.239 object MzLS_697109_z 
    mosaic/CP/V4.3/CP20171001/k4m_171002_054418_ooi_zd_ls9.fits.fz
  k4m_180206_125911_ooi_zd_ls9-CCD3 z expnum 160867 exptime 84.104 propid 2016A-0453 seeing 0.86 MJD 58155.541 object MzLS_698229_z 
    mosaic/CP/V4.3/CP20180205/k4m_180206_125911_ooi_zd_ls9.fits.fz
  ksb_170622_075117_ooi_g_ls9-CCD2 g expnum 79260099 exptime 176.652 propid 2015A-0801 seeing 1.50 MJD 57926.327 object 41523 
    90prime/CP/V2.3/CP20170621/ksb_170622_075117_ooi_g_ls9.fits.fz
  ksb_170624_105727_ooi_r_ls9-CCD2 r expnum 79280177 exptime 150.605 propid BASS seeing 1.24 MJD 57928.457 object 41523 
    90prime/CP/V2.3/CP20170623/ksb_170624_105727_ooi_r_ls9.fits.fz
  ksb_160711_103513_ooi_g_ls9-CCD1 g expnum 75800144 exptime 148.2 propid BASS seeing 1.54 MJD 57580.441 object 40632 
    90prime/CP/V2.3/CP20160710/ksb_160711_103513_ooi_g_ls9.fits.fz
  ksb_180519_084839_ooi_r_ls9-CCD1 r expnum 82570157 exptime 71.439 propid NOTBASS seeing 1.10 MJD 58257.367 object 40632 
    90prime/CP/V2.3/CP20180518/ksb_180519_084839_ooi_r_ls9.fits.fz
  ksb_180519_101208_ooi_g_ls9-CCD4 g expnum 82570193 exptime 102.439 propid NOTBASS seeing 1.25 MJD 58257.425 object 41521 
    90prime/CP/V2.3/CP20180518/ksb_180519_101208_ooi_g_ls9.fits.fz
  ksb_180508_104811_ooi_r_ls9-CCD4 r expnum 82460216 exptime 116.385 propid NOTBASS seeing 1.27 MJD 58246.450 object 41521 
    90prime/CP/V2.3/CP20180507/ksb_180508_104811_ooi_r_ls9.fits.fz
mosaic get_fwhm: self.fwhm = 3.47
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 211 459 and amp range 0 2047 and image y 544 795 and amp range 0 2039
Range in image: x 0 249 , y 0 252 (with image size 249 x 252)
dzp 0.007023884 -> scaling image by 1.0064902069278665
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 4.59
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2902 3155 and amp range 2048 4095 and image y 1417 1668 and amp range 0 2039
Range in image: x 0 254 , y 0 252 (with image size 254 x 252)
dzp -0.012875672 -> scaling image by 0.9882111074411305
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 79260099
Faked up EXPNUM 79260099
Faked up EXPNUM 79260099
90prime get_fwhm: self.fwhm = 3.31
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 412 555 and amp range 0 2015 and image y 3373 3516 and amp range 2048 4095
Range in image: x 0 144 , y 0 144 (with image size 144 x 144)
dzp 0.051889952 -> scaling image by 1.0489528829333021
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
mosaic get_fwhm: self.fwhm = 3.31
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 1784 2033 and amp range 0 2005 and image y 828 1076 and amp range 0 2039
Range in image: x 0 222 , y 0 249 (with image size 250 x 249)
dzp 0.0014252509 -> scaling image by 1.0013135665633692
Found overlap: image x 1784 2033 and amp range 2006 4053 and image y 828 1076 and amp range 0 2039
Range in image: x 222 250 , y 0 249 (with image size 250 x 249)
dzp -0.006757846 -> scaling image by 0.993795123967763
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 75800144
Faked up EXPNUM 75800144
Faked up EXPNUM 75800144
90prime get_fwhm: self.fwhm = 3.41
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 614 758 and amp range 0 2015 and image y 3556 3700 and amp range 2048 4095
Range in image: x 0 145 , y 0 145 (with image size 145 x 145)
dzp -0.0051191514 -> scaling image by 0.9952961850385206
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 82570193
Faked up EXPNUM 82570193
Faked up EXPNUM 82570193
90prime get_fwhm: self.fwhm = 2.75
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2434 2577 and amp range 2016 4031 and image y 882 1025 and amp range 0 2047
Range in image: x 0 144 , y 0 144 (with image size 144 x 144)
dzp 0.023266561 -> scaling image by 1.0216605511853067
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 82460216
Faked up EXPNUM 82460216
Faked up EXPNUM 82460216
90prime get_fwhm: self.fwhm = 2.81
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2422 2565 and amp range 2016 4031 and image y 884 1026 and amp range 0 2047
Range in image: x 0 144 , y 0 143 (with image size 144 x 143)
dzp 0.043530058 -> scaling image by 1.0409072255640852
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 82570157
Faked up EXPNUM 82570157
Faked up EXPNUM 82570157
90prime get_fwhm: self.fwhm = 2.43
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 603 748 and amp range 0 2015 and image y 3571 3715 and amp range 2048 4095
Range in image: x 0 146 , y 0 145 (with image size 146 x 145)
dzp -0.05213645 -> scaling image by 0.953115252660837
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Faked up EXPNUM 79280177
Faked up EXPNUM 79280177
Faked up EXPNUM 79280177
90prime get_fwhm: self.fwhm = 2.73
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 409 552 and amp range 0 2015 and image y 3381 3524 and amp range 2048 4095
Range in image: x 0 144 , y 0 144 (with image size 144 x 144)
dzp 0.05525464 -> scaling image by 1.0522086198429996
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Stage tims : Wall: 11.01 s, CPU: 1.34 s, VmPeak: 957 MB, VmSize: 957 MB, VmRSS: 174 MB, VmData: 206 MB, maxrss: 0.178356 MB
Stage tims finished: 2020-08-05T20:12:06.578791
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage tims : 4.850 s worker CPU, 45.894 s worker Wall, Wall: 11.011 s, Cores in use: 0.56, Total efficiency (on 32 cores): 1.8 %
Stage tims finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-tims.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-tims.p
Running stage refs
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage refs at 2020-08-05T20:12:06.802334
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00744.fits
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00597.fits
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00668.fits
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00669.fits
Matched 4 large galaxies to Gaia stars.
Matched 0 large galaxies to Tycho-2 stars.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/metrics/cus/reference-custom-283968p63679.fits
Stage refs : Wall: 7.81 s, CPU: 4.69 s, VmPeak: 1020 MB, VmSize: 1014 MB, VmRSS: 233 MB, VmData: 263 MB, maxrss: 0.238844 MB
Stage refs finished: 2020-08-05T20:12:14.612819
Resources for stage refs : 0.000 s worker CPU, 0.001 s worker Wall, Wall: 7.813 s, Cores in use: 0.60, Total efficiency (on 32 cores): 1.9 %
Stage refs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-refs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-refs.p
Running stage outliers
Running stage outliers at 2020-08-05T20:12:14.633671
Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/metrics/cus/outlier-mask-custom-283968p63679.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/metrics/cus/outliers-pre-custom-283968p63679.jpg
Image ksb_160711_103513_ooi_g_ls9-CCD1 g : masking 147 positive outlier pixels and 0 negative outlier pixels
Image ksb_180519_084839_ooi_r_ls9-CCD1 r : masking 66 positive outlier pixels and 0 negative outlier pixels
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/metrics/cus/outlier-mask-custom-283968p63679.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/metrics/cus/outliers-post-custom-283968p63679.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/metrics/cus/outliers-masked-pos-custom-283968p63679.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/metrics/cus/outliers-masked-neg-custom-283968p63679.jpg
Stage outliers : Wall: 2.73 s, CPU: 1.86 s, VmPeak: 1024 MB, VmSize: 1024 MB, VmRSS: 247 MB, VmData: 273 MB, maxrss: 0.252476 MB
Stage outliers finished: 2020-08-05T20:12:17.368253
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage outliers : 0.778 s worker CPU, 1.468 s worker Wall, Wall: 2.737 s, Cores in use: 0.96, Total efficiency (on 32 cores): 3.0 %
Stage outliers finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-outliers.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-outliers.p
Running stage halos
Running stage halos at 2020-08-05T20:12:17.456276
Stage halos : Wall: 3.21 s, CPU: 3.15 s, VmPeak: 1024 MB, VmSize: 1024 MB, VmRSS: 247 MB, VmData: 274 MB, maxrss: 0.252796 MB
Stage halos finished: 2020-08-05T20:12:20.669024
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 3.215 s, Cores in use: 0.98, Total efficiency (on 32 cores): 3.1 %
Stage halos finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-halos.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-halos.p
Running stage fit_on_coadds
Warning: Stellar halo subtraction is only implemented for DECam
Running stage fit_on_coadds at 2020-08-05T20:12:20.738143
PSF sigmas in coadd pixels: ['2.44', '2.51', '2.03']
Using average PSF sigma 2.3231764864877995
Gaussian PSF norm 0.12142632874196818 vs pixelized 0.102694176
average tim pixel scale / coadd scale: 1.7296033866973664
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-copsf-g.fits.fz
PSF sigmas in coadd pixels: ['2.01', '1.79', '2.07']
Using average PSF sigma 1.9556303441132592
Gaussian PSF norm 0.14424750189780283 vs pixelized 0.12053512
average tim pixel scale / coadd scale: 1.7298821836198104
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-copsf-r.fits.fz
PSF sigmas in coadd pixels: ['1.46', '1.94', '1.40']
Using average PSF sigma 1.5983717892361387
Gaussian PSF norm 0.1764888455073967 vs pixelized 0.13782366
average tim pixel scale / coadd scale: 0.9910767375427575
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-copsf-z.fits.fz
Stage fit_on_coadds : Wall: 0.23 s, CPU: 0.15 s, VmPeak: 1024 MB, VmSize: 1024 MB, VmRSS: 249 MB, VmData: 274 MB, maxrss: 0.25488 MB
Stage fit_on_coadds finished: 2020-08-05T20:12:20.963294
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage fit_on_coadds : 0.167 s worker CPU, 0.167 s worker Wall, Wall: 0.227 s, Cores in use: 1.38, Total efficiency (on 32 cores): 4.3 %
Stage fit_on_coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-fit_on_coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-fit_on_coadds.p
Running stage srcs
Running stage srcs at 2020-08-05T20:12:21.049290
SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])]
Found 19 potential peaks
Of 19 potential peaks: 6 in veto map, 2 cut by saddle test, 0 cut by aper test, 11 kept
SED z : 11 new peaks
Found 19 potential peaks
Of 19 potential peaks: 15 in veto map, 2 cut by saddle test, 0 cut by aper test, 2 kept
SED r : 2 new peaks
Found 17 potential peaks
Of 17 potential peaks: 16 in veto map, 0 cut by saddle test, 0 cut by aper test, 1 kept
SED g : 1 new peaks
Found 27 potential peaks
Of 27 potential peaks: 24 in veto map, 3 cut by saddle test, 0 cut by aper test, 0 kept
SED Flat : 0 new peaks
Found 24 potential peaks
Of 24 potential peaks: 20 in veto map, 4 cut by saddle test, 0 cut by aper test, 0 kept
SED Red : 0 new peaks
Adding 14 new sources
Median coadd sky for g : 5.3601267e-05
Median coadd sky for r : 0.00038072618
Median coadd sky for z : 0.00010552217
Stage srcs : Wall: 2.59 s, CPU: 2.52 s, VmPeak: 1024 MB, VmSize: 1024 MB, VmRSS: 250 MB, VmData: 274 MB, maxrss: 0.255588 MB
Stage srcs finished: 2020-08-05T20:12:23.642435
Resources for stage srcs : 0.068 s worker CPU, 0.068 s worker Wall, Wall: 2.595 s, Cores in use: 1.00, Total efficiency (on 32 cores): 3.1 %
Stage srcs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-srcs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-srcs.p
Running stage fitblobs
Running stage fitblobs at 2020-08-05T20:12:23.657419
Blob 1 of 10, id: 5, sources: 7, size: 106x150, npix 10431, brick X: 77,183, Y: 46,196, one pixel: 167 46
Blob 2 of 10, id: 9, sources: 2, size: 54x70, npix 2275, brick X: 188,242, Y: 147,217, one pixel: 214 147
Blob 3 of 10, id: 3, sources: 2, size: 44x54, npix 1415, brick X: 177,221, Y: 17,71, one pixel: 192 17
Blob 4 of 10, id: 4, sources: 2, size: 52x36, npix 1119, brick X: 19,71, Y: 33,69, one pixel: 49 33
Blob 5 of 10, id: 8, sources: 2, size: 41x40, npix 1063, brick X: 42,83, Y: 119,159, one pixel: 53 119
Blob 6 of 10, id: 6, sources: 1, size: 24x42, npix 735, brick X: 0,24, Y: 54,96, one pixel: 5 54
Blob 7 of 10, id: 0, sources: 1, size: 33x22, npix 530, brick X: 51,84, Y: 0,22, one pixel: 56 0
Blob 8 of 10, id: 1, sources: 1, size: 20x25, npix 335, brick X: 225,245, Y: 4,29, one pixel: 236 4
Blob 9 of 10, id: 2, sources: 1, size: 19x24, npix 313, brick X: 0,19, Y: 13,37, one pixel: 5 13
Blob 10 of 10, id: 7, sources: 1, size: 20x23, npix 294, brick X: 225,245, Y: 91,114, one pixel: 235 91
Blob 9 finished, total: Wall: 0.16 s, CPU: 0.16 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 90 MB, VmData: 138 MB, maxrss: 0.09202 MB
Blob 10 finished, total: Wall: 0.20 s, CPU: 0.19 s, VmPeak: 810 MB, VmSize: 731 MB, VmRSS: 146 MB, VmData: 183 MB, maxrss: 0.14946 MB
Blob 2 finished, total: Wall: 0.35 s, CPU: 0.35 s, VmPeak: 811 MB, VmSize: 731 MB, VmRSS: 146 MB, VmData: 183 MB, maxrss: 0.149952 MB
Blob 8 finished, total: Wall: 0.51 s, CPU: 0.51 s, VmPeak: 666 MB, VmSize: 666 MB, VmRSS: 90 MB, VmData: 137 MB, maxrss: 0.09168 MB
Blob 7 finished, total: Wall: 0.83 s, CPU: 0.81 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 91 MB, VmData: 138 MB, maxrss: 0.09342 MB
Measuring 2 iterative sources
Blob 5 finished, total: Wall: 1.10 s, CPU: 1.10 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 91 MB, VmData: 139 MB, maxrss: 0.093256 MB
Blob 3 finished, total: Wall: 1.41 s, CPU: 1.40 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 91 MB, VmData: 138 MB, maxrss: 0.092952 MB
Measuring 1 iterative sources
Blob 4 finished, total: Wall: 1.77 s, CPU: 1.74 s, VmPeak: 810 MB, VmSize: 732 MB, VmRSS: 148 MB, VmData: 183 MB, maxrss: 0.151156 MB
Blob 6 finished, total: Wall: 1.87 s, CPU: 1.87 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 91 MB, VmData: 139 MB, maxrss: 0.093464 MB
Known galaxy.  Initial shape: EllipseWithPriors(0.25): log r_e=1.09847, ee1=0.28495, ee2=-0.0760802
Measuring 1 iterative sources
Blob 1 finished, total: Wall: 8.03 s, CPU: 8.01 s, VmPeak: 683 MB, VmSize: 683 MB, VmRSS: 110 MB, VmData: 154 MB, maxrss: 0.112412 MB
Old catalog: 23
New catalog: 24
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/metrics/cus/blobs-custom-283968p63679.fits.gz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/metrics/cus/all-models-custom-283968p63679.fits
Stage fitblobs : Wall: 9.26 s, CPU: 0.60 s, VmPeak: 1024 MB, VmSize: 1024 MB, VmRSS: 250 MB, VmData: 274 MB, maxrss: 0.255752 MB
Stage fitblobs finished: 2020-08-05T20:12:32.914879
Resources for stage fitblobs : 16.284 s worker CPU, 16.461 s worker Wall, Wall: 9.260 s, Cores in use: 1.82, Total efficiency (on 32 cores): 5.7 %
Stage fitblobs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-fitblobs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-fitblobs.p
Running stage coadds
Running stage coadds at 2020-08-05T20:12:32.930356
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-ccds.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-image-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-invvar-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-nexp-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-depth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-galdepth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-psfsize-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-model-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-chi2-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-blobmodel-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-image-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-invvar-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-nexp-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-depth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-galdepth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-psfsize-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-model-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-chi2-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-blobmodel-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-image-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-invvar-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-nexp-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-depth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-galdepth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-psfsize-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-model-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-chi2-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-blobmodel-z.fits.fz
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock'
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-depth.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/tmp-legacysurvey-custom-283968p63679-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/tmp-legacysurvey-custom-283968p63679-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/tmp-legacysurvey-custom-283968p63679-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/tmp-legacysurvey-custom-283968p63679-resid.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-resid.jpg
Stage coadds : Wall: 4.17 s, CPU: 1.47 s, VmPeak: 1024 MB, VmSize: 1024 MB, VmRSS: 250 MB, VmData: 274 MB, maxrss: 0.256076 MB
Stage coadds finished: 2020-08-05T20:12:37.098602
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage coadds : 25.636 s worker CPU, 51.070 s worker Wall, Wall: 4.171 s, Cores in use: 6.50, Total efficiency (on 32 cores): 20.3 %
Stage coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-coadds.p
Running stage wise_forced
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage wise_forced at 2020-08-05T20:12:37.113547
Cut to 1 unWISE tiles
Reading WISE tile 2840p636 band 1
Reading WISE tile 2840p636 band 2
Reading WISE tile 2840p636 band 3
Reading WISE tile 2840p636 band 4
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w3-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w3-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w3-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.05 s, CPU: 0.05 s, VmPeak: 870 MB, VmSize: 870 MB, VmRSS: 211 MB, VmData: 238 MB, maxrss: 0.215764 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w2-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w2-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w2-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w2-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.05 s, CPU: 0.05 s, VmPeak: 803 MB, VmSize: 802 MB, VmRSS: 144 MB, VmData: 174 MB, maxrss: 0.205964 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w1-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w1-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w1-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w1-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.05 s, CPU: 0.05 s, VmPeak: 870 MB, VmSize: 870 MB, VmRSS: 208 MB, VmData: 237 MB, maxrss: 0.213088 MB
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w4-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w4-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w4-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.05 s, CPU: 0.05 s, VmPeak: 810 MB, VmSize: 804 MB, VmRSS: 146 MB, VmData: 176 MB, maxrss: 0.186528 MB
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-image-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-model-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-invvar-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-image-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-model-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-invvar-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-image-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-model-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-invvar-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-image-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-model-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-invvar-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/tmp-legacysurvey-custom-283968p63679-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/tmp-legacysurvey-custom-283968p63679-wisemodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-wisemodel.jpg
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Stage wise_forced : Wall: 2.94 s, CPU: 0.82 s, VmPeak: 1024 MB, VmSize: 1024 MB, VmRSS: 251 MB, VmData: 274 MB, maxrss: 0.256588 MB
Stage wise_forced finished: 2020-08-05T20:12:40.049439
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Resources for stage wise_forced : 3.294 s worker CPU, 5.552 s worker Wall, Wall: 2.938 s, Cores in use: 1.40, Total efficiency (on 32 cores): 4.4 %
Stage wise_forced finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-wise_forced.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-wise_forced.p
Running stage writecat
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/homedir/.astropy/cache/download/py3/lock' [astropy.utils.data]
Running stage writecat at 2020-08-05T20:12:40.072831
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/coadd/cus/custom-283968p63679/legacysurvey-custom-283968p63679-maskbits.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/tractor-i/cus/tractor-custom-283968p63679.fits
Reading SFD maps...
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/tractor/cus/tractor-custom-283968p63679.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/tractor/cus/brick-custom-283968p63679.sha256sum
Stage writecat : Wall: 4.55 s, CPU: 2.03 s, VmPeak: 1156 MB, VmSize: 1028 MB, VmRSS: 252 MB, VmData: 274 MB, maxrss: 0.389468 MB
Stage writecat finished: 2020-08-05T20:12:44.623272
Resources for stage writecat : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 4.553 s, Cores in use: 0.45, Total efficiency (on 32 cores): 1.4 %
Stage writecat finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-writecat.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9delta/283/PGC2659520/PGC2659520-largegalaxy-writecat.p
All done: 51.083 s worker CPU, 120.686 s worker Wall, Wall: 49.086 s, Cores in use: 1.42, Total efficiency (on 32 cores): 4.4 %
Finished galaxy PGC2659520 in 0.888 minutes.
Started working on galaxy PGC2659520 at Fri Aug  7 09:24:15 2020
Bands touching this brick, g r z
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 283.9670187702787 63.67954310998241 --width 245 --height 245 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0 

runbrick.py starting at 2020-08-07T09:24:16.154087
legacypipe git version: DR9.5.11-watershed-3-gd509eaaf
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '283.9670187702787', '63.67954310998241', '--width', '245', '--height', '245', '--pixscale', '0.262', '--threads', '16', '--outdir', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520', '--survey-dir', '/global/cfs/cdirs/cosmo/work/legacysurvey/dr9', '--run', 'north', '--skip-calibs', '--checkpoint', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-checkpoint.p', '--pickle', '/global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-%%(stage)s.p', '--force-all', '--fit-on-coadds', '--saddle-fraction', '0.2', '--saddle-min', '4.0']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 283.9670187702787 63.67954310998241 --width 245 --height 245 --pixscale 0.262 --threads 16 --outdir /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520 --survey-dir /global/cfs/cdirs/cosmo/work/legacysurvey/dr9 --run north --skip-calibs --checkpoint /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-checkpoint.p --pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-%%(stage)s.p --force-all --fit-on-coadds --saddle-fraction 0.2 --saddle-min 4.0

Parsed RA,Dec 283.9670187702787 63.67954310998241
Starting process 23113 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059244 MB
Starting process 23115 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 23117 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 23119 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059248 MB
Starting process 23121 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059348 MB
Starting process 23123 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059336 MB
Starting process 23124 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059336 MB
Starting process 23126 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 23129 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.05926 MB
Starting process 23131 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 23133 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059264 MB
Starting process 23135 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059268 MB
Starting process 23137 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 23139 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 23142 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Starting process 23145 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 642 MB, VmSize: 642 MB, VmRSS: 58 MB, VmData: 129 MB, maxrss: 0.059272 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage fit_on_coadds
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-08-07T09:24:16.606653
Keeping 9 CCDs:
  k4m_171029_050735_ooi_zd_ls9-CCD2 z expnum 143937 exptime 125.106 propid 2016A-0453 seeing 0.90 MJD 58055.214 object MzLS_697671_z 
    mosaic/CP/V4.3/CP20171028/k4m_171029_050735_ooi_zd_ls9.fits.fz
  k4m_171002_054418_ooi_zd_ls9-CCD2 z expnum 141841 exptime 153.104 propid 2016A-0453 seeing 1.19 MJD 58028.239 object MzLS_697109_z 
    mosaic/CP/V4.3/CP20171001/k4m_171002_054418_ooi_zd_ls9.fits.fz
  k4m_180206_125911_ooi_zd_ls9-CCD3 z expnum 160867 exptime 84.104 propid 2016A-0453 seeing 0.86 MJD 58155.541 object MzLS_698229_z 
    mosaic/CP/V4.3/CP20180205/k4m_180206_125911_ooi_zd_ls9.fits.fz
  ksb_170622_075117_ooi_g_ls9-CCD2 g expnum 79260099 exptime 176.652 propid 2015A-0801 seeing 1.50 MJD 57926.327 object 41523 
    90prime/CP/V2.3/CP20170621/ksb_170622_075117_ooi_g_ls9.fits.fz
  ksb_170624_105727_ooi_r_ls9-CCD2 r expnum 79280177 exptime 150.605 propid BASS seeing 1.24 MJD 57928.457 object 41523 
    90prime/CP/V2.3/CP20170623/ksb_170624_105727_ooi_r_ls9.fits.fz
  ksb_160711_103513_ooi_g_ls9-CCD1 g expnum 75800144 exptime 148.2 propid BASS seeing 1.54 MJD 57580.441 object 40632 
    90prime/CP/V2.3/CP20160710/ksb_160711_103513_ooi_g_ls9.fits.fz
  ksb_180519_084839_ooi_r_ls9-CCD1 r expnum 82570157 exptime 71.439 propid NOTBASS seeing 1.10 MJD 58257.367 object 40632 
    90prime/CP/V2.3/CP20180518/ksb_180519_084839_ooi_r_ls9.fits.fz
  ksb_180519_101208_ooi_g_ls9-CCD4 g expnum 82570193 exptime 102.439 propid NOTBASS seeing 1.25 MJD 58257.425 object 41521 
    90prime/CP/V2.3/CP20180518/ksb_180519_101208_ooi_g_ls9.fits.fz
  ksb_180508_104811_ooi_r_ls9-CCD4 r expnum 82460216 exptime 116.385 propid NOTBASS seeing 1.27 MJD 58246.450 object 41521 
    90prime/CP/V2.3/CP20180507/ksb_180508_104811_ooi_r_ls9.fits.fz
mosaic get_fwhm: self.fwhm = 3.31
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 1788 2036 and amp range 0 2005 and image y 822 1070 and amp range 0 2039
Range in image: x 0 218 , y 0 249 (with image size 249 x 249)
dzp 0.0014252509 -> scaling image by 1.0013135665633692
Found overlap: image x 1788 2036 and amp range 2006 4053 and image y 822 1070 and amp range 0 2039
Range in image: x 218 249 , y 0 249 (with image size 249 x 249)
dzp -0.006757846 -> scaling image by 0.993795123967763
mosaic get_fwhm: self.fwhm = 4.59
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2906 3159 and amp range 2048 4095 and image y 1411 1662 and amp range 0 2039
Range in image: x 0 254 , y 0 252 (with image size 254 x 252)
dzp -0.012875672 -> scaling image by 0.9882111074411305
mosaic get_fwhm: self.fwhm = 3.47
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 215 463 and amp range 0 2047 and image y 538 789 and amp range 0 2039
Range in image: x 0 249 , y 0 252 (with image size 249 x 252)
dzp 0.007023884 -> scaling image by 1.0064902069278665
Faked up EXPNUM 82460216
Faked up EXPNUM 82460216
Faked up EXPNUM 82460216
90prime get_fwhm: self.fwhm = 2.81
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2420 2563 and amp range 2016 4031 and image y 887 1030 and amp range 0 2047
Range in image: x 0 144 , y 0 144 (with image size 144 x 144)
dzp 0.043530058 -> scaling image by 1.0409072255640852
Faked up EXPNUM 82570157
Faked up EXPNUM 82570157
Faked up EXPNUM 82570157
90prime get_fwhm: self.fwhm = 2.43
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 601 746 and amp range 0 2015 and image y 3575 3719 and amp range 2048 4095
Range in image: x 0 146 , y 0 145 (with image size 146 x 145)
dzp -0.05213645 -> scaling image by 0.953115252660837
Faked up EXPNUM 75800144
Faked up EXPNUM 75800144
Faked up EXPNUM 75800144
90prime get_fwhm: self.fwhm = 3.41
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 611 756 and amp range 0 2015 and image y 3560 3703 and amp range 2048 4095
Range in image: x 0 146 , y 0 144 (with image size 146 x 144)
dzp -0.0051191514 -> scaling image by 0.9952961850385206
Faked up EXPNUM 79260099
Faked up EXPNUM 79260099
Faked up EXPNUM 79260099
90prime get_fwhm: self.fwhm = 3.31
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 410 553 and amp range 0 2015 and image y 3377 3520 and amp range 2048 4095
Range in image: x 0 144 , y 0 144 (with image size 144 x 144)
dzp 0.051889952 -> scaling image by 1.0489528829333021
Faked up EXPNUM 82570193
Faked up EXPNUM 82570193
Faked up EXPNUM 82570193
90prime get_fwhm: self.fwhm = 2.75
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 2431 2575 and amp range 2016 4031 and image y 886 1028 and amp range 0 2047
Range in image: x 0 145 , y 0 143 (with image size 145 x 143)
dzp 0.023266561 -> scaling image by 1.0216605511853067
Faked up EXPNUM 79280177
Faked up EXPNUM 79280177
Faked up EXPNUM 79280177
90prime get_fwhm: self.fwhm = 2.73
Found 4 relevant rows in amp-corrections file.
Found overlap: image x 406 550 and amp range 0 2015 and image y 3384 3528 and amp range 2048 4095
Range in image: x 0 145 , y 0 145 (with image size 145 x 145)
dzp 0.05525464 -> scaling image by 1.0522086198429996
Stage tims : Wall: 58.18 s, CPU: 0.94 s, VmPeak: 951 MB, VmSize: 951 MB, VmRSS: 172 MB, VmData: 206 MB, maxrss: 0.176524 MB
Stage tims finished: 2020-08-07T09:25:14.784907
Resources for stage tims : 4.425 s worker CPU, 295.502 s worker Wall, Wall: 58.180 s, Cores in use: 0.09, Total efficiency (on 16 cores): 0.6 %
Stage tims finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-tims.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-tims.p
Running stage refs
Running stage refs at 2020-08-07T09:25:19.397207
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00744.fits
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00668.fits
Reading /global/cfs/cdirs/cosmo/work/gaia/chunks-gaia-dr2-astrom-2/chunk-00597.fits
Matched 4 large galaxies to Gaia stars.
Matched 0 large galaxies to Tycho-2 stars.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/metrics/cus/reference-custom-283967p63679.fits
Stage refs : Wall: 5.47 s, CPU: 4.33 s, VmPeak: 1004 MB, VmSize: 998 MB, VmRSS: 222 MB, VmData: 253 MB, maxrss: 0.22748 MB
Stage refs finished: 2020-08-07T09:25:24.867618
Resources for stage refs : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 5.472 s, Cores in use: 0.79, Total efficiency (on 16 cores): 4.9 %
Stage refs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-refs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-refs.p
Running stage outliers
Running stage outliers at 2020-08-07T09:25:24.995597
Failed to apply outlier mask: No such file: /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/metrics/cus/outlier-mask-custom-283967p63679.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/metrics/cus/outliers-pre-custom-283967p63679.jpg
Image ksb_160711_103513_ooi_g_ls9-CCD1 g : masking 145 positive outlier pixels and 0 negative outlier pixels
Image ksb_180519_084839_ooi_r_ls9-CCD1 r : masking 67 positive outlier pixels and 0 negative outlier pixels
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/metrics/cus/outlier-mask-custom-283967p63679.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/metrics/cus/outliers-post-custom-283967p63679.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/metrics/cus/outliers-masked-pos-custom-283967p63679.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/metrics/cus/outliers-masked-neg-custom-283967p63679.jpg
Stage outliers : Wall: 0.88 s, CPU: 0.56 s, VmPeak: 1008 MB, VmSize: 1008 MB, VmRSS: 235 MB, VmData: 264 MB, maxrss: 0.241084 MB
Stage outliers finished: 2020-08-07T09:25:25.876180
Resources for stage outliers : 0.626 s worker CPU, 0.772 s worker Wall, Wall: 0.882 s, Cores in use: 1.35, Total efficiency (on 16 cores): 8.4 %
Stage outliers finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-outliers.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-outliers.p
Running stage halos
Running stage halos at 2020-08-07T09:25:25.918402
Stage halos : Wall: 0.30 s, CPU: 0.29 s, VmPeak: 1008 MB, VmSize: 1008 MB, VmRSS: 236 MB, VmData: 264 MB, maxrss: 0.241336 MB
Stage halos finished: 2020-08-07T09:25:26.219878
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage halos : 0.001 s worker CPU, 0.001 s worker Wall, Wall: 0.303 s, Cores in use: 0.97, Total efficiency (on 16 cores): 6.1 %
Stage halos finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-halos.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-halos.p
Running stage fit_on_coadds
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Warning: Stellar halo subtraction is only implemented for DECam
Running stage fit_on_coadds at 2020-08-07T09:25:26.240266
PSF sigmas in coadd pixels: ['2.44', '2.51', '2.03']
Using average PSF sigma 2.3231764864877995
Gaussian PSF norm 0.12142632874196818 vs pixelized 0.10269011
average tim pixel scale / coadd scale: 1.7296033866973664
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-copsf-g.fits.fz
PSF sigmas in coadd pixels: ['2.01', '1.79', '2.07']
Using average PSF sigma 1.9556303441132592
Gaussian PSF norm 0.14424750189780283 vs pixelized 0.12054023
average tim pixel scale / coadd scale: 1.7298821836198104
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-copsf-r.fits.fz
PSF sigmas in coadd pixels: ['1.46', '1.94', '1.40']
Using average PSF sigma 1.5983717892361387
Gaussian PSF norm 0.1764888455073967 vs pixelized 0.13781436
average tim pixel scale / coadd scale: 0.9910767375427575
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-copsf-z.fits.fz
Stage fit_on_coadds : Wall: 0.17 s, CPU: 0.13 s, VmPeak: 1008 MB, VmSize: 1008 MB, VmRSS: 238 MB, VmData: 265 MB, maxrss: 0.24344 MB
Stage fit_on_coadds finished: 2020-08-07T09:25:26.412084
Warning: Stellar halo subtraction is only implemented for DECam
Resources for stage fit_on_coadds : 0.140 s worker CPU, 0.140 s worker Wall, Wall: 0.173 s, Cores in use: 1.58, Total efficiency (on 16 cores): 9.9 %
Stage fit_on_coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-fit_on_coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-fit_on_coadds.p
Running stage srcs
Running stage srcs at 2020-08-07T09:25:26.430733
SED-matched filters: [('z', array([0., 0., 1.])), ('r', array([0., 1., 0.])), ('g', array([1., 0., 0.])), ('Flat', [1.0, 1.0, 1.0]), ('Red', [2.5, 1.0, 0.4])]
Found 20 potential peaks
Of 20 potential peaks: 5 in veto map, 2 cut by saddle test, 0 cut by aper test, 13 kept
SED z : 13 new peaks
Found 18 potential peaks
Of 18 potential peaks: 15 in veto map, 2 cut by saddle test, 0 cut by aper test, 1 kept
SED r : 1 new peaks
Found 17 potential peaks
Of 17 potential peaks: 15 in veto map, 1 cut by saddle test, 0 cut by aper test, 1 kept
SED g : 1 new peaks
Found 26 potential peaks
Of 26 potential peaks: 23 in veto map, 3 cut by saddle test, 0 cut by aper test, 0 kept
SED Flat : 0 new peaks
Found 27 potential peaks
Of 27 potential peaks: 22 in veto map, 4 cut by saddle test, 0 cut by aper test, 1 kept
SED Red : 1 new peaks
Adding 16 new sources
Median coadd sky for g : 6.7862624e-05
Median coadd sky for r : 0.0003838291
Median coadd sky for z : 0.00011672263
Stage srcs : Wall: 0.45 s, CPU: 0.41 s, VmPeak: 1008 MB, VmSize: 1008 MB, VmRSS: 238 MB, VmData: 265 MB, maxrss: 0.244108 MB
Stage srcs finished: 2020-08-07T09:25:26.879839
Resources for stage srcs : 0.078 s worker CPU, 0.078 s worker Wall, Wall: 0.450 s, Cores in use: 1.09, Total efficiency (on 16 cores): 6.8 %
Stage srcs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-srcs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-srcs.p
Running stage fitblobs
Running stage fitblobs at 2020-08-07T09:25:26.929718
Blob 1 of 10, id: 4, sources: 6, size: 106x150, npix 10431, brick X: 71,177, Y: 42,192, one pixel: 161 42
Blob 2 of 10, id: 9, sources: 2, size: 54x70, npix 2277, brick X: 182,236, Y: 143,213, one pixel: 208 143
Blob 3 of 10, id: 2, sources: 2, size: 44x54, npix 1424, brick X: 171,215, Y: 13,67, one pixel: 187 13
Blob 4 of 10, id: 3, sources: 2, size: 52x36, npix 1120, brick X: 13,65, Y: 29,65, one pixel: 43 29
Blob 5 of 10, id: 8, sources: 2, size: 41x40, npix 1063, brick X: 36,77, Y: 115,155, one pixel: 47 115
Blob 6 of 10, id: 1, sources: 3, size: 26x49, npix 670, brick X: 219,245, Y: 1,50, one pixel: 229 1
Blob 7 of 10, id: 5, sources: 2, size: 18x34, npix 456, brick X: 0,18, Y: 54,88, one pixel: 1 54
Blob 8 of 10, id: 0, sources: 1, size: 31x18, npix 409, brick X: 46,77, Y: 0,18, one pixel: 48 0
Blob 9 of 10, id: 6, sources: 1, size: 23x23, npix 326, brick X: 219,242, Y: 87,110, one pixel: 230 87
Blob 10 of 10, id: 7, sources: 1, size: 14x19, npix 178, brick X: 231,245, Y: 113,132, one pixel: 239 113
Blob 9 finished, total: Wall: 0.28 s, CPU: 0.28 s, VmPeak: 805 MB, VmSize: 724 MB, VmRSS: 142 MB, VmData: 182 MB, maxrss: 0.145296 MB
Blob 2 finished, total: Wall: 0.30 s, CPU: 0.30 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 92 MB, VmData: 139 MB, maxrss: 0.093788 MB
Blob 10 finished, total: Wall: 0.59 s, CPU: 0.59 s, VmPeak: 808 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.146208 MB
Dropped reference star: GaiaSource at GaiaPosition: RA, Dec = (283.98749, 63.67582), pm (-2.0, -3.9), parallax 0.527 with NanoMaggies: g=19.1, r=18.2, z=17.3
Reset brightness to NanoMaggies: g=19.1, r=18, z=17.4
Blob 7 finished, total: Wall: 0.63 s, CPU: 0.63 s, VmPeak: 810 MB, VmSize: 725 MB, VmRSS: 142 MB, VmData: 183 MB, maxrss: 0.145896 MB
Blob 6 finished, total: Wall: 0.71 s, CPU: 0.71 s, VmPeak: 667 MB, VmSize: 667 MB, VmRSS: 91 MB, VmData: 139 MB, maxrss: 0.093168 MB
Blob 8 finished, total: Wall: 0.73 s, CPU: 0.73 s, VmPeak: 810 MB, VmSize: 725 MB, VmRSS: 143 MB, VmData: 183 MB, maxrss: 0.146776 MB
Blob 5 finished, total: Wall: 1.08 s, CPU: 1.07 s, VmPeak: 669 MB, VmSize: 669 MB, VmRSS: 93 MB, VmData: 140 MB, maxrss: 0.095256 MB
Blob 3 finished, total: Wall: 1.31 s, CPU: 1.30 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 91 MB, VmData: 139 MB, maxrss: 0.093604 MB
Measuring 1 iterative sources
Blob 4 finished, total: Wall: 1.88 s, CPU: 1.88 s, VmPeak: 668 MB, VmSize: 668 MB, VmRSS: 92 MB, VmData: 139 MB, maxrss: 0.0942 MB
Known galaxy.  Initial shape: EllipseWithPriors(0.25): log r_e=0.615517, ee1=0.264581, ee2=-0.117336
Measuring 2 iterative sources
Blob 1 finished, total: Wall: 7.03 s, CPU: 7.02 s, VmPeak: 807 MB, VmSize: 728 MB, VmRSS: 146 MB, VmData: 183 MB, maxrss: 0.149644 MB
Old catalog: 25
New catalog: 23
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/metrics/cus/blobs-custom-283967p63679.fits.gz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/metrics/cus/all-models-custom-283967p63679.fits
Stage fitblobs : Wall: 7.94 s, CPU: 0.30 s, VmPeak: 1008 MB, VmSize: 1008 MB, VmRSS: 239 MB, VmData: 265 MB, maxrss: 0.244272 MB
Stage fitblobs finished: 2020-08-07T09:25:34.867206
Resources for stage fitblobs : 14.664 s worker CPU, 14.744 s worker Wall, Wall: 7.939 s, Cores in use: 1.88, Total efficiency (on 16 cores): 11.8 %
Stage fitblobs finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-fitblobs.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-fitblobs.p
Running stage coadds
Running stage coadds at 2020-08-07T09:25:34.908539
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-ccds.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-image-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-invvar-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-nexp-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-depth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-galdepth-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-psfsize-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-model-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-chi2-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-blobmodel-g.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-image-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-invvar-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-nexp-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-depth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-galdepth-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-psfsize-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-model-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-chi2-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-blobmodel-r.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-image-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-invvar-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-nexp-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-depth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-galdepth-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-psfsize-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-model-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-chi2-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-blobmodel-z.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-depth.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/tmp-legacysurvey-custom-283967p63679-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-image.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/tmp-legacysurvey-custom-283967p63679-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-model.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/tmp-legacysurvey-custom-283967p63679-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-blobmodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/tmp-legacysurvey-custom-283967p63679-resid.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-resid.jpg
Stage coadds : Wall: 2.35 s, CPU: 0.54 s, VmPeak: 1008 MB, VmSize: 1008 MB, VmRSS: 239 MB, VmData: 265 MB, maxrss: 0.244592 MB
Stage coadds finished: 2020-08-07T09:25:37.261586
Resources for stage coadds : 8.878 s worker CPU, 9.289 s worker Wall, Wall: 2.354 s, Cores in use: 4.00, Total efficiency (on 16 cores): 25.0 %
Stage coadds finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-coadds.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-coadds.p
Running stage wise_forced
Running stage wise_forced at 2020-08-07T09:25:37.295272
Cut to 1 unWISE tiles
Reading WISE tile 2840p636 band 1
Reading WISE tile 2840p636 band 2
Reading WISE tile 2840p636 band 3
Reading WISE tile 2840p636 band 4
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w4-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w4-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w4-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w4-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.04 s, CPU: 0.04 s, VmPeak: 808 MB, VmSize: 802 MB, VmRSS: 147 MB, VmData: 176 MB, maxrss: 0.187956 MB
Does not exist: /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w3-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w3-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w3-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/allwise/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w3-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.04 s, CPU: 0.04 s, VmPeak: 810 MB, VmSize: 792 MB, VmRSS: 136 MB, VmData: 170 MB, maxrss: 0.198776 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w2-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w2-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w2-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w2-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.03 s, CPU: 0.03 s, VmPeak: 810 MB, VmSize: 806 MB, VmRSS: 150 MB, VmData: 180 MB, maxrss: 0.207044 MB
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w1-img-m.fits
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w1-invvar-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w1-n-m.fits.gz
Reading /global/cfs/cdirs/cosmo/data/unwise/neo6/unwise-coadds/fulldepth/284/2840p636/unwise-2840p636-w1-n-u.fits.gz
Reading /usr/local/lib/python/wise/wise-psf-avg.fits
unWISE forced photometry took Wall: 0.04 s, CPU: 0.04 s, VmPeak: 803 MB, VmSize: 795 MB, VmRSS: 140 MB, VmData: 173 MB, maxrss: 0.2025 MB
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-image-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-model-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-invvar-W1.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-image-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-model-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-invvar-W2.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-image-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-model-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-invvar-W3.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-image-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-model-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-invvar-W4.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/tmp-legacysurvey-custom-283967p63679-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-wise.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/tmp-legacysurvey-custom-283967p63679-wisemodel.jpg
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-wisemodel.jpg
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Setting WISE apphot
Stage wise_forced : Wall: 15.79 s, CPU: 0.39 s, VmPeak: 1008 MB, VmSize: 1008 MB, VmRSS: 239 MB, VmData: 265 MB, maxrss: 0.245108 MB
Stage wise_forced finished: 2020-08-07T09:25:53.081958
Resources for stage wise_forced : 2.903 s worker CPU, 13.963 s worker Wall, Wall: 15.788 s, Cores in use: 0.21, Total efficiency (on 16 cores): 1.3 %
Stage wise_forced finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-wise_forced.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-wise_forced.p
Running stage writecat
Running stage writecat at 2020-08-07T09:25:53.113781
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/coadd/cus/custom-283967p63679/legacysurvey-custom-283967p63679-maskbits.fits.fz
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/tractor-i/cus/tractor-custom-283967p63679.fits
Reading SFD maps...
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
END                                                                             
  Rejected (1), keyrecords following the END keyrecord were ignored.
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/tractor/cus/tractor-custom-283967p63679.fits
Wrote /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/tractor/cus/brick-custom-283967p63679.sha256sum
Stage writecat : Wall: 1.59 s, CPU: 0.50 s, VmPeak: 1140 MB, VmSize: 1012 MB, VmRSS: 241 MB, VmData: 265 MB, maxrss: 0.377992 MB
Stage writecat finished: 2020-08-07T09:25:54.707956
Resources for stage writecat : 0.000 s worker CPU, 0.000 s worker Wall, Wall: 1.596 s, Cores in use: 0.32, Total efficiency (on 16 cores): 2.0 %
Stage writecat finished
Saving pickle /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-writecat.p
Saved /global/cscratch1/sd/ioannis/SGA-data-dr9alpha/283/PGC2659520/PGC2659520-largegalaxy-writecat.p
All done: 31.718 s worker CPU, 334.492 s worker Wall, Wall: 98.184 s, Cores in use: 0.41, Total efficiency (on 16 cores): 2.6 %
Finished galaxy PGC2659520 in 1.684 minutes.