Started working on galaxy IC1012 at Sat Feb 8 08:18:19 2020 RA, Dec=216.789632, 30.948235: run=decam (3 north CCDs, 17 south CCDs). python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.7896315 30.9482349 --width 747 --height 747 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-%%(stage)s.p --unwise-coadds runbrick.py starting at 2020-02-08T08:18:20.551431 legacypipe git version: dr9e-pre1-7-g54c974a3 Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '216.7896315', '30.9482349', '--width', '747', '--height', '747', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-%%(stage)s.p', '--unwise-coadds'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.7896315 30.9482349 --width 747 --height 747 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-%%(stage)s.p --unwise-coadds WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky Parsed RA,Dec 216.7896315 30.9482349 WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data] Starting process 38185 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071164 MB Starting process 38186 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB Starting process 38187 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB Starting process 38188 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB Starting process 38189 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB Starting process 38190 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB Starting process 38191 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB Starting process 38192 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage largegalaxies Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-02-08T08:18:34.820182 Keeping 17 CCDs: c4d_160226_090529_ooi_z_ls9.fits.fz c4d_160226_090529_ooi_z_ls9-N1 z exptime 140.0 propid 2014B-0404 seeing 0.96 MJD 57444.377 object DECaLS_38603_z c4d_180224_084701_ooi_r_ls9.fits.fz c4d_180224_084701_ooi_r_ls9-N1 r exptime 86.0 propid 2014B-0404 seeing 1.21 MJD 58173.366 object DECaLS_38603_r c4d_180221_090109_ooi_g_ls9.fits.fz c4d_180221_090109_ooi_g_ls9-N1 g exptime 129.0 propid 2014B-0404 seeing 1.41 MJD 58170.376 object DECaLS_38603_g c4d_180517_031813_ooi_g_ls9.fits.fz c4d_180517_031813_ooi_g_ls9-N9 g exptime 149.0 propid 2014B-0404 seeing 1.32 MJD 58255.138 object DECaLS_6859_g c4d_170406_065019_ooi_g_ls9.fits.fz c4d_170406_065019_ooi_g_ls9-N9 g exptime 175.0 propid 2014B-0404 seeing 1.61 MJD 57849.282 object DECaLS_6859_g c4d_170329_070435_ooi_r_ls9.fits.fz c4d_170329_070435_ooi_r_ls9-N9 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57841.294 object DECaLS_6859_r c4d_160228_082310_ooi_z_ls9.fits.fz c4d_160228_082310_ooi_z_ls9-N9 z exptime 115.0 propid 2014B-0404 seeing 0.89 MJD 57446.348 object DECaLS_6859_z c4d_160226_090529_ooi_z_ls9.fits.fz c4d_160226_090529_ooi_z_ls9-N8 z exptime 140.0 propid 2014B-0404 seeing 1.06 MJD 57444.377 object DECaLS_38603_z c4d_180224_084701_ooi_r_ls9.fits.fz c4d_180224_084701_ooi_r_ls9-N8 r exptime 86.0 propid 2014B-0404 seeing 1.21 MJD 58173.366 object DECaLS_38603_r c4d_180221_090109_ooi_g_ls9.fits.fz c4d_180221_090109_ooi_g_ls9-N8 g exptime 129.0 propid 2014B-0404 seeing 1.37 MJD 58170.376 object DECaLS_38603_g c4d_170503_050055_ooi_r_ls9.fits.fz c4d_170503_050055_ooi_r_ls9-N9 r exptime 112.0 propid 2014B-0404 seeing 1.36 MJD 57876.207 object DECaLS_22731_r c4d_180209_085020_ooi_z_ls9.fits.fz c4d_180209_085020_ooi_z_ls9-N13 z exptime 111.0 propid 2014B-0404 seeing 0.89 MJD 58158.368 object DECaLS_38587_z c4d_180222_085753_ooi_r_ls9.fits.fz c4d_180222_085753_ooi_r_ls9-N13 r exptime 46.0 propid 2014B-0404 seeing 0.97 MJD 58171.374 object DECaLS_38587_r c4d_170421_061044_ooi_z_ls9.fits.fz c4d_170421_061044_ooi_z_ls9-N13 z exptime 213.0 propid 2014B-0404 seeing 1.23 MJD 57864.254 object DECaLS_38587_z c4d_170422_051737_ooi_g_ls9.fits.fz c4d_170422_051737_ooi_g_ls9-N9 g exptime 121.0 propid 2014B-0404 seeing 1.27 MJD 57865.219 object DECaLS_22731_g c4d_160226_081655_ooi_z_ls9.fits.fz c4d_160226_081655_ooi_z_ls9-N9 z exptime 135.0 propid 2014B-0404 seeing 1.10 MJD 57444.343 object DECaLS_22731_z c4d_180221_085437_ooi_g_ls9.fits.fz c4d_180221_085437_ooi_g_ls9-N13 g exptime 161.0 propid 2014B-0404 seeing 1.37 MJD 58170.371 object DECaLS_38587_g The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N8,noinherit][2046,4094][real]: DECaLS_3 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_954f856-z-20160228T082024_oi[N9,noinherit][2046,4094][real]: DECaLS_6 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N1,noinherit][2046,4094][real]: DECaLS_3 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T081350_oi[N9,noinherit][2046,4094][real]: DECaLS_2 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_9527115-z-20180209T085020_oi[N13,noinherit][2046,4094][real]: DECaLS_ The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95130c6-z-20170421T060621_oi[N13,noinherit][2046,4094][real]: DECaLS_ WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N1,noinherit][2046,4094][real]: DECaLS_3 [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N8,noinherit][2046,4094][real]: DECaLS_3 [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_954f856-z-20160228T082024_oi[N9,noinherit][2046,4094][real]: DECaLS_6 [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T081350_oi[N9,noinherit][2046,4094][real]: DECaLS_2 [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_9527115-z-20180209T085020_oi[N13,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95130c6-z-20170421T060621_oi[N13,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in sys.exit(main()) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main run_brick(opt.brick, survey, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick initial_args=initargs, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) [Previous line repeated 6 more times] File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage R = stagefunc(stage, **Px) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc R = stagefunc(stage, mp=mp, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__ rtn = super(CallGlobalTime, self).__call__(stage, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__ return func(*self.args, **kwa) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims tims = list(mp.map(read_one_tim, args)) File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map return self.pool.map(f, args, cs) File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get raise self._value OSError: FITSIO status = 104: could not open the named file failed to find or open the following file: (ffopen) /global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2a/CP20180220/c4d_180221_ 090109_ooi_g_ls9.fits.fz Something went wrong; please check the logfile. ERROR: galaxy IC1012; please check the logfile. Finished galaxy IC1012 in 0.308 minutes. Started working on galaxy IC1012 at Sat Feb 8 08:35:36 2020 RA, Dec=216.789632, 30.948235: run=decam (3 north CCDs, 17 south CCDs). python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.7896315 30.9482349 --width 747 --height 747 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-%%(stage)s.p --unwise-coadds runbrick.py starting at 2020-02-08T08:35:37.011003 legacypipe git version: dr9e-pre1-7-g54c974a3 Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '216.7896315', '30.9482349', '--width', '747', '--height', '747', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-%%(stage)s.p', '--unwise-coadds'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.7896315 30.9482349 --width 747 --height 747 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-%%(stage)s.p --unwise-coadds WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky Parsed RA,Dec 216.7896315 30.9482349 WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data] Starting process 42682 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB Starting process 42683 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB Starting process 42684 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB Starting process 42685 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB Starting process 42686 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB Starting process 42687 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB Starting process 42688 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB Starting process 42689 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage largegalaxies Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-02-08T08:35:38.352629 Keeping 17 CCDs: c4d_160226_090529_ooi_z_ls9.fits.fz c4d_160226_090529_ooi_z_ls9-N1 z exptime 140.0 propid 2014B-0404 seeing 0.96 MJD 57444.377 object DECaLS_38603_z c4d_180224_084701_ooi_r_ls9.fits.fz c4d_180224_084701_ooi_r_ls9-N1 r exptime 86.0 propid 2014B-0404 seeing 1.21 MJD 58173.366 object DECaLS_38603_r c4d_180221_090109_ooi_g_ls9.fits.fz c4d_180221_090109_ooi_g_ls9-N1 g exptime 129.0 propid 2014B-0404 seeing 1.41 MJD 58170.376 object DECaLS_38603_g c4d_180517_031813_ooi_g_ls9.fits.fz c4d_180517_031813_ooi_g_ls9-N9 g exptime 149.0 propid 2014B-0404 seeing 1.32 MJD 58255.138 object DECaLS_6859_g c4d_170406_065019_ooi_g_ls9.fits.fz c4d_170406_065019_ooi_g_ls9-N9 g exptime 175.0 propid 2014B-0404 seeing 1.61 MJD 57849.282 object DECaLS_6859_g c4d_170329_070435_ooi_r_ls9.fits.fz c4d_170329_070435_ooi_r_ls9-N9 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57841.294 object DECaLS_6859_r c4d_160228_082310_ooi_z_ls9.fits.fz c4d_160228_082310_ooi_z_ls9-N9 z exptime 115.0 propid 2014B-0404 seeing 0.89 MJD 57446.348 object DECaLS_6859_z c4d_160226_090529_ooi_z_ls9.fits.fz c4d_160226_090529_ooi_z_ls9-N8 z exptime 140.0 propid 2014B-0404 seeing 1.06 MJD 57444.377 object DECaLS_38603_z c4d_180224_084701_ooi_r_ls9.fits.fz c4d_180224_084701_ooi_r_ls9-N8 r exptime 86.0 propid 2014B-0404 seeing 1.21 MJD 58173.366 object DECaLS_38603_r c4d_180221_090109_ooi_g_ls9.fits.fz c4d_180221_090109_ooi_g_ls9-N8 g exptime 129.0 propid 2014B-0404 seeing 1.37 MJD 58170.376 object DECaLS_38603_g c4d_170503_050055_ooi_r_ls9.fits.fz c4d_170503_050055_ooi_r_ls9-N9 r exptime 112.0 propid 2014B-0404 seeing 1.36 MJD 57876.207 object DECaLS_22731_r c4d_180209_085020_ooi_z_ls9.fits.fz c4d_180209_085020_ooi_z_ls9-N13 z exptime 111.0 propid 2014B-0404 seeing 0.89 MJD 58158.368 object DECaLS_38587_z c4d_180222_085753_ooi_r_ls9.fits.fz c4d_180222_085753_ooi_r_ls9-N13 r exptime 46.0 propid 2014B-0404 seeing 0.97 MJD 58171.374 object DECaLS_38587_r c4d_170421_061044_ooi_z_ls9.fits.fz c4d_170421_061044_ooi_z_ls9-N13 z exptime 213.0 propid 2014B-0404 seeing 1.23 MJD 57864.254 object DECaLS_38587_z c4d_170422_051737_ooi_g_ls9.fits.fz c4d_170422_051737_ooi_g_ls9-N9 g exptime 121.0 propid 2014B-0404 seeing 1.27 MJD 57865.219 object DECaLS_22731_g c4d_160226_081655_ooi_z_ls9.fits.fz c4d_160226_081655_ooi_z_ls9-N9 z exptime 135.0 propid 2014B-0404 seeing 1.10 MJD 57444.343 object DECaLS_22731_z c4d_180221_085437_ooi_g_ls9.fits.fz c4d_180221_085437_ooi_g_ls9-N13 g exptime 161.0 propid 2014B-0404 seeing 1.37 MJD 58170.371 object DECaLS_38587_g The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T081350_oi[N9,noinherit][2046,4094][real]: DECaLS_2 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_954f856-z-20160228T082024_oi[N9,noinherit][2046,4094][real]: DECaLS_6 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95130c6-z-20170421T060621_oi[N13,noinherit][2046,4094][real]: DECaLS_ The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N8,noinherit][2046,4094][real]: DECaLS_3 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N1,noinherit][2046,4094][real]: DECaLS_3 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_9527115-z-20180209T085020_oi[N13,noinherit][2046,4094][real]: DECaLS_ WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T081350_oi[N9,noinherit][2046,4094][real]: DECaLS_2 [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_954f856-z-20160228T082024_oi[N9,noinherit][2046,4094][real]: DECaLS_6 [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95130c6-z-20170421T060621_oi[N13,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N8,noinherit][2046,4094][real]: DECaLS_3 [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N1,noinherit][2046,4094][real]: DECaLS_3 [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_9527115-z-20180209T085020_oi[N13,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in sys.exit(main()) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main run_brick(opt.brick, survey, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick initial_args=initargs, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) [Previous line repeated 6 more times] File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage R = stagefunc(stage, **Px) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc R = stagefunc(stage, mp=mp, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__ rtn = super(CallGlobalTime, self).__call__(stage, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__ return func(*self.args, **kwa) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims tims = list(mp.map(read_one_tim, args)) File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map return self.pool.map(f, args, cs) File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get raise self._value OSError: FITSIO status = 104: could not open the named file failed to find or open the following file: (ffopen) /global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2a/CP20180223/c4d_180224_ 084701_ooi_r_ls9.fits.fz Something went wrong; please check the logfile. ERROR: galaxy IC1012; please check the logfile. Finished galaxy IC1012 in 0.097 minutes. Started working on galaxy IC1012 at Sat Feb 8 12:47:31 2020 RA, Dec=216.789632, 30.948235: run=decam (3 north CCDs, 17 south CCDs). python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.7896315 30.9482349 --width 747 --height 747 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-%%(stage)s.p --unwise-coadds runbrick.py starting at 2020-02-08T12:47:31.778572 legacypipe git version: dr9e-pre1-7-g54c974a3 Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '216.7896315', '30.9482349', '--width', '747', '--height', '747', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-%%(stage)s.p', '--unwise-coadds'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.7896315 30.9482349 --width 747 --height 747 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-%%(stage)s.p --unwise-coadds WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky Parsed RA,Dec 216.7896315 30.9482349 WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data] Starting process 54874 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB Starting process 54875 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB Starting process 54876 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB Starting process 54877 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB Starting process 54878 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB Starting process 54879 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB Starting process 54880 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB Starting process 54881 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage largegalaxies Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-02-08T12:47:48.096324 Keeping 17 CCDs: c4d_160226_090529_ooi_z_ls9.fits.fz c4d_160226_090529_ooi_z_ls9-N1 z exptime 140.0 propid 2014B-0404 seeing 0.96 MJD 57444.377 object DECaLS_38603_z c4d_180224_084701_ooi_r_ls9.fits.fz c4d_180224_084701_ooi_r_ls9-N1 r exptime 86.0 propid 2014B-0404 seeing 1.21 MJD 58173.366 object DECaLS_38603_r c4d_180221_090109_ooi_g_ls9.fits.fz c4d_180221_090109_ooi_g_ls9-N1 g exptime 129.0 propid 2014B-0404 seeing 1.41 MJD 58170.376 object DECaLS_38603_g c4d_180517_031813_ooi_g_ls9.fits.fz c4d_180517_031813_ooi_g_ls9-N9 g exptime 149.0 propid 2014B-0404 seeing 1.32 MJD 58255.138 object DECaLS_6859_g c4d_170406_065019_ooi_g_ls9.fits.fz c4d_170406_065019_ooi_g_ls9-N9 g exptime 175.0 propid 2014B-0404 seeing 1.61 MJD 57849.282 object DECaLS_6859_g c4d_170329_070435_ooi_r_ls9.fits.fz c4d_170329_070435_ooi_r_ls9-N9 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57841.294 object DECaLS_6859_r c4d_160228_082310_ooi_z_ls9.fits.fz c4d_160228_082310_ooi_z_ls9-N9 z exptime 115.0 propid 2014B-0404 seeing 0.89 MJD 57446.348 object DECaLS_6859_z c4d_160226_090529_ooi_z_ls9.fits.fz c4d_160226_090529_ooi_z_ls9-N8 z exptime 140.0 propid 2014B-0404 seeing 1.06 MJD 57444.377 object DECaLS_38603_z c4d_180224_084701_ooi_r_ls9.fits.fz c4d_180224_084701_ooi_r_ls9-N8 r exptime 86.0 propid 2014B-0404 seeing 1.21 MJD 58173.366 object DECaLS_38603_r c4d_180221_090109_ooi_g_ls9.fits.fz c4d_180221_090109_ooi_g_ls9-N8 g exptime 129.0 propid 2014B-0404 seeing 1.37 MJD 58170.376 object DECaLS_38603_g c4d_170503_050055_ooi_r_ls9.fits.fz c4d_170503_050055_ooi_r_ls9-N9 r exptime 112.0 propid 2014B-0404 seeing 1.36 MJD 57876.207 object DECaLS_22731_r c4d_180209_085020_ooi_z_ls9.fits.fz c4d_180209_085020_ooi_z_ls9-N13 z exptime 111.0 propid 2014B-0404 seeing 0.89 MJD 58158.368 object DECaLS_38587_z c4d_180222_085753_ooi_r_ls9.fits.fz c4d_180222_085753_ooi_r_ls9-N13 r exptime 46.0 propid 2014B-0404 seeing 0.97 MJD 58171.374 object DECaLS_38587_r c4d_170421_061044_ooi_z_ls9.fits.fz c4d_170421_061044_ooi_z_ls9-N13 z exptime 213.0 propid 2014B-0404 seeing 1.23 MJD 57864.254 object DECaLS_38587_z c4d_170422_051737_ooi_g_ls9.fits.fz c4d_170422_051737_ooi_g_ls9-N9 g exptime 121.0 propid 2014B-0404 seeing 1.27 MJD 57865.219 object DECaLS_22731_g c4d_160226_081655_ooi_z_ls9.fits.fz c4d_160226_081655_ooi_z_ls9-N9 z exptime 135.0 propid 2014B-0404 seeing 1.10 MJD 57444.343 object DECaLS_22731_z c4d_180221_085437_ooi_g_ls9.fits.fz c4d_180221_085437_ooi_g_ls9-N13 g exptime 161.0 propid 2014B-0404 seeing 1.37 MJD 58170.371 object DECaLS_38587_g The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T081350_oi[N9,noinherit][2046,4094][real]: DECaLS_2 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N8,noinherit][2046,4094][real]: DECaLS_3 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N1,noinherit][2046,4094][real]: DECaLS_3 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_954f856-z-20160228T082024_oi[N9,noinherit][2046,4094][real]: DECaLS_6 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95130c6-z-20170421T060621_oi[N13,noinherit][2046,4094][real]: DECaLS_ The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_9527115-z-20180209T085020_oi[N13,noinherit][2046,4094][real]: DECaLS_ WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T081350_oi[N9,noinherit][2046,4094][real]: DECaLS_2 [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N8,noinherit][2046,4094][real]: DECaLS_3 [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N1,noinherit][2046,4094][real]: DECaLS_3 [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95130c6-z-20170421T060621_oi[N13,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_954f856-z-20160228T082024_oi[N9,noinherit][2046,4094][real]: DECaLS_6 [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_9527115-z-20180209T085020_oi[N13,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in sys.exit(main()) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main run_brick(opt.brick, survey, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick initial_args=initargs, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) [Previous line repeated 6 more times] File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage R = stagefunc(stage, **Px) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc R = stagefunc(stage, mp=mp, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__ rtn = super(CallGlobalTime, self).__call__(stage, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__ return func(*self.args, **kwa) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims tims = list(mp.map(read_one_tim, args)) File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map return self.pool.map(f, args, cs) File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get raise self._value OSError: FITSIO status = 104: could not open the named file failed to find or open the following file: (ffopen) /global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2a/CP20180223/c4d_180224_ 084701_ooi_r_ls9.fits.fz Something went wrong; please check the logfile. ERROR: galaxy IC1012; please check the logfile. Finished galaxy IC1012 in 0.337 minutes. Started working on galaxy IC1012 at Sun Feb 9 09:30:26 2020 RA, Dec=216.789632, 30.948235: run=decam (3 north CCDs, 17 south CCDs). python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.7896315 30.9482349 --width 747 --height 747 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-%%(stage)s.p --unwise-coadds runbrick.py starting at 2020-02-09T09:30:26.890430 legacypipe git version: dr9e-pre1-7-g54c974a3 Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '216.7896315', '30.9482349', '--width', '747', '--height', '747', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-%%(stage)s.p', '--unwise-coadds'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.7896315 30.9482349 --width 747 --height 747 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-%%(stage)s.p --unwise-coadds WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky Parsed RA,Dec 216.7896315 30.9482349 WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data] Starting process 32751 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB Starting process 32752 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB Starting process 32753 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB Starting process 32754 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB Starting process 32755 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB Starting process 32756 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB Starting process 32757 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB Starting process 32758 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB Starting process 32759 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB Starting process 32760 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB Starting process 32761 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB Starting process 32762 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB Starting process 32763 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB Starting process 32764 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB Starting process 32765 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB Starting process 32766 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB Starting process 32767 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071192 MB Starting process 32768 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072204 MB Starting process 32769 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072204 MB Starting process 32770 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071912 MB Starting process 32771 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071912 MB Starting process 32772 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB Starting process 32773 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB Starting process 32774 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB Starting process 32775 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB Starting process 32776 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB Starting process 32777 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB Starting process 32778 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB Starting process 32779 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB Starting process 32780 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071484 MB Starting process 32781 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB Starting process 32782 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage largegalaxies Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-02-09T09:30:28.115596 Keeping 17 CCDs: c4d_160226_090529_ooi_z_ls9.fits.fz c4d_160226_090529_ooi_z_ls9-N1 z exptime 140.0 propid 2014B-0404 seeing 0.96 MJD 57444.377 object DECaLS_38603_z c4d_180224_084701_ooi_r_ls9.fits.fz c4d_180224_084701_ooi_r_ls9-N1 r exptime 86.0 propid 2014B-0404 seeing 1.21 MJD 58173.366 object DECaLS_38603_r c4d_180221_090109_ooi_g_ls9.fits.fz c4d_180221_090109_ooi_g_ls9-N1 g exptime 129.0 propid 2014B-0404 seeing 1.41 MJD 58170.376 object DECaLS_38603_g c4d_180517_031813_ooi_g_ls9.fits.fz c4d_180517_031813_ooi_g_ls9-N9 g exptime 149.0 propid 2014B-0404 seeing 1.32 MJD 58255.138 object DECaLS_6859_g c4d_170406_065019_ooi_g_ls9.fits.fz c4d_170406_065019_ooi_g_ls9-N9 g exptime 175.0 propid 2014B-0404 seeing 1.61 MJD 57849.282 object DECaLS_6859_g c4d_170329_070435_ooi_r_ls9.fits.fz c4d_170329_070435_ooi_r_ls9-N9 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57841.294 object DECaLS_6859_r c4d_160228_082310_ooi_z_ls9.fits.fz c4d_160228_082310_ooi_z_ls9-N9 z exptime 115.0 propid 2014B-0404 seeing 0.89 MJD 57446.348 object DECaLS_6859_z c4d_160226_090529_ooi_z_ls9.fits.fz c4d_160226_090529_ooi_z_ls9-N8 z exptime 140.0 propid 2014B-0404 seeing 1.06 MJD 57444.377 object DECaLS_38603_z c4d_180224_084701_ooi_r_ls9.fits.fz c4d_180224_084701_ooi_r_ls9-N8 r exptime 86.0 propid 2014B-0404 seeing 1.21 MJD 58173.366 object DECaLS_38603_r c4d_180221_090109_ooi_g_ls9.fits.fz c4d_180221_090109_ooi_g_ls9-N8 g exptime 129.0 propid 2014B-0404 seeing 1.37 MJD 58170.376 object DECaLS_38603_g c4d_170503_050055_ooi_r_ls9.fits.fz c4d_170503_050055_ooi_r_ls9-N9 r exptime 112.0 propid 2014B-0404 seeing 1.36 MJD 57876.207 object DECaLS_22731_r c4d_180209_085020_ooi_z_ls9.fits.fz c4d_180209_085020_ooi_z_ls9-N13 z exptime 111.0 propid 2014B-0404 seeing 0.89 MJD 58158.368 object DECaLS_38587_z c4d_180222_085753_ooi_r_ls9.fits.fz c4d_180222_085753_ooi_r_ls9-N13 r exptime 46.0 propid 2014B-0404 seeing 0.97 MJD 58171.374 object DECaLS_38587_r c4d_170421_061044_ooi_z_ls9.fits.fz c4d_170421_061044_ooi_z_ls9-N13 z exptime 213.0 propid 2014B-0404 seeing 1.23 MJD 57864.254 object DECaLS_38587_z c4d_170422_051737_ooi_g_ls9.fits.fz c4d_170422_051737_ooi_g_ls9-N9 g exptime 121.0 propid 2014B-0404 seeing 1.27 MJD 57865.219 object DECaLS_22731_g c4d_160226_081655_ooi_z_ls9.fits.fz c4d_160226_081655_ooi_z_ls9-N9 z exptime 135.0 propid 2014B-0404 seeing 1.10 MJD 57444.343 object DECaLS_22731_z c4d_180221_085437_ooi_g_ls9.fits.fz c4d_180221_085437_ooi_g_ls9-N13 g exptime 161.0 propid 2014B-0404 seeing 1.37 MJD 58170.371 object DECaLS_38587_g The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N8,noinherit][2046,4094][real]: DECaLS_3 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_9527115-z-20180209T085020_oi[N13,noinherit][2046,4094][real]: DECaLS_ The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N1,noinherit][2046,4094][real]: DECaLS_3 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95130c6-z-20170421T060621_oi[N13,noinherit][2046,4094][real]: DECaLS_ The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T081350_oi[N9,noinherit][2046,4094][real]: DECaLS_2 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_954f856-z-20160228T082024_oi[N9,noinherit][2046,4094][real]: DECaLS_6 WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N8,noinherit][2046,4094][real]: DECaLS_3 [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_9527115-z-20180209T085020_oi[N13,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N1,noinherit][2046,4094][real]: DECaLS_3 [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95130c6-z-20170421T060621_oi[N13,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T081350_oi[N9,noinherit][2046,4094][real]: DECaLS_2 [astropy.io.fits.card] Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in sys.exit(main()) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main run_brick(opt.brick, survey, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick initial_args=initargs, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) [Previous line repeated 6 more times] File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage R = stagefunc(stage, **Px) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc R = stagefunc(stage, mp=mp, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__ rtn = super(CallGlobalTime, self).__call__(stage, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__ return func(*self.args, **kwa) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims tims = list(mp.map(read_one_tim, args)) File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map return self.pool.map(f, args, cs) File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get raise self._value OSError: FITSIO status = 104: could not open the named file failed to find or open the following file: (ffopen) /global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2a/CP20180223/c4d_180224_ 084701_ooi_r_ls9.fits.fz Something went wrong; please check the logfile. ERROR: galaxy IC1012; please check the logfile. Finished galaxy IC1012 in 0.059 minutes. Started working on galaxy IC1012 at Sun Feb 9 10:21:30 2020 RA, Dec=216.789632, 30.948235: run=decam (3 north CCDs, 17 south CCDs). python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.7896315 30.9482349 --width 747 --height 747 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-%%(stage)s.p --unwise-coadds runbrick.py starting at 2020-02-09T10:21:31.860343 legacypipe git version: dr9e-pre1-7-g54c974a3 Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '216.7896315', '30.9482349', '--width', '747', '--height', '747', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-%%(stage)s.p', '--unwise-coadds'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.7896315 30.9482349 --width 747 --height 747 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-%%(stage)s.p --unwise-coadds WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky Parsed RA,Dec 216.7896315 30.9482349 WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data] Starting process 43596 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07116 MB Starting process 43597 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB Starting process 43598 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB Starting process 43599 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB Starting process 43600 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB Starting process 43601 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB Starting process 43602 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB Starting process 43603 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB Starting process 43604 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB Starting process 43605 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB Starting process 43606 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB Starting process 43607 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB Starting process 43608 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB Starting process 43609 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB Starting process 43610 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB Starting process 43611 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB Starting process 43612 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071188 MB Starting process 43613 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.0722 MB Starting process 43614 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.0722 MB Starting process 43615 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071912 MB Starting process 43616 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071904 MB Starting process 43617 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB Starting process 43618 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07154 MB Starting process 43619 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07154 MB Starting process 43620 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07154 MB Starting process 43621 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB Starting process 43622 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB Starting process 43623 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB Starting process 43624 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07148 MB Starting process 43625 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07148 MB Starting process 43626 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07148 MB Starting process 43627 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07148 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage largegalaxies Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-02-09T10:21:33.829399 Keeping 17 CCDs: c4d_160226_090529_ooi_z_ls9.fits.fz c4d_160226_090529_ooi_z_ls9-N1 z exptime 140.0 propid 2014B-0404 seeing 0.96 MJD 57444.377 object DECaLS_38603_z c4d_180224_084701_ooi_r_ls9.fits.fz c4d_180224_084701_ooi_r_ls9-N1 r exptime 86.0 propid 2014B-0404 seeing 1.21 MJD 58173.366 object DECaLS_38603_r c4d_180221_090109_ooi_g_ls9.fits.fz c4d_180221_090109_ooi_g_ls9-N1 g exptime 129.0 propid 2014B-0404 seeing 1.41 MJD 58170.376 object DECaLS_38603_g c4d_180517_031813_ooi_g_ls9.fits.fz c4d_180517_031813_ooi_g_ls9-N9 g exptime 149.0 propid 2014B-0404 seeing 1.32 MJD 58255.138 object DECaLS_6859_g c4d_170406_065019_ooi_g_ls9.fits.fz c4d_170406_065019_ooi_g_ls9-N9 g exptime 175.0 propid 2014B-0404 seeing 1.61 MJD 57849.282 object DECaLS_6859_g c4d_170329_070435_ooi_r_ls9.fits.fz c4d_170329_070435_ooi_r_ls9-N9 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57841.294 object DECaLS_6859_r c4d_160228_082310_ooi_z_ls9.fits.fz c4d_160228_082310_ooi_z_ls9-N9 z exptime 115.0 propid 2014B-0404 seeing 0.89 MJD 57446.348 object DECaLS_6859_z c4d_160226_090529_ooi_z_ls9.fits.fz c4d_160226_090529_ooi_z_ls9-N8 z exptime 140.0 propid 2014B-0404 seeing 1.06 MJD 57444.377 object DECaLS_38603_z c4d_180224_084701_ooi_r_ls9.fits.fz c4d_180224_084701_ooi_r_ls9-N8 r exptime 86.0 propid 2014B-0404 seeing 1.21 MJD 58173.366 object DECaLS_38603_r c4d_180221_090109_ooi_g_ls9.fits.fz c4d_180221_090109_ooi_g_ls9-N8 g exptime 129.0 propid 2014B-0404 seeing 1.37 MJD 58170.376 object DECaLS_38603_g c4d_170503_050055_ooi_r_ls9.fits.fz c4d_170503_050055_ooi_r_ls9-N9 r exptime 112.0 propid 2014B-0404 seeing 1.36 MJD 57876.207 object DECaLS_22731_r c4d_180209_085020_ooi_z_ls9.fits.fz c4d_180209_085020_ooi_z_ls9-N13 z exptime 111.0 propid 2014B-0404 seeing 0.89 MJD 58158.368 object DECaLS_38587_z c4d_180222_085753_ooi_r_ls9.fits.fz c4d_180222_085753_ooi_r_ls9-N13 r exptime 46.0 propid 2014B-0404 seeing 0.97 MJD 58171.374 object DECaLS_38587_r c4d_170421_061044_ooi_z_ls9.fits.fz c4d_170421_061044_ooi_z_ls9-N13 z exptime 213.0 propid 2014B-0404 seeing 1.23 MJD 57864.254 object DECaLS_38587_z c4d_170422_051737_ooi_g_ls9.fits.fz c4d_170422_051737_ooi_g_ls9-N9 g exptime 121.0 propid 2014B-0404 seeing 1.27 MJD 57865.219 object DECaLS_22731_g c4d_160226_081655_ooi_z_ls9.fits.fz c4d_160226_081655_ooi_z_ls9-N9 z exptime 135.0 propid 2014B-0404 seeing 1.10 MJD 57444.343 object DECaLS_22731_z c4d_180221_085437_ooi_g_ls9.fits.fz c4d_180221_085437_ooi_g_ls9-N13 g exptime 161.0 propid 2014B-0404 seeing 1.37 MJD 58170.371 object DECaLS_38587_g The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_9527115-z-20180209T085020_oi[N13,noinherit][2046,4094][real]: DECaLS_ The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N8,noinherit][2046,4094][real]: DECaLS_3 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95130c6-z-20170421T060621_oi[N13,noinherit][2046,4094][real]: DECaLS_ The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_954f856-z-20160228T082024_oi[N9,noinherit][2046,4094][real]: DECaLS_6 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T081350_oi[N9,noinherit][2046,4094][real]: DECaLS_2 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N1,noinherit][2046,4094][real]: DECaLS_3 WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N8,noinherit][2046,4094][real]: DECaLS_3 [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_9527115-z-20180209T085020_oi[N13,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95130c6-z-20170421T060621_oi[N13,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T081350_oi[N9,noinherit][2046,4094][real]: DECaLS_2 [astropy.io.fits.card] Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in sys.exit(main()) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main run_brick(opt.brick, survey, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick initial_args=initargs, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) [Previous line repeated 6 more times] File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage R = stagefunc(stage, **Px) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc R = stagefunc(stage, mp=mp, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__ rtn = super(CallGlobalTime, self).__call__(stage, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__ return func(*self.args, **kwa) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims tims = list(mp.map(read_one_tim, args)) File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map return self.pool.map(f, args, cs) File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get raise self._value OSError: FITSIO status = 104: could not open the named file failed to find or open the following file: (ffopen) /global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2a/CP20180223/c4d_180224_ 084701_ooi_r_ls9.fits.fz Something went wrong; please check the logfile. ERROR: galaxy IC1012; please check the logfile. Finished galaxy IC1012 in 0.080 minutes. Started working on galaxy IC1012 at Mon Feb 10 06:17:02 2020 RA, Dec=216.789632, 30.948235: run=decam (3 north CCDs, 17 south CCDs). python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.7896315 30.9482349 --width 747 --height 747 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-%%(stage)s.p --unwise-coadds runbrick.py starting at 2020-02-10T06:17:03.599466 legacypipe git version: dr9e-pre1-9-g4a51be93 Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '216.7896315', '30.9482349', '--width', '747', '--height', '747', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-%%(stage)s.p', '--unwise-coadds'] python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.7896315 30.9482349 --width 747 --height 747 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/IC1012/IC1012-runbrick-%%(stage)s.p --unwise-coadds WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky Parsed RA,Dec 216.7896315 30.9482349 WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data] Starting process 14987 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071128 MB Starting process 14988 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071128 MB Starting process 14989 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071128 MB Starting process 14990 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071128 MB Starting process 14991 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071128 MB Starting process 14992 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071132 MB Starting process 14993 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071136 MB Starting process 14994 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07114 MB Starting process 14995 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07114 MB Starting process 14996 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07114 MB Starting process 14997 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071144 MB Starting process 14998 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071144 MB Starting process 14999 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071148 MB Starting process 15000 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071148 MB Starting process 15001 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071148 MB Starting process 15002 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071148 MB Starting process 15003 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071152 MB Starting process 15004 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072164 MB Starting process 15005 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072164 MB Starting process 15006 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071876 MB Starting process 15007 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071872 MB Starting process 15008 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071508 MB Starting process 15009 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB Starting process 15010 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB Starting process 15011 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB Starting process 15012 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB Starting process 15013 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB Starting process 15014 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB Starting process 15015 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB Starting process 15016 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB Starting process 15017 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB Starting process 15018 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB Runstage writecat Runstage wise_forced Runstage coadds Runstage fitblobs Runstage srcs Runstage largegalaxies Runstage halos Runstage outliers Runstage refs Runstage tims Running stage tims Running stage tims at 2020-02-10T06:17:04.857919 Keeping 17 CCDs: c4d_160226_090529_ooi_z_ls9.fits.fz c4d_160226_090529_ooi_z_ls9-N1 z exptime 140.0 propid 2014B-0404 seeing 0.96 MJD 57444.377 object DECaLS_38603_z c4d_180224_084701_ooi_r_ls9.fits.fz c4d_180224_084701_ooi_r_ls9-N1 r exptime 86.0 propid 2014B-0404 seeing 1.21 MJD 58173.366 object DECaLS_38603_r c4d_180221_090109_ooi_g_ls9.fits.fz c4d_180221_090109_ooi_g_ls9-N1 g exptime 129.0 propid 2014B-0404 seeing 1.41 MJD 58170.376 object DECaLS_38603_g c4d_180517_031813_ooi_g_ls9.fits.fz c4d_180517_031813_ooi_g_ls9-N9 g exptime 149.0 propid 2014B-0404 seeing 1.32 MJD 58255.138 object DECaLS_6859_g c4d_170406_065019_ooi_g_ls9.fits.fz c4d_170406_065019_ooi_g_ls9-N9 g exptime 175.0 propid 2014B-0404 seeing 1.61 MJD 57849.282 object DECaLS_6859_g c4d_170329_070435_ooi_r_ls9.fits.fz c4d_170329_070435_ooi_r_ls9-N9 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57841.294 object DECaLS_6859_r c4d_160228_082310_ooi_z_ls9.fits.fz c4d_160228_082310_ooi_z_ls9-N9 z exptime 115.0 propid 2014B-0404 seeing 0.89 MJD 57446.348 object DECaLS_6859_z c4d_160226_090529_ooi_z_ls9.fits.fz c4d_160226_090529_ooi_z_ls9-N8 z exptime 140.0 propid 2014B-0404 seeing 1.06 MJD 57444.377 object DECaLS_38603_z c4d_180224_084701_ooi_r_ls9.fits.fz c4d_180224_084701_ooi_r_ls9-N8 r exptime 86.0 propid 2014B-0404 seeing 1.21 MJD 58173.366 object DECaLS_38603_r c4d_180221_090109_ooi_g_ls9.fits.fz c4d_180221_090109_ooi_g_ls9-N8 g exptime 129.0 propid 2014B-0404 seeing 1.37 MJD 58170.376 object DECaLS_38603_g c4d_170503_050055_ooi_r_ls9.fits.fz c4d_170503_050055_ooi_r_ls9-N9 r exptime 112.0 propid 2014B-0404 seeing 1.36 MJD 57876.207 object DECaLS_22731_r c4d_180209_085020_ooi_z_ls9.fits.fz c4d_180209_085020_ooi_z_ls9-N13 z exptime 111.0 propid 2014B-0404 seeing 0.89 MJD 58158.368 object DECaLS_38587_z c4d_180222_085753_ooi_r_ls9.fits.fz c4d_180222_085753_ooi_r_ls9-N13 r exptime 46.0 propid 2014B-0404 seeing 0.97 MJD 58171.374 object DECaLS_38587_r c4d_170421_061044_ooi_z_ls9.fits.fz c4d_170421_061044_ooi_z_ls9-N13 z exptime 213.0 propid 2014B-0404 seeing 1.23 MJD 57864.254 object DECaLS_38587_z c4d_170422_051737_ooi_g_ls9.fits.fz c4d_170422_051737_ooi_g_ls9-N9 g exptime 121.0 propid 2014B-0404 seeing 1.27 MJD 57865.219 object DECaLS_22731_g c4d_160226_081655_ooi_z_ls9.fits.fz c4d_160226_081655_ooi_z_ls9-N9 z exptime 135.0 propid 2014B-0404 seeing 1.10 MJD 57444.343 object DECaLS_22731_z c4d_180221_085437_ooi_g_ls9.fits.fz c4d_180221_085437_ooi_g_ls9-N13 g exptime 161.0 propid 2014B-0404 seeing 1.37 MJD 58170.371 object DECaLS_38587_g The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N8,noinherit][2046,4094][real]: DECaLS_3 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_9527115-z-20180209T085020_oi[N13,noinherit][2046,4094][real]: DECaLS_ The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95130c6-z-20170421T060621_oi[N13,noinherit][2046,4094][real]: DECaLS_ The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_954f856-z-20160228T082024_oi[N9,noinherit][2046,4094][real]: DECaLS_6 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N1,noinherit][2046,4094][real]: DECaLS_3 The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T081350_oi[N9,noinherit][2046,4094][real]: DECaLS_2 WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N8,noinherit][2046,4094][real]: DECaLS_3 [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95130c6-z-20170421T060621_oi[N13,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_9527115-z-20180209T085020_oi[N13,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_954f856-z-20160228T082024_oi[N9,noinherit][2046,4094][real]: DECaLS_6 [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T090219_oi[N1,noinherit][2046,4094][real]: DECaLS_3 [astropy.io.fits.card] WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention: DECALS_DR9_95508ff-z-20160226T081350_oi[N9,noinherit][2046,4094][real]: DECaLS_2 [astropy.io.fits.card] Traceback (most recent call last): File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in sys.exit(main()) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main run_brick(opt.brick, survey, **kwargs) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick initial_args=initargs, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage write=write, initial_args=initial_args, **kwargs) [Previous line repeated 6 more times] File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage R = stagefunc(stage, **Px) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc R = stagefunc(stage, mp=mp, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__ rtn = super(CallGlobalTime, self).__call__(stage, **kwargs) File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__ return func(*self.args, **kwa) File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims tims = list(mp.map(read_one_tim, args)) File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map return self.pool.map(f, args, cs) File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map return self._map_async(func, iterable, mapstar, chunksize).get() File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get raise self._value OSError: FITSIO status = 104: could not open the named file failed to find or open the following file: (ffopen) /global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2a/CP20180223/c4d_180224_ 084701_ooi_r_ls9.fits.fz Something went wrong; please check the logfile. ERROR: galaxy IC1012; please check the logfile. Finished galaxy IC1012 in 0.061 minutes.