Started working on galaxy PGC1898150 at Sat Feb  8 08:14:12 2020
RA, Dec=218.829702, 30.296722: run=decam (2 north CCDs, 16 south CCDs).
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.829702 30.2967217 --width 451 --height 451 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T08:14:13.295844
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '218.829702', '30.2967217', '--width', '451', '--height', '451', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.829702 30.2967217 --width 451 --height 451 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 218.829702 30.2967217
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 38026 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 38027 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 38028 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 38029 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 38030 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 38031 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 38032 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 38033 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T08:14:14.553720
Keeping 16 CCDs:
   c4d_170331_065325_ooi_r_ls9.fits.fz c4d_170331_065325_ooi_r_ls9-N25 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57843.286 object DECaLS_22744_r
   c4d_170331_065648_ooi_g_ls9.fits.fz c4d_170331_065648_ooi_g_ls9-N25 g exptime 175.0 propid 2014B-0404 seeing 1.39 MJD 57843.287 object DECaLS_22744_g
   c4d_160226_082233_ooi_z_ls9.fits.fz c4d_160226_082233_ooi_z_ls9-N25 z exptime 142.0 propid 2014B-0404 seeing 1.08 MJD 57444.347 object DECaLS_22744_z
   c4d_180210_082602_ooi_z_ls9.fits.fz c4d_180210_082602_ooi_z_ls9-S24 z exptime 138.0 propid 2014B-0404 seeing 1.36 MJD 58159.351 object DECaLS_6858_z
   c4d_170329_071015_ooi_g_ls9.fits.fz c4d_170329_071015_ooi_g_ls9-S24 g exptime 132.0 propid 2014B-0404 seeing 1.51 MJD 57841.297 object DECaLS_6858_g
   c4d_170329_070735_ooi_r_ls9.fits.fz c4d_170329_070735_ooi_r_ls9-S24 r exptime 60.0 propid 2014B-0404 seeing 1.30 MJD 57841.296 object DECaLS_6858_r
   c4d_180209_091446_ooi_z_ls9.fits.fz c4d_180209_091446_ooi_z_ls9-S24 z exptime 113.0 propid 2014B-0404 seeing 1.13 MJD 58158.385 object DECaLS_6858_z
   c4d_170306_093651_ooi_z_ls9.fits.fz c4d_170306_093651_ooi_z_ls9-S27 z exptime 99.0 propid 2014B-0404 seeing 0.95 MJD 57818.399 object DECaLS_38602_z
   c4d_180224_084906_ooi_g_ls9.fits.fz c4d_180224_084906_ooi_g_ls9-S27 g exptime 128.0 propid 2014B-0404 seeing 1.41 MJD 58173.367 object DECaLS_38602_g
   c4d_180225_081201_ooi_r_ls9.fits.fz c4d_180225_081201_ooi_r_ls9-S27 r exptime 88.0 propid 2014B-0404 seeing 1.18 MJD 58174.342 object DECaLS_38602_r
   c4d_170422_050829_ooi_g_ls9.fits.fz c4d_170422_050829_ooi_g_ls9-N29 g exptime 175.0 propid 2014B-0404 seeing 1.27 MJD 57865.212 object DECaLS_6872_g
   c4d_170329_070606_ooi_r_ls9.fits.fz c4d_170329_070606_ooi_r_ls9-N29 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57841.295 object DECaLS_6872_r
   c4d_160228_083710_ooi_z_ls9.fits.fz c4d_160228_083710_ooi_z_ls9-N29 z exptime 141.0 propid 2014B-0404 seeing 0.90 MJD 57446.357 object DECaLS_6872_z
   c4d_150408_060636_ooi_z_ls9.fits.fz c4d_150408_060636_ooi_z_ls9-S28 z exptime 109.0 propid 2014B-0404 seeing 1.25 MJD 57120.253 object DECaLS_22730_z
   c4d_170718_005053_ooi_r_ls9.fits.fz c4d_170718_005053_ooi_r_ls9-S28 r exptime 113.0 propid 2014B-0404 seeing 1.57 MJD 57952.033 object DECaLS_22730_r
   c4d_170421_055123_ooi_g_ls9.fits.fz c4d_170421_055123_ooi_g_ls9-S28 g exptime 175.0 propid 2014B-0404 seeing 1.53 MJD 57864.241 object DECaLS_22730_g
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081923_oi[N25,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T083400_oi[N29,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9527115-z-20180209T091446_oi[S24,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_950ea1d-z-20170306T093421_oi[S27,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9577b3f-z-20150408T060358_oi[S28,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9527115-z-20180210T082602_oi[S24,noinherit][2046,4094][real]: DECaLS_
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T083400_oi[N29,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081923_oi[N25,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9527115-z-20180209T091446_oi[S24,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_950ea1d-z-20170306T093421_oi[S27,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9577b3f-z-20150408T060358_oi[S28,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2/CP20170330/c4d_170331_0
65325_ooi_r_ls9.fits.fz

Something went wrong; please check the logfile.
ERROR: galaxy PGC1898150; please check the logfile.
Finished galaxy PGC1898150 in 0.086 minutes.
Started working on galaxy PGC1898150 at Sat Feb  8 12:46:41 2020
RA, Dec=218.829702, 30.296722: run=decam (2 north CCDs, 16 south CCDs).
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.829702 30.2967217 --width 451 --height 451 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T12:46:42.509555
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '218.829702', '30.2967217', '--width', '451', '--height', '451', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.829702 30.2967217 --width 451 --height 451 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 218.829702 30.2967217
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 65533 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 65534 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071332 MB
Starting process 65535 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071336 MB
Starting process 303 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071336 MB
Starting process 308 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071336 MB
Starting process 313 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07134 MB
Starting process 318 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071272 MB
Starting process 323 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071276 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T12:46:43.777206
Keeping 16 CCDs:
   c4d_170331_065325_ooi_r_ls9.fits.fz c4d_170331_065325_ooi_r_ls9-N25 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57843.286 object DECaLS_22744_r
   c4d_170331_065648_ooi_g_ls9.fits.fz c4d_170331_065648_ooi_g_ls9-N25 g exptime 175.0 propid 2014B-0404 seeing 1.39 MJD 57843.287 object DECaLS_22744_g
   c4d_160226_082233_ooi_z_ls9.fits.fz c4d_160226_082233_ooi_z_ls9-N25 z exptime 142.0 propid 2014B-0404 seeing 1.08 MJD 57444.347 object DECaLS_22744_z
   c4d_180210_082602_ooi_z_ls9.fits.fz c4d_180210_082602_ooi_z_ls9-S24 z exptime 138.0 propid 2014B-0404 seeing 1.36 MJD 58159.351 object DECaLS_6858_z
   c4d_170329_071015_ooi_g_ls9.fits.fz c4d_170329_071015_ooi_g_ls9-S24 g exptime 132.0 propid 2014B-0404 seeing 1.51 MJD 57841.297 object DECaLS_6858_g
   c4d_170329_070735_ooi_r_ls9.fits.fz c4d_170329_070735_ooi_r_ls9-S24 r exptime 60.0 propid 2014B-0404 seeing 1.30 MJD 57841.296 object DECaLS_6858_r
   c4d_180209_091446_ooi_z_ls9.fits.fz c4d_180209_091446_ooi_z_ls9-S24 z exptime 113.0 propid 2014B-0404 seeing 1.13 MJD 58158.385 object DECaLS_6858_z
   c4d_170306_093651_ooi_z_ls9.fits.fz c4d_170306_093651_ooi_z_ls9-S27 z exptime 99.0 propid 2014B-0404 seeing 0.95 MJD 57818.399 object DECaLS_38602_z
   c4d_180224_084906_ooi_g_ls9.fits.fz c4d_180224_084906_ooi_g_ls9-S27 g exptime 128.0 propid 2014B-0404 seeing 1.41 MJD 58173.367 object DECaLS_38602_g
   c4d_180225_081201_ooi_r_ls9.fits.fz c4d_180225_081201_ooi_r_ls9-S27 r exptime 88.0 propid 2014B-0404 seeing 1.18 MJD 58174.342 object DECaLS_38602_r
   c4d_170422_050829_ooi_g_ls9.fits.fz c4d_170422_050829_ooi_g_ls9-N29 g exptime 175.0 propid 2014B-0404 seeing 1.27 MJD 57865.212 object DECaLS_6872_g
   c4d_170329_070606_ooi_r_ls9.fits.fz c4d_170329_070606_ooi_r_ls9-N29 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57841.295 object DECaLS_6872_r
   c4d_160228_083710_ooi_z_ls9.fits.fz c4d_160228_083710_ooi_z_ls9-N29 z exptime 141.0 propid 2014B-0404 seeing 0.90 MJD 57446.357 object DECaLS_6872_z
   c4d_150408_060636_ooi_z_ls9.fits.fz c4d_150408_060636_ooi_z_ls9-S28 z exptime 109.0 propid 2014B-0404 seeing 1.25 MJD 57120.253 object DECaLS_22730_z
   c4d_170718_005053_ooi_r_ls9.fits.fz c4d_170718_005053_ooi_r_ls9-S28 r exptime 113.0 propid 2014B-0404 seeing 1.57 MJD 57952.033 object DECaLS_22730_r
   c4d_170421_055123_ooi_g_ls9.fits.fz c4d_170421_055123_ooi_g_ls9-S28 g exptime 175.0 propid 2014B-0404 seeing 1.53 MJD 57864.241 object DECaLS_22730_g
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9577b3f-z-20150408T060358_oi[S28,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081923_oi[N25,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9527115-z-20180209T091446_oi[S24,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T083400_oi[N29,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9527115-z-20180210T082602_oi[S24,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_950ea1d-z-20170306T093421_oi[S27,noinherit][2046,4094][real]: DECaLS_
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9577b3f-z-20150408T060358_oi[S28,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9527115-z-20180209T091446_oi[S24,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081923_oi[N25,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9527115-z-20180210T082602_oi[S24,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T083400_oi[N29,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2/CP20170330/c4d_170331_0
65648_ooi_g_ls9.fits.fz

Something went wrong; please check the logfile.
ERROR: galaxy PGC1898150; please check the logfile.
Finished galaxy PGC1898150 in 0.738 minutes.
Started working on galaxy PGC1898150 at Sun Feb  9 07:24:29 2020
RA, Dec=218.829702, 30.296722: run=decam (2 north CCDs, 16 south CCDs).
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.829702 30.2967217 --width 451 --height 451 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-09T07:24:30.551241
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '218.829702', '30.2967217', '--width', '451', '--height', '451', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.829702 30.2967217 --width 451 --height 451 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 218.829702 30.2967217
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 24560 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 24561 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 24562 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 24563 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 24564 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 24565 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 24566 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 24567 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 24568 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 24569 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 24570 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 24571 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 24572 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 24573 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 24574 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 24575 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 24576 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071156 MB
Starting process 24577 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072168 MB
Starting process 24578 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072168 MB
Starting process 24579 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071876 MB
Starting process 24580 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071876 MB
Starting process 24581 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071156 MB
Starting process 24582 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071512 MB
Starting process 24583 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071512 MB
Starting process 24584 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071512 MB
Starting process 24585 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071512 MB
Starting process 24586 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071512 MB
Starting process 24587 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071512 MB
Starting process 24588 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071512 MB
Starting process 24589 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071512 MB
Starting process 24590 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071512 MB
Starting process 24591 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071512 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-09T07:24:31.821137
Keeping 16 CCDs:
   c4d_170331_065325_ooi_r_ls9.fits.fz c4d_170331_065325_ooi_r_ls9-N25 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57843.286 object DECaLS_22744_r
   c4d_170331_065648_ooi_g_ls9.fits.fz c4d_170331_065648_ooi_g_ls9-N25 g exptime 175.0 propid 2014B-0404 seeing 1.39 MJD 57843.287 object DECaLS_22744_g
   c4d_160226_082233_ooi_z_ls9.fits.fz c4d_160226_082233_ooi_z_ls9-N25 z exptime 142.0 propid 2014B-0404 seeing 1.08 MJD 57444.347 object DECaLS_22744_z
   c4d_180210_082602_ooi_z_ls9.fits.fz c4d_180210_082602_ooi_z_ls9-S24 z exptime 138.0 propid 2014B-0404 seeing 1.36 MJD 58159.351 object DECaLS_6858_z
   c4d_170329_071015_ooi_g_ls9.fits.fz c4d_170329_071015_ooi_g_ls9-S24 g exptime 132.0 propid 2014B-0404 seeing 1.51 MJD 57841.297 object DECaLS_6858_g
   c4d_170329_070735_ooi_r_ls9.fits.fz c4d_170329_070735_ooi_r_ls9-S24 r exptime 60.0 propid 2014B-0404 seeing 1.30 MJD 57841.296 object DECaLS_6858_r
   c4d_180209_091446_ooi_z_ls9.fits.fz c4d_180209_091446_ooi_z_ls9-S24 z exptime 113.0 propid 2014B-0404 seeing 1.13 MJD 58158.385 object DECaLS_6858_z
   c4d_170306_093651_ooi_z_ls9.fits.fz c4d_170306_093651_ooi_z_ls9-S27 z exptime 99.0 propid 2014B-0404 seeing 0.95 MJD 57818.399 object DECaLS_38602_z
   c4d_180224_084906_ooi_g_ls9.fits.fz c4d_180224_084906_ooi_g_ls9-S27 g exptime 128.0 propid 2014B-0404 seeing 1.41 MJD 58173.367 object DECaLS_38602_g
   c4d_180225_081201_ooi_r_ls9.fits.fz c4d_180225_081201_ooi_r_ls9-S27 r exptime 88.0 propid 2014B-0404 seeing 1.18 MJD 58174.342 object DECaLS_38602_r
   c4d_170422_050829_ooi_g_ls9.fits.fz c4d_170422_050829_ooi_g_ls9-N29 g exptime 175.0 propid 2014B-0404 seeing 1.27 MJD 57865.212 object DECaLS_6872_g
   c4d_170329_070606_ooi_r_ls9.fits.fz c4d_170329_070606_ooi_r_ls9-N29 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57841.295 object DECaLS_6872_r
   c4d_160228_083710_ooi_z_ls9.fits.fz c4d_160228_083710_ooi_z_ls9-N29 z exptime 141.0 propid 2014B-0404 seeing 0.90 MJD 57446.357 object DECaLS_6872_z
   c4d_150408_060636_ooi_z_ls9.fits.fz c4d_150408_060636_ooi_z_ls9-S28 z exptime 109.0 propid 2014B-0404 seeing 1.25 MJD 57120.253 object DECaLS_22730_z
   c4d_170718_005053_ooi_r_ls9.fits.fz c4d_170718_005053_ooi_r_ls9-S28 r exptime 113.0 propid 2014B-0404 seeing 1.57 MJD 57952.033 object DECaLS_22730_r
   c4d_170421_055123_ooi_g_ls9.fits.fz c4d_170421_055123_ooi_g_ls9-S28 g exptime 175.0 propid 2014B-0404 seeing 1.53 MJD 57864.241 object DECaLS_22730_g
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9577b3f-z-20150408T060358_oi[S28,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081923_oi[N25,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T083400_oi[N29,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9527115-z-20180209T091446_oi[S24,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9527115-z-20180210T082602_oi[S24,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_950ea1d-z-20170306T093421_oi[S27,noinherit][2046,4094][real]: DECaLS_
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9577b3f-z-20150408T060358_oi[S28,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T083400_oi[N29,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081923_oi[N25,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9527115-z-20180209T091446_oi[S24,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2/CP20170330/c4d_170331_0
65648_ooi_g_ls9.fits.fz

Something went wrong; please check the logfile.
ERROR: galaxy PGC1898150; please check the logfile.
Finished galaxy PGC1898150 in 0.102 minutes.
Started working on galaxy PGC1898150 at Sun Feb  9 09:58:09 2020
RA, Dec=218.829702, 30.296722: run=decam (2 north CCDs, 16 south CCDs).
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.829702 30.2967217 --width 451 --height 451 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-09T09:58:10.531431
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '218.829702', '30.2967217', '--width', '451', '--height', '451', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.829702 30.2967217 --width 451 --height 451 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 218.829702 30.2967217
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 38040 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 38041 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 38042 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 38043 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 38044 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 38045 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 38046 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 38047 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 38048 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 38049 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 38050 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 38051 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 38052 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071196 MB
Starting process 38053 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071196 MB
Starting process 38054 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071196 MB
Starting process 38055 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071196 MB
Starting process 38056 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.0712 MB
Starting process 38057 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072216 MB
Starting process 38058 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072216 MB
Starting process 38059 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07192 MB
Starting process 38060 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07192 MB
Starting process 38061 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.0712 MB
Starting process 38062 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071556 MB
Starting process 38063 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071556 MB
Starting process 38064 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071556 MB
Starting process 38065 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071556 MB
Starting process 38066 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071556 MB
Starting process 38067 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071556 MB
Starting process 38068 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071556 MB
Starting process 38069 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071492 MB
Starting process 38070 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071556 MB
Starting process 38071 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071492 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-09T09:58:11.812440
Keeping 16 CCDs:
   c4d_170331_065325_ooi_r_ls9.fits.fz c4d_170331_065325_ooi_r_ls9-N25 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57843.286 object DECaLS_22744_r
   c4d_170331_065648_ooi_g_ls9.fits.fz c4d_170331_065648_ooi_g_ls9-N25 g exptime 175.0 propid 2014B-0404 seeing 1.39 MJD 57843.287 object DECaLS_22744_g
   c4d_160226_082233_ooi_z_ls9.fits.fz c4d_160226_082233_ooi_z_ls9-N25 z exptime 142.0 propid 2014B-0404 seeing 1.08 MJD 57444.347 object DECaLS_22744_z
   c4d_180210_082602_ooi_z_ls9.fits.fz c4d_180210_082602_ooi_z_ls9-S24 z exptime 138.0 propid 2014B-0404 seeing 1.36 MJD 58159.351 object DECaLS_6858_z
   c4d_170329_071015_ooi_g_ls9.fits.fz c4d_170329_071015_ooi_g_ls9-S24 g exptime 132.0 propid 2014B-0404 seeing 1.51 MJD 57841.297 object DECaLS_6858_g
   c4d_170329_070735_ooi_r_ls9.fits.fz c4d_170329_070735_ooi_r_ls9-S24 r exptime 60.0 propid 2014B-0404 seeing 1.30 MJD 57841.296 object DECaLS_6858_r
   c4d_180209_091446_ooi_z_ls9.fits.fz c4d_180209_091446_ooi_z_ls9-S24 z exptime 113.0 propid 2014B-0404 seeing 1.13 MJD 58158.385 object DECaLS_6858_z
   c4d_170306_093651_ooi_z_ls9.fits.fz c4d_170306_093651_ooi_z_ls9-S27 z exptime 99.0 propid 2014B-0404 seeing 0.95 MJD 57818.399 object DECaLS_38602_z
   c4d_180224_084906_ooi_g_ls9.fits.fz c4d_180224_084906_ooi_g_ls9-S27 g exptime 128.0 propid 2014B-0404 seeing 1.41 MJD 58173.367 object DECaLS_38602_g
   c4d_180225_081201_ooi_r_ls9.fits.fz c4d_180225_081201_ooi_r_ls9-S27 r exptime 88.0 propid 2014B-0404 seeing 1.18 MJD 58174.342 object DECaLS_38602_r
   c4d_170422_050829_ooi_g_ls9.fits.fz c4d_170422_050829_ooi_g_ls9-N29 g exptime 175.0 propid 2014B-0404 seeing 1.27 MJD 57865.212 object DECaLS_6872_g
   c4d_170329_070606_ooi_r_ls9.fits.fz c4d_170329_070606_ooi_r_ls9-N29 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57841.295 object DECaLS_6872_r
   c4d_160228_083710_ooi_z_ls9.fits.fz c4d_160228_083710_ooi_z_ls9-N29 z exptime 141.0 propid 2014B-0404 seeing 0.90 MJD 57446.357 object DECaLS_6872_z
   c4d_150408_060636_ooi_z_ls9.fits.fz c4d_150408_060636_ooi_z_ls9-S28 z exptime 109.0 propid 2014B-0404 seeing 1.25 MJD 57120.253 object DECaLS_22730_z
   c4d_170718_005053_ooi_r_ls9.fits.fz c4d_170718_005053_ooi_r_ls9-S28 r exptime 113.0 propid 2014B-0404 seeing 1.57 MJD 57952.033 object DECaLS_22730_r
   c4d_170421_055123_ooi_g_ls9.fits.fz c4d_170421_055123_ooi_g_ls9-S28 g exptime 175.0 propid 2014B-0404 seeing 1.53 MJD 57864.241 object DECaLS_22730_g
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9577b3f-z-20150408T060358_oi[S28,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_950ea1d-z-20170306T093421_oi[S27,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T083400_oi[N29,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081923_oi[N25,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9527115-z-20180210T082602_oi[S24,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9527115-z-20180209T091446_oi[S24,noinherit][2046,4094][real]: DECaLS_
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9577b3f-z-20150408T060358_oi[S28,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_950ea1d-z-20170306T093421_oi[S27,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T083400_oi[N29,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081923_oi[N25,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9527115-z-20180209T091446_oi[S24,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2/CP20170330/c4d_170331_0
65648_ooi_g_ls9.fits.fz

Something went wrong; please check the logfile.
ERROR: galaxy PGC1898150; please check the logfile.
Finished galaxy PGC1898150 in 0.056 minutes.
Started working on galaxy PGC1898150 at Mon Feb 10 06:13:46 2020
RA, Dec=218.829702, 30.296722: run=decam (2 north CCDs, 16 south CCDs).
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.829702 30.2967217 --width 451 --height 451 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-10T06:13:47.472189
legacypipe git version: dr9e-pre1-9-g4a51be93
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '218.829702', '30.2967217', '--width', '451', '--height', '451', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.829702 30.2967217 --width 451 --height 451 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC1898150/PGC1898150-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 218.829702 30.2967217
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 12538 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 12539 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 12540 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 12541 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 12542 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 12543 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 12544 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 12545 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 12546 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 12547 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 12548 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 12549 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 12550 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 12551 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 12552 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 12553 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 12554 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 12555 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072164 MB
Starting process 12556 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072164 MB
Starting process 12557 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071876 MB
Starting process 12558 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071876 MB
Starting process 12559 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071508 MB
Starting process 12560 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071444 MB
Starting process 12561 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071444 MB
Starting process 12562 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071444 MB
Starting process 12563 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071444 MB
Starting process 12564 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071444 MB
Starting process 12565 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071444 MB
Starting process 12566 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071448 MB
Starting process 12567 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071448 MB
Starting process 12568 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071448 MB
Starting process 12569 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071448 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-10T06:13:48.721215
Keeping 16 CCDs:
   c4d_170331_065325_ooi_r_ls9.fits.fz c4d_170331_065325_ooi_r_ls9-N25 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57843.286 object DECaLS_22744_r
   c4d_170331_065648_ooi_g_ls9.fits.fz c4d_170331_065648_ooi_g_ls9-N25 g exptime 175.0 propid 2014B-0404 seeing 1.39 MJD 57843.287 object DECaLS_22744_g
   c4d_160226_082233_ooi_z_ls9.fits.fz c4d_160226_082233_ooi_z_ls9-N25 z exptime 142.0 propid 2014B-0404 seeing 1.08 MJD 57444.347 object DECaLS_22744_z
   c4d_180210_082602_ooi_z_ls9.fits.fz c4d_180210_082602_ooi_z_ls9-S24 z exptime 138.0 propid 2014B-0404 seeing 1.36 MJD 58159.351 object DECaLS_6858_z
   c4d_170329_071015_ooi_g_ls9.fits.fz c4d_170329_071015_ooi_g_ls9-S24 g exptime 132.0 propid 2014B-0404 seeing 1.51 MJD 57841.297 object DECaLS_6858_g
   c4d_170329_070735_ooi_r_ls9.fits.fz c4d_170329_070735_ooi_r_ls9-S24 r exptime 60.0 propid 2014B-0404 seeing 1.30 MJD 57841.296 object DECaLS_6858_r
   c4d_180209_091446_ooi_z_ls9.fits.fz c4d_180209_091446_ooi_z_ls9-S24 z exptime 113.0 propid 2014B-0404 seeing 1.13 MJD 58158.385 object DECaLS_6858_z
   c4d_170306_093651_ooi_z_ls9.fits.fz c4d_170306_093651_ooi_z_ls9-S27 z exptime 99.0 propid 2014B-0404 seeing 0.95 MJD 57818.399 object DECaLS_38602_z
   c4d_180224_084906_ooi_g_ls9.fits.fz c4d_180224_084906_ooi_g_ls9-S27 g exptime 128.0 propid 2014B-0404 seeing 1.41 MJD 58173.367 object DECaLS_38602_g
   c4d_180225_081201_ooi_r_ls9.fits.fz c4d_180225_081201_ooi_r_ls9-S27 r exptime 88.0 propid 2014B-0404 seeing 1.18 MJD 58174.342 object DECaLS_38602_r
   c4d_170422_050829_ooi_g_ls9.fits.fz c4d_170422_050829_ooi_g_ls9-N29 g exptime 175.0 propid 2014B-0404 seeing 1.27 MJD 57865.212 object DECaLS_6872_g
   c4d_170329_070606_ooi_r_ls9.fits.fz c4d_170329_070606_ooi_r_ls9-N29 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57841.295 object DECaLS_6872_r
   c4d_160228_083710_ooi_z_ls9.fits.fz c4d_160228_083710_ooi_z_ls9-N29 z exptime 141.0 propid 2014B-0404 seeing 0.90 MJD 57446.357 object DECaLS_6872_z
   c4d_150408_060636_ooi_z_ls9.fits.fz c4d_150408_060636_ooi_z_ls9-S28 z exptime 109.0 propid 2014B-0404 seeing 1.25 MJD 57120.253 object DECaLS_22730_z
   c4d_170718_005053_ooi_r_ls9.fits.fz c4d_170718_005053_ooi_r_ls9-S28 r exptime 113.0 propid 2014B-0404 seeing 1.57 MJD 57952.033 object DECaLS_22730_r
   c4d_170421_055123_ooi_g_ls9.fits.fz c4d_170421_055123_ooi_g_ls9-S28 g exptime 175.0 propid 2014B-0404 seeing 1.53 MJD 57864.241 object DECaLS_22730_g
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_950ea1d-z-20170306T093421_oi[S27,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9577b3f-z-20150408T060358_oi[S28,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9527115-z-20180209T091446_oi[S24,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081923_oi[N25,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9527115-z-20180210T082602_oi[S24,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T083400_oi[N29,noinherit][2046,4094][real]: DECaLS_
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9577b3f-z-20150408T060358_oi[S28,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_950ea1d-z-20170306T093421_oi[S27,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9527115-z-20180209T091446_oi[S24,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T083400_oi[N29,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081923_oi[N25,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2/CP20170330/c4d_170331_0
65648_ooi_g_ls9.fits.fz

Something went wrong; please check the logfile.
ERROR: galaxy PGC1898150; please check the logfile.
Finished galaxy PGC1898150 in 0.055 minutes.