Started working on galaxy PGC2801018 at Sat Feb  8 08:52:33 2020
RA, Dec=218.074965, 31.508543: run=decam (4 north CCDs, 10 south CCDs).
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.074965 31.508542900000002 --width 429 --height 429 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T08:52:33.702194
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '218.074965', '31.508542900000002', '--width', '429', '--height', '429', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.074965 31.508542900000002 --width 429 --height 429 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 218.074965 31.508542900000002
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 50498 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 50499 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 50500 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 50501 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 50502 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 50503 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 50504 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 50505 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T08:52:35.078861
Keeping 10 CCDs:
   c4d_170422_051737_ooi_g_ls9.fits.fz c4d_170422_051737_ooi_g_ls9-S18 g exptime 121.0 propid 2014B-0404 seeing 1.33 MJD 57865.219 object DECaLS_22731_g
   c4d_160226_081655_ooi_z_ls9.fits.fz c4d_160226_081655_ooi_z_ls9-S18 z exptime 135.0 propid 2014B-0404 seeing 1.15 MJD 57444.343 object DECaLS_22731_z
   c4d_160226_090529_ooi_z_ls9.fits.fz c4d_160226_090529_ooi_z_ls9-S17 z exptime 140.0 propid 2014B-0404 seeing 0.97 MJD 57444.377 object DECaLS_38603_z
   c4d_180224_084701_ooi_r_ls9.fits.fz c4d_180224_084701_ooi_r_ls9-S17 r exptime 86.0 propid 2014B-0404 seeing 1.14 MJD 58173.366 object DECaLS_38603_r
   c4d_180221_090109_ooi_g_ls9.fits.fz c4d_180221_090109_ooi_g_ls9-S17 g exptime 129.0 propid 2014B-0404 seeing 1.36 MJD 58170.376 object DECaLS_38603_g
   c4d_170503_050055_ooi_r_ls9.fits.fz c4d_170503_050055_ooi_r_ls9-S18 r exptime 112.0 propid 2014B-0404 seeing 1.29 MJD 57876.207 object DECaLS_22731_r
   c4d_180517_031813_ooi_g_ls9.fits.fz c4d_180517_031813_ooi_g_ls9-S13 g exptime 149.0 propid 2014B-0404 seeing 1.34 MJD 58255.138 object DECaLS_6859_g
   c4d_170406_065019_ooi_g_ls9.fits.fz c4d_170406_065019_ooi_g_ls9-S13 g exptime 175.0 propid 2014B-0404 seeing 1.61 MJD 57849.282 object DECaLS_6859_g
   c4d_170329_070435_ooi_r_ls9.fits.fz c4d_170329_070435_ooi_r_ls9-S13 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57841.294 object DECaLS_6859_r
   c4d_160228_082310_ooi_z_ls9.fits.fz c4d_160228_082310_ooi_z_ls9-S13 z exptime 115.0 propid 2014B-0404 seeing 0.96 MJD 57446.348 object DECaLS_6859_z
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T090219_oi[S17,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082024_oi[S13,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081350_oi[S18,noinherit][2046,4094][real]: DECaLS_
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082024_oi[S13,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T090219_oi[S17,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2/CP20170421/c4d_170422_0
51737_ooi_g_ls9.fits.fz

Something went wrong; please check the logfile.
ERROR: galaxy PGC2801018; please check the logfile.
Finished galaxy PGC2801018 in 0.087 minutes.
Started working on galaxy PGC2801018 at Sat Feb  8 12:44:52 2020
RA, Dec=218.074965, 31.508543: run=decam (4 north CCDs, 10 south CCDs).
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.074965 31.508542900000002 --width 429 --height 429 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T12:44:53.438029
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '218.074965', '31.508542900000002', '--width', '429', '--height', '429', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.074965 31.508542900000002 --width 429 --height 429 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 218.074965 31.508542900000002
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 63267 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 63268 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 63269 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 63270 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 63271 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 63272 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 63273 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 63275 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T12:44:54.868193
Keeping 10 CCDs:
   c4d_170422_051737_ooi_g_ls9.fits.fz c4d_170422_051737_ooi_g_ls9-S18 g exptime 121.0 propid 2014B-0404 seeing 1.33 MJD 57865.219 object DECaLS_22731_g
   c4d_160226_081655_ooi_z_ls9.fits.fz c4d_160226_081655_ooi_z_ls9-S18 z exptime 135.0 propid 2014B-0404 seeing 1.15 MJD 57444.343 object DECaLS_22731_z
   c4d_160226_090529_ooi_z_ls9.fits.fz c4d_160226_090529_ooi_z_ls9-S17 z exptime 140.0 propid 2014B-0404 seeing 0.97 MJD 57444.377 object DECaLS_38603_z
   c4d_180224_084701_ooi_r_ls9.fits.fz c4d_180224_084701_ooi_r_ls9-S17 r exptime 86.0 propid 2014B-0404 seeing 1.14 MJD 58173.366 object DECaLS_38603_r
   c4d_180221_090109_ooi_g_ls9.fits.fz c4d_180221_090109_ooi_g_ls9-S17 g exptime 129.0 propid 2014B-0404 seeing 1.36 MJD 58170.376 object DECaLS_38603_g
   c4d_170503_050055_ooi_r_ls9.fits.fz c4d_170503_050055_ooi_r_ls9-S18 r exptime 112.0 propid 2014B-0404 seeing 1.29 MJD 57876.207 object DECaLS_22731_r
   c4d_180517_031813_ooi_g_ls9.fits.fz c4d_180517_031813_ooi_g_ls9-S13 g exptime 149.0 propid 2014B-0404 seeing 1.34 MJD 58255.138 object DECaLS_6859_g
   c4d_170406_065019_ooi_g_ls9.fits.fz c4d_170406_065019_ooi_g_ls9-S13 g exptime 175.0 propid 2014B-0404 seeing 1.61 MJD 57849.282 object DECaLS_6859_g
   c4d_170329_070435_ooi_r_ls9.fits.fz c4d_170329_070435_ooi_r_ls9-S13 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57841.294 object DECaLS_6859_r
   c4d_160228_082310_ooi_z_ls9.fits.fz c4d_160228_082310_ooi_z_ls9-S13 z exptime 115.0 propid 2014B-0404 seeing 0.96 MJD 57446.348 object DECaLS_6859_z
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T090219_oi[S17,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082024_oi[S13,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081350_oi[S18,noinherit][2046,4094][real]: DECaLS_
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T090219_oi[S17,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2/CP20170421/c4d_170422_0
51737_ooi_g_ls9.fits.fz

Something went wrong; please check the logfile.
ERROR: galaxy PGC2801018; please check the logfile.
Finished galaxy PGC2801018 in 0.095 minutes.
Started working on galaxy PGC2801018 at Sun Feb  9 08:13:07 2020
RA, Dec=218.074965, 31.508543: run=decam (4 north CCDs, 10 south CCDs).
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.074965 31.508542900000002 --width 429 --height 429 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-09T08:13:08.293016
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '218.074965', '31.508542900000002', '--width', '429', '--height', '429', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.074965 31.508542900000002 --width 429 --height 429 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 218.074965 31.508542900000002
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 27474 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 27475 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 27476 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 27477 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 27478 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 27479 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 27480 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 27481 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 27482 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 27483 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 27484 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 27485 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 27486 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 27487 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 27488 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 27489 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 27490 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 27491 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.0722 MB
Starting process 27492 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.0722 MB
Starting process 27493 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071912 MB
Starting process 27494 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071912 MB
Starting process 27495 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 27496 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 27497 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 27498 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 27499 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 27500 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 27501 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 27502 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07148 MB
Starting process 27503 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07148 MB
Starting process 27504 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071484 MB
Starting process 27505 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071484 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-09T08:13:09.567588
Keeping 10 CCDs:
   c4d_170422_051737_ooi_g_ls9.fits.fz c4d_170422_051737_ooi_g_ls9-S18 g exptime 121.0 propid 2014B-0404 seeing 1.33 MJD 57865.219 object DECaLS_22731_g
   c4d_160226_081655_ooi_z_ls9.fits.fz c4d_160226_081655_ooi_z_ls9-S18 z exptime 135.0 propid 2014B-0404 seeing 1.15 MJD 57444.343 object DECaLS_22731_z
   c4d_160226_090529_ooi_z_ls9.fits.fz c4d_160226_090529_ooi_z_ls9-S17 z exptime 140.0 propid 2014B-0404 seeing 0.97 MJD 57444.377 object DECaLS_38603_z
   c4d_180224_084701_ooi_r_ls9.fits.fz c4d_180224_084701_ooi_r_ls9-S17 r exptime 86.0 propid 2014B-0404 seeing 1.14 MJD 58173.366 object DECaLS_38603_r
   c4d_180221_090109_ooi_g_ls9.fits.fz c4d_180221_090109_ooi_g_ls9-S17 g exptime 129.0 propid 2014B-0404 seeing 1.36 MJD 58170.376 object DECaLS_38603_g
   c4d_170503_050055_ooi_r_ls9.fits.fz c4d_170503_050055_ooi_r_ls9-S18 r exptime 112.0 propid 2014B-0404 seeing 1.29 MJD 57876.207 object DECaLS_22731_r
   c4d_180517_031813_ooi_g_ls9.fits.fz c4d_180517_031813_ooi_g_ls9-S13 g exptime 149.0 propid 2014B-0404 seeing 1.34 MJD 58255.138 object DECaLS_6859_g
   c4d_170406_065019_ooi_g_ls9.fits.fz c4d_170406_065019_ooi_g_ls9-S13 g exptime 175.0 propid 2014B-0404 seeing 1.61 MJD 57849.282 object DECaLS_6859_g
   c4d_170329_070435_ooi_r_ls9.fits.fz c4d_170329_070435_ooi_r_ls9-S13 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57841.294 object DECaLS_6859_r
   c4d_160228_082310_ooi_z_ls9.fits.fz c4d_160228_082310_ooi_z_ls9-S13 z exptime 115.0 propid 2014B-0404 seeing 0.96 MJD 57446.348 object DECaLS_6859_z
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081350_oi[S18,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082024_oi[S13,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T090219_oi[S17,noinherit][2046,4094][real]: DECaLS_
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T090219_oi[S17,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081350_oi[S18,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082024_oi[S13,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2/CP20170421/c4d_170422_0
51737_ooi_g_ls9.fits.fz

Something went wrong; please check the logfile.
ERROR: galaxy PGC2801018; please check the logfile.
Finished galaxy PGC2801018 in 0.063 minutes.
Started working on galaxy PGC2801018 at Sun Feb  9 09:56:46 2020
RA, Dec=218.074965, 31.508543: run=decam (4 north CCDs, 10 south CCDs).
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.074965 31.508542900000002 --width 429 --height 429 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-09T09:56:46.847081
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '218.074965', '31.508542900000002', '--width', '429', '--height', '429', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.074965 31.508542900000002 --width 429 --height 429 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 218.074965 31.508542900000002
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 37308 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 37309 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 37310 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 37311 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 37312 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 37313 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 37314 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 37315 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 37316 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 37317 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 37318 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 37319 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 37320 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 37321 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 37322 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 37323 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071196 MB
Starting process 37324 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071196 MB
Starting process 37325 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072208 MB
Starting process 37326 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072208 MB
Starting process 37327 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071916 MB
Starting process 37328 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071916 MB
Starting process 37329 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 37330 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 37331 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 37332 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 37333 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071552 MB
Starting process 37334 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071552 MB
Starting process 37335 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071552 MB
Starting process 37336 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 37337 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 37338 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 37339 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-09T09:56:48.096686
Keeping 10 CCDs:
   c4d_170422_051737_ooi_g_ls9.fits.fz c4d_170422_051737_ooi_g_ls9-S18 g exptime 121.0 propid 2014B-0404 seeing 1.33 MJD 57865.219 object DECaLS_22731_g
   c4d_160226_081655_ooi_z_ls9.fits.fz c4d_160226_081655_ooi_z_ls9-S18 z exptime 135.0 propid 2014B-0404 seeing 1.15 MJD 57444.343 object DECaLS_22731_z
   c4d_160226_090529_ooi_z_ls9.fits.fz c4d_160226_090529_ooi_z_ls9-S17 z exptime 140.0 propid 2014B-0404 seeing 0.97 MJD 57444.377 object DECaLS_38603_z
   c4d_180224_084701_ooi_r_ls9.fits.fz c4d_180224_084701_ooi_r_ls9-S17 r exptime 86.0 propid 2014B-0404 seeing 1.14 MJD 58173.366 object DECaLS_38603_r
   c4d_180221_090109_ooi_g_ls9.fits.fz c4d_180221_090109_ooi_g_ls9-S17 g exptime 129.0 propid 2014B-0404 seeing 1.36 MJD 58170.376 object DECaLS_38603_g
   c4d_170503_050055_ooi_r_ls9.fits.fz c4d_170503_050055_ooi_r_ls9-S18 r exptime 112.0 propid 2014B-0404 seeing 1.29 MJD 57876.207 object DECaLS_22731_r
   c4d_180517_031813_ooi_g_ls9.fits.fz c4d_180517_031813_ooi_g_ls9-S13 g exptime 149.0 propid 2014B-0404 seeing 1.34 MJD 58255.138 object DECaLS_6859_g
   c4d_170406_065019_ooi_g_ls9.fits.fz c4d_170406_065019_ooi_g_ls9-S13 g exptime 175.0 propid 2014B-0404 seeing 1.61 MJD 57849.282 object DECaLS_6859_g
   c4d_170329_070435_ooi_r_ls9.fits.fz c4d_170329_070435_ooi_r_ls9-S13 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57841.294 object DECaLS_6859_r
   c4d_160228_082310_ooi_z_ls9.fits.fz c4d_160228_082310_ooi_z_ls9-S13 z exptime 115.0 propid 2014B-0404 seeing 0.96 MJD 57446.348 object DECaLS_6859_z
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082024_oi[S13,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081350_oi[S18,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T090219_oi[S17,noinherit][2046,4094][real]: DECaLS_
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T090219_oi[S17,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082024_oi[S13,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2a/CP20180223/c4d_180224_
084701_ooi_r_ls9.fits.fz

Something went wrong; please check the logfile.
ERROR: galaxy PGC2801018; please check the logfile.
Finished galaxy PGC2801018 in 0.053 minutes.
Started working on galaxy PGC2801018 at Mon Feb 10 06:13:15 2020
RA, Dec=218.074965, 31.508543: run=decam (4 north CCDs, 10 south CCDs).
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.074965 31.508542900000002 --width 429 --height 429 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-10T06:13:15.638729
legacypipe git version: dr9e-pre1-9-g4a51be93
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '218.074965', '31.508542900000002', '--width', '429', '--height', '429', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 218.074965 31.508542900000002 --width 429 --height 429 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/218/PGC2801018/PGC2801018-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 218.074965 31.508542900000002
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 12021 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 12022 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 12023 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 12024 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 12026 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 12027 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 12028 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 12029 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 12030 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 12031 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 12032 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 12033 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 12034 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 12035 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 12036 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 12037 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 12038 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071156 MB
Starting process 12039 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072168 MB
Starting process 12040 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072168 MB
Starting process 12041 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071876 MB
Starting process 12042 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071876 MB
Starting process 12043 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071512 MB
Starting process 12044 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB
Starting process 12045 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB
Starting process 12046 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB
Starting process 12047 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB
Starting process 12048 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB
Starting process 12049 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB
Starting process 12050 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07144 MB
Starting process 12051 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07144 MB
Starting process 12052 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB
Starting process 12053 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07144 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-10T06:13:16.900513
Keeping 10 CCDs:
   c4d_170422_051737_ooi_g_ls9.fits.fz c4d_170422_051737_ooi_g_ls9-S18 g exptime 121.0 propid 2014B-0404 seeing 1.33 MJD 57865.219 object DECaLS_22731_g
   c4d_160226_081655_ooi_z_ls9.fits.fz c4d_160226_081655_ooi_z_ls9-S18 z exptime 135.0 propid 2014B-0404 seeing 1.15 MJD 57444.343 object DECaLS_22731_z
   c4d_160226_090529_ooi_z_ls9.fits.fz c4d_160226_090529_ooi_z_ls9-S17 z exptime 140.0 propid 2014B-0404 seeing 0.97 MJD 57444.377 object DECaLS_38603_z
   c4d_180224_084701_ooi_r_ls9.fits.fz c4d_180224_084701_ooi_r_ls9-S17 r exptime 86.0 propid 2014B-0404 seeing 1.14 MJD 58173.366 object DECaLS_38603_r
   c4d_180221_090109_ooi_g_ls9.fits.fz c4d_180221_090109_ooi_g_ls9-S17 g exptime 129.0 propid 2014B-0404 seeing 1.36 MJD 58170.376 object DECaLS_38603_g
   c4d_170503_050055_ooi_r_ls9.fits.fz c4d_170503_050055_ooi_r_ls9-S18 r exptime 112.0 propid 2014B-0404 seeing 1.29 MJD 57876.207 object DECaLS_22731_r
   c4d_180517_031813_ooi_g_ls9.fits.fz c4d_180517_031813_ooi_g_ls9-S13 g exptime 149.0 propid 2014B-0404 seeing 1.34 MJD 58255.138 object DECaLS_6859_g
   c4d_170406_065019_ooi_g_ls9.fits.fz c4d_170406_065019_ooi_g_ls9-S13 g exptime 175.0 propid 2014B-0404 seeing 1.61 MJD 57849.282 object DECaLS_6859_g
   c4d_170329_070435_ooi_r_ls9.fits.fz c4d_170329_070435_ooi_r_ls9-S13 r exptime 62.0 propid 2014B-0404 seeing 1.26 MJD 57841.294 object DECaLS_6859_r
   c4d_160228_082310_ooi_z_ls9.fits.fz c4d_160228_082310_ooi_z_ls9-S13 z exptime 115.0 propid 2014B-0404 seeing 0.96 MJD 57446.348 object DECaLS_6859_z
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081350_oi[S18,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082024_oi[S13,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T090219_oi[S17,noinherit][2046,4094][real]: DECaLS_
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081350_oi[S18,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082024_oi[S13,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2/CP20170421/c4d_170422_0
51737_ooi_g_ls9.fits.fz

Something went wrong; please check the logfile.
ERROR: galaxy PGC2801018; please check the logfile.
Finished galaxy PGC2801018 in 0.056 minutes.