Started working on galaxy PGC3097949_GROUP at Sat Feb  8 08:30:35 2020
RA, Dec=215.031205, 30.363256: run=decam (0 north CCDs, 19 south CCDs).
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 215.03189947899952 30.364729941272973 --width 631 --height 631 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T08:30:36.697328
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '215.03189947899952', '30.364729941272973', '--width', '631', '--height', '631', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 215.03189947899952 30.364729941272973 --width 631 --height 631 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 215.03189947899952 30.364729941272973
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 5896 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 5897 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 5898 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 5899 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 5900 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 5901 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 5902 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 5903 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T08:30:38.222126
Keeping 19 CCDs:
   c4d_170406_064656_ooi_r_ls9.fits.fz c4d_170406_064656_ooi_r_ls9-S24 r exptime 85.0 propid 2014B-0404 seeing 1.28 MJD 57849.281 object DECaLS_6823_r
   c4d_150411_054325_ooi_z_ls9.fits.fz c4d_150411_054325_ooi_z_ls9-S24 z exptime 88.0 propid 2014B-0404 seeing 1.08 MJD 57123.237 object DECaLS_6823_z
   c4d_180213_085746_ooi_g_ls9.fits.fz c4d_180213_085746_ooi_g_ls9-S24 g exptime 117.0 propid 2014B-0404 seeing 1.27 MJD 58162.373 object DECaLS_6823_g
   c4d_170521_031658_ooi_r_ls9.fits.fz c4d_170521_031658_ooi_r_ls9-N29 r exptime 125.0 propid 2014B-0404 seeing 1.56 MJD 57894.135 object DECaLS_22715_r
   c4d_160226_081124_ooi_z_ls9.fits.fz c4d_160226_081124_ooi_z_ls9-N29 z exptime 133.0 propid 2014B-0404 seeing 0.94 MJD 57444.339 object DECaLS_22715_z
   c4d_170522_025630_ooi_g_ls9.fits.fz c4d_170522_025630_ooi_g_ls9-N29 g exptime 175.0 propid 2014B-0404 seeing 1.45 MJD 57895.120 object DECaLS_22715_g
   c4d_170317_072842_ooi_z_ls9.fits.fz c4d_170317_072842_ooi_z_ls9-S27 z exptime 80.0 propid 2014B-0404 seeing 1.13 MJD 57829.310 object DECaLS_38567_z
   c4d_180222_084327_ooi_r_ls9.fits.fz c4d_180222_084327_ooi_r_ls9-S27 r exptime 47.0 propid 2014B-0404 seeing 0.97 MJD 58171.364 object DECaLS_38567_r
   c4d_180222_084443_ooi_g_ls9.fits.fz c4d_180222_084443_ooi_g_ls9-S27 g exptime 105.0 propid 2014B-0404 seeing 1.21 MJD 58171.364 object DECaLS_38567_g
   c4d_190305_071119_ooi_z_ls9.fits.fz c4d_190305_071119_ooi_z_ls9-S27 z exptime 87.0 propid 2014B-0404 seeing 1.31 MJD 58547.300 object DECaLS_38567_z
   c4d_170329_070305_ooi_r_ls9.fits.fz c4d_170329_070305_ooi_r_ls9-N29 r exptime 64.0 propid 2014B-0404 seeing 1.22 MJD 57841.292 object DECaLS_6843_r
   c4d_170422_052449_ooi_g_ls9.fits.fz c4d_170422_052449_ooi_g_ls9-N29 g exptime 147.0 propid 2014B-0404 seeing 1.16 MJD 57865.223 object DECaLS_6843_g
   c4d_160228_082044_ooi_z_ls9.fits.fz c4d_160228_082044_ooi_z_ls9-N29 z exptime 131.0 propid 2014B-0404 seeing 1.01 MJD 57446.346 object DECaLS_6843_z
   c4d_170522_025302_ooi_g_ls9.fits.fz c4d_170522_025302_ooi_g_ls9-S28 g exptime 175.0 propid 2014B-0404 seeing 1.82 MJD 57895.118 object DECaLS_22695_g
   c4d_170422_050255_ooi_r_ls9.fits.fz c4d_170422_050255_ooi_r_ls9-S28 r exptime 72.0 propid 2014B-0404 seeing 1.21 MJD 57865.209 object DECaLS_22695_r
   c4d_150408_055503_ooi_z_ls9.fits.fz c4d_150408_055503_ooi_z_ls9-S28 z exptime 109.0 propid 2014B-0404 seeing 1.18 MJD 57120.245 object DECaLS_22695_z
   c4d_170522_025630_ooi_g_ls9.fits.fz c4d_170522_025630_ooi_g_ls9-N25 g exptime 175.0 propid 2014B-0404 seeing 1.72 MJD 57895.120 object DECaLS_22715_g
   c4d_170521_031658_ooi_r_ls9.fits.fz c4d_170521_031658_ooi_r_ls9-N25 r exptime 125.0 propid 2014B-0404 seeing 1.57 MJD 57894.135 object DECaLS_22715_r
   c4d_160226_081124_ooi_z_ls9.fits.fz c4d_160226_081124_ooi_z_ls9-N25 z exptime 133.0 propid 2014B-0404 seeing 0.97 MJD 57444.339 object DECaLS_22715_z
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T081743_oi[N29,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T080822_oi[N25,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T080822_oi[N29,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9510adb-z-20170317T072630_oi[S27,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9577b3f-z-20150408T055224_oi[S28,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95351a0-z-20190305T071119_oi[S27,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_956d285-z-20150411T054106_oi[S24,noinherit][2046,4094][real]: DECaLS_
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T081743_oi[N29,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T080822_oi[N29,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T080822_oi[N25,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9510adb-z-20170317T072630_oi[S27,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2a/CP20180212/c4d_180213_
085746_ooi_g_ls9.fits.fz

Something went wrong; please check the logfile.
ERROR: galaxy PGC3097949_GROUP; please check the logfile.
Finished galaxy PGC3097949_GROUP in 0.105 minutes.
Started working on galaxy PGC3097949_GROUP at Sat Feb  8 12:45:31 2020
RA, Dec=215.031205, 30.363256: run=decam (0 north CCDs, 19 south CCDs).
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 215.03189947899952 30.364729941272973 --width 631 --height 631 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T12:45:31.904906
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '215.03189947899952', '30.364729941272973', '--width', '631', '--height', '631', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 215.03189947899952 30.364729941272973 --width 631 --height 631 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 215.03189947899952 30.364729941272973
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 10540 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 10541 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 10542 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071244 MB
Starting process 10543 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071244 MB
Starting process 10544 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071244 MB
Starting process 10545 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071252 MB
Starting process 10546 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 10547 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T12:45:33.142767
Keeping 19 CCDs:
   c4d_170406_064656_ooi_r_ls9.fits.fz c4d_170406_064656_ooi_r_ls9-S24 r exptime 85.0 propid 2014B-0404 seeing 1.28 MJD 57849.281 object DECaLS_6823_r
   c4d_150411_054325_ooi_z_ls9.fits.fz c4d_150411_054325_ooi_z_ls9-S24 z exptime 88.0 propid 2014B-0404 seeing 1.08 MJD 57123.237 object DECaLS_6823_z
   c4d_180213_085746_ooi_g_ls9.fits.fz c4d_180213_085746_ooi_g_ls9-S24 g exptime 117.0 propid 2014B-0404 seeing 1.27 MJD 58162.373 object DECaLS_6823_g
   c4d_170521_031658_ooi_r_ls9.fits.fz c4d_170521_031658_ooi_r_ls9-N29 r exptime 125.0 propid 2014B-0404 seeing 1.56 MJD 57894.135 object DECaLS_22715_r
   c4d_160226_081124_ooi_z_ls9.fits.fz c4d_160226_081124_ooi_z_ls9-N29 z exptime 133.0 propid 2014B-0404 seeing 0.94 MJD 57444.339 object DECaLS_22715_z
   c4d_170522_025630_ooi_g_ls9.fits.fz c4d_170522_025630_ooi_g_ls9-N29 g exptime 175.0 propid 2014B-0404 seeing 1.45 MJD 57895.120 object DECaLS_22715_g
   c4d_170317_072842_ooi_z_ls9.fits.fz c4d_170317_072842_ooi_z_ls9-S27 z exptime 80.0 propid 2014B-0404 seeing 1.13 MJD 57829.310 object DECaLS_38567_z
   c4d_180222_084327_ooi_r_ls9.fits.fz c4d_180222_084327_ooi_r_ls9-S27 r exptime 47.0 propid 2014B-0404 seeing 0.97 MJD 58171.364 object DECaLS_38567_r
   c4d_180222_084443_ooi_g_ls9.fits.fz c4d_180222_084443_ooi_g_ls9-S27 g exptime 105.0 propid 2014B-0404 seeing 1.21 MJD 58171.364 object DECaLS_38567_g
   c4d_190305_071119_ooi_z_ls9.fits.fz c4d_190305_071119_ooi_z_ls9-S27 z exptime 87.0 propid 2014B-0404 seeing 1.31 MJD 58547.300 object DECaLS_38567_z
   c4d_170329_070305_ooi_r_ls9.fits.fz c4d_170329_070305_ooi_r_ls9-N29 r exptime 64.0 propid 2014B-0404 seeing 1.22 MJD 57841.292 object DECaLS_6843_r
   c4d_170422_052449_ooi_g_ls9.fits.fz c4d_170422_052449_ooi_g_ls9-N29 g exptime 147.0 propid 2014B-0404 seeing 1.16 MJD 57865.223 object DECaLS_6843_g
   c4d_160228_082044_ooi_z_ls9.fits.fz c4d_160228_082044_ooi_z_ls9-N29 z exptime 131.0 propid 2014B-0404 seeing 1.01 MJD 57446.346 object DECaLS_6843_z
   c4d_170522_025302_ooi_g_ls9.fits.fz c4d_170522_025302_ooi_g_ls9-S28 g exptime 175.0 propid 2014B-0404 seeing 1.82 MJD 57895.118 object DECaLS_22695_g
   c4d_170422_050255_ooi_r_ls9.fits.fz c4d_170422_050255_ooi_r_ls9-S28 r exptime 72.0 propid 2014B-0404 seeing 1.21 MJD 57865.209 object DECaLS_22695_r
   c4d_150408_055503_ooi_z_ls9.fits.fz c4d_150408_055503_ooi_z_ls9-S28 z exptime 109.0 propid 2014B-0404 seeing 1.18 MJD 57120.245 object DECaLS_22695_z
   c4d_170522_025630_ooi_g_ls9.fits.fz c4d_170522_025630_ooi_g_ls9-N25 g exptime 175.0 propid 2014B-0404 seeing 1.72 MJD 57895.120 object DECaLS_22715_g
   c4d_170521_031658_ooi_r_ls9.fits.fz c4d_170521_031658_ooi_r_ls9-N25 r exptime 125.0 propid 2014B-0404 seeing 1.57 MJD 57894.135 object DECaLS_22715_r
   c4d_160226_081124_ooi_z_ls9.fits.fz c4d_160226_081124_ooi_z_ls9-N25 z exptime 133.0 propid 2014B-0404 seeing 0.97 MJD 57444.339 object DECaLS_22715_z
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T080822_oi[N25,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T080822_oi[N29,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95351a0-z-20190305T071119_oi[S27,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9510adb-z-20170317T072630_oi[S27,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_956d285-z-20150411T054106_oi[S24,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T081743_oi[N29,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9577b3f-z-20150408T055224_oi[S28,noinherit][2046,4094][real]: DECaLS_
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T080822_oi[N25,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95351a0-z-20190305T071119_oi[S27,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T080822_oi[N29,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9510adb-z-20170317T072630_oi[S27,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T081743_oi[N29,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_956d285-z-20150411T054106_oi[S24,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2a/CP20180212/c4d_180213_
085746_ooi_g_ls9.fits.fz

Something went wrong; please check the logfile.
ERROR: galaxy PGC3097949_GROUP; please check the logfile.
Finished galaxy PGC3097949_GROUP in 0.086 minutes.
Started working on galaxy PGC3097949_GROUP at Sun Feb  9 10:14:07 2020
RA, Dec=215.031205, 30.363256: run=decam (0 north CCDs, 19 south CCDs).
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 215.03189947899952 30.364729941272973 --width 631 --height 631 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-09T10:14:07.695132
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '215.03189947899952', '30.364729941272973', '--width', '631', '--height', '631', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 215.03189947899952 30.364729941272973 --width 631 --height 631 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 215.03189947899952 30.364729941272973
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 42619 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07116 MB
Starting process 42620 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 42621 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 42622 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 42623 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 42624 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 42625 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 42626 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 42627 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 42628 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 42629 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 42630 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 42631 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 42632 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 42633 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 42634 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 42635 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 42636 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072204 MB
Starting process 42637 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072204 MB
Starting process 42638 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071912 MB
Starting process 42639 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071908 MB
Starting process 42640 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 42641 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 42642 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 42643 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 42644 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 42645 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07148 MB
Starting process 42646 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07148 MB
Starting process 42647 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07148 MB
Starting process 42648 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07148 MB
Starting process 42649 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 42650 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-09T10:14:08.981393
Keeping 19 CCDs:
   c4d_170406_064656_ooi_r_ls9.fits.fz c4d_170406_064656_ooi_r_ls9-S24 r exptime 85.0 propid 2014B-0404 seeing 1.28 MJD 57849.281 object DECaLS_6823_r
   c4d_150411_054325_ooi_z_ls9.fits.fz c4d_150411_054325_ooi_z_ls9-S24 z exptime 88.0 propid 2014B-0404 seeing 1.08 MJD 57123.237 object DECaLS_6823_z
   c4d_180213_085746_ooi_g_ls9.fits.fz c4d_180213_085746_ooi_g_ls9-S24 g exptime 117.0 propid 2014B-0404 seeing 1.27 MJD 58162.373 object DECaLS_6823_g
   c4d_170521_031658_ooi_r_ls9.fits.fz c4d_170521_031658_ooi_r_ls9-N29 r exptime 125.0 propid 2014B-0404 seeing 1.56 MJD 57894.135 object DECaLS_22715_r
   c4d_160226_081124_ooi_z_ls9.fits.fz c4d_160226_081124_ooi_z_ls9-N29 z exptime 133.0 propid 2014B-0404 seeing 0.94 MJD 57444.339 object DECaLS_22715_z
   c4d_170522_025630_ooi_g_ls9.fits.fz c4d_170522_025630_ooi_g_ls9-N29 g exptime 175.0 propid 2014B-0404 seeing 1.45 MJD 57895.120 object DECaLS_22715_g
   c4d_170317_072842_ooi_z_ls9.fits.fz c4d_170317_072842_ooi_z_ls9-S27 z exptime 80.0 propid 2014B-0404 seeing 1.13 MJD 57829.310 object DECaLS_38567_z
   c4d_180222_084327_ooi_r_ls9.fits.fz c4d_180222_084327_ooi_r_ls9-S27 r exptime 47.0 propid 2014B-0404 seeing 0.97 MJD 58171.364 object DECaLS_38567_r
   c4d_180222_084443_ooi_g_ls9.fits.fz c4d_180222_084443_ooi_g_ls9-S27 g exptime 105.0 propid 2014B-0404 seeing 1.21 MJD 58171.364 object DECaLS_38567_g
   c4d_190305_071119_ooi_z_ls9.fits.fz c4d_190305_071119_ooi_z_ls9-S27 z exptime 87.0 propid 2014B-0404 seeing 1.31 MJD 58547.300 object DECaLS_38567_z
   c4d_170329_070305_ooi_r_ls9.fits.fz c4d_170329_070305_ooi_r_ls9-N29 r exptime 64.0 propid 2014B-0404 seeing 1.22 MJD 57841.292 object DECaLS_6843_r
   c4d_170422_052449_ooi_g_ls9.fits.fz c4d_170422_052449_ooi_g_ls9-N29 g exptime 147.0 propid 2014B-0404 seeing 1.16 MJD 57865.223 object DECaLS_6843_g
   c4d_160228_082044_ooi_z_ls9.fits.fz c4d_160228_082044_ooi_z_ls9-N29 z exptime 131.0 propid 2014B-0404 seeing 1.01 MJD 57446.346 object DECaLS_6843_z
   c4d_170522_025302_ooi_g_ls9.fits.fz c4d_170522_025302_ooi_g_ls9-S28 g exptime 175.0 propid 2014B-0404 seeing 1.82 MJD 57895.118 object DECaLS_22695_g
   c4d_170422_050255_ooi_r_ls9.fits.fz c4d_170422_050255_ooi_r_ls9-S28 r exptime 72.0 propid 2014B-0404 seeing 1.21 MJD 57865.209 object DECaLS_22695_r
   c4d_150408_055503_ooi_z_ls9.fits.fz c4d_150408_055503_ooi_z_ls9-S28 z exptime 109.0 propid 2014B-0404 seeing 1.18 MJD 57120.245 object DECaLS_22695_z
   c4d_170522_025630_ooi_g_ls9.fits.fz c4d_170522_025630_ooi_g_ls9-N25 g exptime 175.0 propid 2014B-0404 seeing 1.72 MJD 57895.120 object DECaLS_22715_g
   c4d_170521_031658_ooi_r_ls9.fits.fz c4d_170521_031658_ooi_r_ls9-N25 r exptime 125.0 propid 2014B-0404 seeing 1.57 MJD 57894.135 object DECaLS_22715_r
   c4d_160226_081124_ooi_z_ls9.fits.fz c4d_160226_081124_ooi_z_ls9-N25 z exptime 133.0 propid 2014B-0404 seeing 0.97 MJD 57444.339 object DECaLS_22715_z
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_956d285-z-20150411T054106_oi[S24,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9577b3f-z-20150408T055224_oi[S28,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9510adb-z-20170317T072630_oi[S27,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T080822_oi[N25,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T080822_oi[N29,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95351a0-z-20190305T071119_oi[S27,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T081743_oi[N29,noinherit][2046,4094][real]: DECaLS_
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_956d285-z-20150411T054106_oi[S24,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9577b3f-z-20150408T055224_oi[S28,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9510adb-z-20170317T072630_oi[S27,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T080822_oi[N25,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T080822_oi[N29,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T081743_oi[N29,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95351a0-z-20190305T071119_oi[S27,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2/CP20170405/c4d_170406_0
64656_ooi_r_ls9.fits.fz

Something went wrong; please check the logfile.
ERROR: galaxy PGC3097949_GROUP; please check the logfile.
Finished galaxy PGC3097949_GROUP in 0.064 minutes.
Started working on galaxy PGC3097949_GROUP at Mon Feb 10 06:16:36 2020
RA, Dec=215.031205, 30.363256: run=decam (0 north CCDs, 19 south CCDs).
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 215.03189947899952 30.364729941272973 --width 631 --height 631 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-10T06:16:36.668625
legacypipe git version: dr9e-pre1-9-g4a51be93
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '215.03189947899952', '30.364729941272973', '--width', '631', '--height', '631', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 215.03189947899952 30.364729941272973 --width 631 --height 631 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/215/PGC3097949_GROUP/PGC3097949_GROUP-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 215.03189947899952 30.364729941272973
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 14708 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07112 MB
Starting process 14709 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07112 MB
Starting process 14710 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07112 MB
Starting process 14711 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 14712 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071124 MB
Starting process 14713 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 14714 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 14715 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 14716 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 14717 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 14718 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 14719 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 14720 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 14721 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 14722 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 14723 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 14724 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 14725 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072156 MB
Starting process 14726 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072156 MB
Starting process 14727 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071868 MB
Starting process 14728 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071864 MB
Starting process 14729 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.0715 MB
Starting process 14730 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.0715 MB
Starting process 14731 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.0715 MB
Starting process 14732 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.0715 MB
Starting process 14733 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.0715 MB
Starting process 14734 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.0715 MB
Starting process 14735 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.0715 MB
Starting process 14736 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.0715 MB
Starting process 14737 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.0715 MB
Starting process 14738 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB
Starting process 14739 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-10T06:16:37.944044
Keeping 19 CCDs:
   c4d_170406_064656_ooi_r_ls9.fits.fz c4d_170406_064656_ooi_r_ls9-S24 r exptime 85.0 propid 2014B-0404 seeing 1.28 MJD 57849.281 object DECaLS_6823_r
   c4d_150411_054325_ooi_z_ls9.fits.fz c4d_150411_054325_ooi_z_ls9-S24 z exptime 88.0 propid 2014B-0404 seeing 1.08 MJD 57123.237 object DECaLS_6823_z
   c4d_180213_085746_ooi_g_ls9.fits.fz c4d_180213_085746_ooi_g_ls9-S24 g exptime 117.0 propid 2014B-0404 seeing 1.27 MJD 58162.373 object DECaLS_6823_g
   c4d_170521_031658_ooi_r_ls9.fits.fz c4d_170521_031658_ooi_r_ls9-N29 r exptime 125.0 propid 2014B-0404 seeing 1.56 MJD 57894.135 object DECaLS_22715_r
   c4d_160226_081124_ooi_z_ls9.fits.fz c4d_160226_081124_ooi_z_ls9-N29 z exptime 133.0 propid 2014B-0404 seeing 0.94 MJD 57444.339 object DECaLS_22715_z
   c4d_170522_025630_ooi_g_ls9.fits.fz c4d_170522_025630_ooi_g_ls9-N29 g exptime 175.0 propid 2014B-0404 seeing 1.45 MJD 57895.120 object DECaLS_22715_g
   c4d_170317_072842_ooi_z_ls9.fits.fz c4d_170317_072842_ooi_z_ls9-S27 z exptime 80.0 propid 2014B-0404 seeing 1.13 MJD 57829.310 object DECaLS_38567_z
   c4d_180222_084327_ooi_r_ls9.fits.fz c4d_180222_084327_ooi_r_ls9-S27 r exptime 47.0 propid 2014B-0404 seeing 0.97 MJD 58171.364 object DECaLS_38567_r
   c4d_180222_084443_ooi_g_ls9.fits.fz c4d_180222_084443_ooi_g_ls9-S27 g exptime 105.0 propid 2014B-0404 seeing 1.21 MJD 58171.364 object DECaLS_38567_g
   c4d_190305_071119_ooi_z_ls9.fits.fz c4d_190305_071119_ooi_z_ls9-S27 z exptime 87.0 propid 2014B-0404 seeing 1.31 MJD 58547.300 object DECaLS_38567_z
   c4d_170329_070305_ooi_r_ls9.fits.fz c4d_170329_070305_ooi_r_ls9-N29 r exptime 64.0 propid 2014B-0404 seeing 1.22 MJD 57841.292 object DECaLS_6843_r
   c4d_170422_052449_ooi_g_ls9.fits.fz c4d_170422_052449_ooi_g_ls9-N29 g exptime 147.0 propid 2014B-0404 seeing 1.16 MJD 57865.223 object DECaLS_6843_g
   c4d_160228_082044_ooi_z_ls9.fits.fz c4d_160228_082044_ooi_z_ls9-N29 z exptime 131.0 propid 2014B-0404 seeing 1.01 MJD 57446.346 object DECaLS_6843_z
   c4d_170522_025302_ooi_g_ls9.fits.fz c4d_170522_025302_ooi_g_ls9-S28 g exptime 175.0 propid 2014B-0404 seeing 1.82 MJD 57895.118 object DECaLS_22695_g
   c4d_170422_050255_ooi_r_ls9.fits.fz c4d_170422_050255_ooi_r_ls9-S28 r exptime 72.0 propid 2014B-0404 seeing 1.21 MJD 57865.209 object DECaLS_22695_r
   c4d_150408_055503_ooi_z_ls9.fits.fz c4d_150408_055503_ooi_z_ls9-S28 z exptime 109.0 propid 2014B-0404 seeing 1.18 MJD 57120.245 object DECaLS_22695_z
   c4d_170522_025630_ooi_g_ls9.fits.fz c4d_170522_025630_ooi_g_ls9-N25 g exptime 175.0 propid 2014B-0404 seeing 1.72 MJD 57895.120 object DECaLS_22715_g
   c4d_170521_031658_ooi_r_ls9.fits.fz c4d_170521_031658_ooi_r_ls9-N25 r exptime 125.0 propid 2014B-0404 seeing 1.57 MJD 57894.135 object DECaLS_22715_r
   c4d_160226_081124_ooi_z_ls9.fits.fz c4d_160226_081124_ooi_z_ls9-N25 z exptime 133.0 propid 2014B-0404 seeing 0.97 MJD 57444.339 object DECaLS_22715_z
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9577b3f-z-20150408T055224_oi[S28,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_956d285-z-20150411T054106_oi[S24,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95351a0-z-20190305T071119_oi[S27,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T080822_oi[N29,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T080822_oi[N25,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T081743_oi[N29,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9510adb-z-20170317T072630_oi[S27,noinherit][2046,4094][real]: DECaLS_
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9577b3f-z-20150408T055224_oi[S28,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_956d285-z-20150411T054106_oi[S24,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95351a0-z-20190305T071119_oi[S27,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T080822_oi[N29,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T080822_oi[N25,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_9510adb-z-20170317T072630_oi[S27,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T081743_oi[N29,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2a/CP20180212/c4d_180213_
085746_ooi_g_ls9.fits.fz

Something went wrong; please check the logfile.
ERROR: galaxy PGC3097949_GROUP; please check the logfile.
Finished galaxy PGC3097949_GROUP in 0.062 minutes.