Started working on galaxy PGC2028578 at Sat Feb  8 08:28:15 2020
RA, Dec=216.988866, 33.289186: run=decam (11 north CCDs, 13 south CCDs).
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.988866 33.2891858 --width 361 --height 361 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T08:28:15.833558
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '216.988866', '33.2891858', '--width', '361', '--height', '361', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.988866 33.2891858 --width 361 --height 361 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 216.988866 33.2891858
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 56273 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 56274 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 56275 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 56276 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 56277 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 56278 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 56279 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 56280 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T08:28:17.114545
Keeping 13 CCDs:
   c4d_180213_090013_ooi_r_ls9.fits.fz c4d_180213_090013_ooi_r_ls9-S24 r exptime 52.0 propid 2014B-0404 seeing 1.05 MJD 58162.375 object DECaLS_6860_r
   c4d_170422_052010_ooi_g_ls9.fits.fz c4d_170422_052010_ooi_g_ls9-S24 g exptime 125.0 propid 2014B-0404 seeing 1.45 MJD 57865.220 object DECaLS_6860_g
   c4d_160228_083134_ooi_z_ls9.fits.fz c4d_160228_083134_ooi_z_ls9-S24 z exptime 147.0 propid 2014B-0404 seeing 0.97 MJD 57446.353 object DECaLS_6860_z
   c4d_170717_235713_ooi_g_ls9.fits.fz c4d_170717_235713_ooi_g_ls9-S23 g exptime 169.0 propid 2014B-0404 seeing 1.65 MJD 57951.996 object DECaLS_22732_g
   c4d_170717_235355_ooi_r_ls9.fits.fz c4d_170717_235355_ooi_r_ls9-S23 r exptime 96.0 propid 2014B-0404 seeing 1.46 MJD 57951.994 object DECaLS_22732_r
   c4d_160226_081409_ooi_z_ls9.fits.fz c4d_160226_081409_ooi_z_ls9-S23 z exptime 135.0 propid 2014B-0404 seeing 1.04 MJD 57444.341 object DECaLS_22732_z
   c4d_180213_090013_ooi_r_ls9.fits.fz c4d_180213_090013_ooi_r_ls9-S18 r exptime 52.0 propid 2014B-0404 seeing 1.13 MJD 58162.375 object DECaLS_6860_r
   c4d_160228_083134_ooi_z_ls9.fits.fz c4d_160228_083134_ooi_z_ls9-S18 z exptime 147.0 propid 2014B-0404 seeing 1.04 MJD 57446.353 object DECaLS_6860_z
   c4d_170422_052010_ooi_g_ls9.fits.fz c4d_170422_052010_ooi_g_ls9-S18 g exptime 125.0 propid 2014B-0404 seeing 1.38 MJD 57865.220 object DECaLS_6860_g
   c4d_190307_071335_ooi_r_ls9.fits.fz c4d_190307_071335_ooi_r_ls9-S22 r exptime 49.0 propid 2014B-0404 seeing 1.44 MJD 58549.301 object DECaLS_38604_r
   c4d_180513_031444_ooi_r_ls9.fits.fz c4d_180513_031444_ooi_r_ls9-S22 r exptime 166.0 propid 2014B-0404 seeing 0.73 MJD 58251.135 object DECaLS_38604_r
   c4d_180221_085747_ooi_g_ls9.fits.fz c4d_180221_085747_ooi_g_ls9-S22 g exptime 173.0 propid 2014B-0404 seeing 1.43 MJD 58170.373 object DECaLS_38604_g
   c4d_160226_090239_ooi_z_ls9.fits.fz c4d_160226_090239_ooi_z_ls9-S22 z exptime 140.0 propid 2014B-0404 seeing 0.96 MJD 57444.375 object DECaLS_38604_z
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082816_oi[S24,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082816_oi[S18,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081104_oi[S23,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T085930_oi[S22,noinherit][2046,4094][real]: DECaLS_
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082816_oi[S18,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082816_oi[S24,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081104_oi[S23,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2a/CP20180212/c4d_180213_
090013_ooi_r_ls9.fits.fz

Something went wrong; please check the logfile.
ERROR: galaxy PGC2028578; please check the logfile.
Finished galaxy PGC2028578 in 0.101 minutes.
Started working on galaxy PGC2028578 at Sat Feb  8 08:31:53 2020
RA, Dec=216.988866, 33.289186: run=decam (11 north CCDs, 13 south CCDs).
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.988866 33.2891858 --width 361 --height 361 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T08:31:54.476272
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '216.988866', '33.2891858', '--width', '361', '--height', '361', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.988866 33.2891858 --width 361 --height 361 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 216.988866 33.2891858
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 15841 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 15842 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 15843 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 15844 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 15845 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 15846 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 15847 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 15848 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T08:31:55.729810
Keeping 13 CCDs:
   c4d_180213_090013_ooi_r_ls9.fits.fz c4d_180213_090013_ooi_r_ls9-S24 r exptime 52.0 propid 2014B-0404 seeing 1.05 MJD 58162.375 object DECaLS_6860_r
   c4d_170422_052010_ooi_g_ls9.fits.fz c4d_170422_052010_ooi_g_ls9-S24 g exptime 125.0 propid 2014B-0404 seeing 1.45 MJD 57865.220 object DECaLS_6860_g
   c4d_160228_083134_ooi_z_ls9.fits.fz c4d_160228_083134_ooi_z_ls9-S24 z exptime 147.0 propid 2014B-0404 seeing 0.97 MJD 57446.353 object DECaLS_6860_z
   c4d_170717_235713_ooi_g_ls9.fits.fz c4d_170717_235713_ooi_g_ls9-S23 g exptime 169.0 propid 2014B-0404 seeing 1.65 MJD 57951.996 object DECaLS_22732_g
   c4d_170717_235355_ooi_r_ls9.fits.fz c4d_170717_235355_ooi_r_ls9-S23 r exptime 96.0 propid 2014B-0404 seeing 1.46 MJD 57951.994 object DECaLS_22732_r
   c4d_160226_081409_ooi_z_ls9.fits.fz c4d_160226_081409_ooi_z_ls9-S23 z exptime 135.0 propid 2014B-0404 seeing 1.04 MJD 57444.341 object DECaLS_22732_z
   c4d_180213_090013_ooi_r_ls9.fits.fz c4d_180213_090013_ooi_r_ls9-S18 r exptime 52.0 propid 2014B-0404 seeing 1.13 MJD 58162.375 object DECaLS_6860_r
   c4d_160228_083134_ooi_z_ls9.fits.fz c4d_160228_083134_ooi_z_ls9-S18 z exptime 147.0 propid 2014B-0404 seeing 1.04 MJD 57446.353 object DECaLS_6860_z
   c4d_170422_052010_ooi_g_ls9.fits.fz c4d_170422_052010_ooi_g_ls9-S18 g exptime 125.0 propid 2014B-0404 seeing 1.38 MJD 57865.220 object DECaLS_6860_g
   c4d_190307_071335_ooi_r_ls9.fits.fz c4d_190307_071335_ooi_r_ls9-S22 r exptime 49.0 propid 2014B-0404 seeing 1.44 MJD 58549.301 object DECaLS_38604_r
   c4d_180513_031444_ooi_r_ls9.fits.fz c4d_180513_031444_ooi_r_ls9-S22 r exptime 166.0 propid 2014B-0404 seeing 0.73 MJD 58251.135 object DECaLS_38604_r
   c4d_180221_085747_ooi_g_ls9.fits.fz c4d_180221_085747_ooi_g_ls9-S22 g exptime 173.0 propid 2014B-0404 seeing 1.43 MJD 58170.373 object DECaLS_38604_g
   c4d_160226_090239_ooi_z_ls9.fits.fz c4d_160226_090239_ooi_z_ls9-S22 z exptime 140.0 propid 2014B-0404 seeing 0.96 MJD 57444.375 object DECaLS_38604_z
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082816_oi[S24,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082816_oi[S18,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081104_oi[S23,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T085930_oi[S22,noinherit][2046,4094][real]: DECaLS_
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082816_oi[S18,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082816_oi[S24,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081104_oi[S23,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2a/CP20180212/c4d_180213_
090013_ooi_r_ls9.fits.fz

Something went wrong; please check the logfile.
ERROR: galaxy PGC2028578; please check the logfile.
Finished galaxy PGC2028578 in 0.102 minutes.
Started working on galaxy PGC2028578 at Sat Feb  8 13:05:45 2020
RA, Dec=216.988866, 33.289186: run=decam (11 north CCDs, 13 south CCDs).
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.988866 33.2891858 --width 361 --height 361 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T13:05:46.092029
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '216.988866', '33.2891858', '--width', '361', '--height', '361', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.988866 33.2891858 --width 361 --height 361 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 216.988866 33.2891858
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 13874 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 13875 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 13876 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 13877 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 13878 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 13880 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 13881 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 13879 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T13:05:47.356065
Keeping 13 CCDs:
   c4d_180213_090013_ooi_r_ls9.fits.fz c4d_180213_090013_ooi_r_ls9-S24 r exptime 52.0 propid 2014B-0404 seeing 1.05 MJD 58162.375 object DECaLS_6860_r
   c4d_170422_052010_ooi_g_ls9.fits.fz c4d_170422_052010_ooi_g_ls9-S24 g exptime 125.0 propid 2014B-0404 seeing 1.45 MJD 57865.220 object DECaLS_6860_g
   c4d_160228_083134_ooi_z_ls9.fits.fz c4d_160228_083134_ooi_z_ls9-S24 z exptime 147.0 propid 2014B-0404 seeing 0.97 MJD 57446.353 object DECaLS_6860_z
   c4d_170717_235713_ooi_g_ls9.fits.fz c4d_170717_235713_ooi_g_ls9-S23 g exptime 169.0 propid 2014B-0404 seeing 1.65 MJD 57951.996 object DECaLS_22732_g
   c4d_170717_235355_ooi_r_ls9.fits.fz c4d_170717_235355_ooi_r_ls9-S23 r exptime 96.0 propid 2014B-0404 seeing 1.46 MJD 57951.994 object DECaLS_22732_r
   c4d_160226_081409_ooi_z_ls9.fits.fz c4d_160226_081409_ooi_z_ls9-S23 z exptime 135.0 propid 2014B-0404 seeing 1.04 MJD 57444.341 object DECaLS_22732_z
   c4d_180213_090013_ooi_r_ls9.fits.fz c4d_180213_090013_ooi_r_ls9-S18 r exptime 52.0 propid 2014B-0404 seeing 1.13 MJD 58162.375 object DECaLS_6860_r
   c4d_160228_083134_ooi_z_ls9.fits.fz c4d_160228_083134_ooi_z_ls9-S18 z exptime 147.0 propid 2014B-0404 seeing 1.04 MJD 57446.353 object DECaLS_6860_z
   c4d_170422_052010_ooi_g_ls9.fits.fz c4d_170422_052010_ooi_g_ls9-S18 g exptime 125.0 propid 2014B-0404 seeing 1.38 MJD 57865.220 object DECaLS_6860_g
   c4d_190307_071335_ooi_r_ls9.fits.fz c4d_190307_071335_ooi_r_ls9-S22 r exptime 49.0 propid 2014B-0404 seeing 1.44 MJD 58549.301 object DECaLS_38604_r
   c4d_180513_031444_ooi_r_ls9.fits.fz c4d_180513_031444_ooi_r_ls9-S22 r exptime 166.0 propid 2014B-0404 seeing 0.73 MJD 58251.135 object DECaLS_38604_r
   c4d_180221_085747_ooi_g_ls9.fits.fz c4d_180221_085747_ooi_g_ls9-S22 g exptime 173.0 propid 2014B-0404 seeing 1.43 MJD 58170.373 object DECaLS_38604_g
   c4d_160226_090239_ooi_z_ls9.fits.fz c4d_160226_090239_ooi_z_ls9-S22 z exptime 140.0 propid 2014B-0404 seeing 0.96 MJD 57444.375 object DECaLS_38604_z
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082816_oi[S18,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082816_oi[S24,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T085930_oi[S22,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081104_oi[S23,noinherit][2046,4094][real]: DECaLS_
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082816_oi[S18,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082816_oi[S24,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T085930_oi[S22,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081104_oi[S23,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2/CP20170717/c4d_170717_2
35713_ooi_g_ls9.fits.fz

Something went wrong; please check the logfile.
ERROR: galaxy PGC2028578; please check the logfile.
Finished galaxy PGC2028578 in 0.094 minutes.
Started working on galaxy PGC2028578 at Sun Feb  9 09:48:02 2020
RA, Dec=216.988866, 33.289186: run=decam (11 north CCDs, 13 south CCDs).
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.988866 33.2891858 --width 361 --height 361 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-09T09:48:03.237292
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '216.988866', '33.2891858', '--width', '361', '--height', '361', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.988866 33.2891858 --width 361 --height 361 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 216.988866 33.2891858
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 34529 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 34530 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 34531 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 34532 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 34533 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 34534 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 34535 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 34536 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 34537 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 34538 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 34539 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 34540 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 34541 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 34542 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 34543 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 34544 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 34545 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 34546 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072204 MB
Starting process 34547 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072204 MB
Starting process 34548 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071916 MB
Starting process 34549 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07192 MB
Starting process 34550 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071556 MB
Starting process 34551 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 34552 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 34553 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 34554 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 34555 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 34556 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 34557 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 34558 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 34559 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 34560 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-09T09:48:04.492842
Keeping 13 CCDs:
   c4d_180213_090013_ooi_r_ls9.fits.fz c4d_180213_090013_ooi_r_ls9-S24 r exptime 52.0 propid 2014B-0404 seeing 1.05 MJD 58162.375 object DECaLS_6860_r
   c4d_170422_052010_ooi_g_ls9.fits.fz c4d_170422_052010_ooi_g_ls9-S24 g exptime 125.0 propid 2014B-0404 seeing 1.45 MJD 57865.220 object DECaLS_6860_g
   c4d_160228_083134_ooi_z_ls9.fits.fz c4d_160228_083134_ooi_z_ls9-S24 z exptime 147.0 propid 2014B-0404 seeing 0.97 MJD 57446.353 object DECaLS_6860_z
   c4d_170717_235713_ooi_g_ls9.fits.fz c4d_170717_235713_ooi_g_ls9-S23 g exptime 169.0 propid 2014B-0404 seeing 1.65 MJD 57951.996 object DECaLS_22732_g
   c4d_170717_235355_ooi_r_ls9.fits.fz c4d_170717_235355_ooi_r_ls9-S23 r exptime 96.0 propid 2014B-0404 seeing 1.46 MJD 57951.994 object DECaLS_22732_r
   c4d_160226_081409_ooi_z_ls9.fits.fz c4d_160226_081409_ooi_z_ls9-S23 z exptime 135.0 propid 2014B-0404 seeing 1.04 MJD 57444.341 object DECaLS_22732_z
   c4d_180213_090013_ooi_r_ls9.fits.fz c4d_180213_090013_ooi_r_ls9-S18 r exptime 52.0 propid 2014B-0404 seeing 1.13 MJD 58162.375 object DECaLS_6860_r
   c4d_160228_083134_ooi_z_ls9.fits.fz c4d_160228_083134_ooi_z_ls9-S18 z exptime 147.0 propid 2014B-0404 seeing 1.04 MJD 57446.353 object DECaLS_6860_z
   c4d_170422_052010_ooi_g_ls9.fits.fz c4d_170422_052010_ooi_g_ls9-S18 g exptime 125.0 propid 2014B-0404 seeing 1.38 MJD 57865.220 object DECaLS_6860_g
   c4d_190307_071335_ooi_r_ls9.fits.fz c4d_190307_071335_ooi_r_ls9-S22 r exptime 49.0 propid 2014B-0404 seeing 1.44 MJD 58549.301 object DECaLS_38604_r
   c4d_180513_031444_ooi_r_ls9.fits.fz c4d_180513_031444_ooi_r_ls9-S22 r exptime 166.0 propid 2014B-0404 seeing 0.73 MJD 58251.135 object DECaLS_38604_r
   c4d_180221_085747_ooi_g_ls9.fits.fz c4d_180221_085747_ooi_g_ls9-S22 g exptime 173.0 propid 2014B-0404 seeing 1.43 MJD 58170.373 object DECaLS_38604_g
   c4d_160226_090239_ooi_z_ls9.fits.fz c4d_160226_090239_ooi_z_ls9-S22 z exptime 140.0 propid 2014B-0404 seeing 0.96 MJD 57444.375 object DECaLS_38604_z
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081104_oi[S23,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082816_oi[S18,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T085930_oi[S22,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082816_oi[S24,noinherit][2046,4094][real]: DECaLS_
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081104_oi[S23,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082816_oi[S24,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082816_oi[S18,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T085930_oi[S22,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2a/CP20180212/c4d_180213_
090013_ooi_r_ls9.fits.fz

Something went wrong; please check the logfile.
ERROR: galaxy PGC2028578; please check the logfile.
Finished galaxy PGC2028578 in 0.056 minutes.
Started working on galaxy PGC2028578 at Mon Feb 10 06:11:14 2020
RA, Dec=216.988866, 33.289186: run=decam (11 north CCDs, 13 south CCDs).
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.988866 33.2891858 --width 361 --height 361 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-10T06:11:15.655127
legacypipe git version: dr9e-pre1-9-g4a51be93
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '216.988866', '33.2891858', '--width', '361', '--height', '361', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', 'decam', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 216.988866 33.2891858 --width 361 --height 361 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run decam --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/216/PGC2028578/PGC2028578-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 216.988866 33.2891858
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 10602 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 10603 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 10608 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 10609 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 10610 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 10611 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 10612 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 10613 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 10614 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 10615 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 10616 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 10617 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 10618 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071156 MB
Starting process 10619 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071156 MB
Starting process 10620 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071156 MB
Starting process 10621 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07116 MB
Starting process 10622 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 10623 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072176 MB
Starting process 10624 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072176 MB
Starting process 10625 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07218 MB
Starting process 10626 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072176 MB
Starting process 10627 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 10628 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07152 MB
Starting process 10629 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07152 MB
Starting process 10630 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07152 MB
Starting process 10631 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07152 MB
Starting process 10632 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07152 MB
Starting process 10633 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07152 MB
Starting process 10634 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071456 MB
Starting process 10635 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071456 MB
Starting process 10636 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07152 MB
Starting process 10637 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071456 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-10T06:11:16.952304
Keeping 13 CCDs:
   c4d_180213_090013_ooi_r_ls9.fits.fz c4d_180213_090013_ooi_r_ls9-S24 r exptime 52.0 propid 2014B-0404 seeing 1.05 MJD 58162.375 object DECaLS_6860_r
   c4d_170422_052010_ooi_g_ls9.fits.fz c4d_170422_052010_ooi_g_ls9-S24 g exptime 125.0 propid 2014B-0404 seeing 1.45 MJD 57865.220 object DECaLS_6860_g
   c4d_160228_083134_ooi_z_ls9.fits.fz c4d_160228_083134_ooi_z_ls9-S24 z exptime 147.0 propid 2014B-0404 seeing 0.97 MJD 57446.353 object DECaLS_6860_z
   c4d_170717_235713_ooi_g_ls9.fits.fz c4d_170717_235713_ooi_g_ls9-S23 g exptime 169.0 propid 2014B-0404 seeing 1.65 MJD 57951.996 object DECaLS_22732_g
   c4d_170717_235355_ooi_r_ls9.fits.fz c4d_170717_235355_ooi_r_ls9-S23 r exptime 96.0 propid 2014B-0404 seeing 1.46 MJD 57951.994 object DECaLS_22732_r
   c4d_160226_081409_ooi_z_ls9.fits.fz c4d_160226_081409_ooi_z_ls9-S23 z exptime 135.0 propid 2014B-0404 seeing 1.04 MJD 57444.341 object DECaLS_22732_z
   c4d_180213_090013_ooi_r_ls9.fits.fz c4d_180213_090013_ooi_r_ls9-S18 r exptime 52.0 propid 2014B-0404 seeing 1.13 MJD 58162.375 object DECaLS_6860_r
   c4d_160228_083134_ooi_z_ls9.fits.fz c4d_160228_083134_ooi_z_ls9-S18 z exptime 147.0 propid 2014B-0404 seeing 1.04 MJD 57446.353 object DECaLS_6860_z
   c4d_170422_052010_ooi_g_ls9.fits.fz c4d_170422_052010_ooi_g_ls9-S18 g exptime 125.0 propid 2014B-0404 seeing 1.38 MJD 57865.220 object DECaLS_6860_g
   c4d_190307_071335_ooi_r_ls9.fits.fz c4d_190307_071335_ooi_r_ls9-S22 r exptime 49.0 propid 2014B-0404 seeing 1.44 MJD 58549.301 object DECaLS_38604_r
   c4d_180513_031444_ooi_r_ls9.fits.fz c4d_180513_031444_ooi_r_ls9-S22 r exptime 166.0 propid 2014B-0404 seeing 0.73 MJD 58251.135 object DECaLS_38604_r
   c4d_180221_085747_ooi_g_ls9.fits.fz c4d_180221_085747_ooi_g_ls9-S22 g exptime 173.0 propid 2014B-0404 seeing 1.43 MJD 58170.373 object DECaLS_38604_g
   c4d_160226_090239_ooi_z_ls9.fits.fz c4d_160226_090239_ooi_z_ls9-S22 z exptime 140.0 propid 2014B-0404 seeing 0.96 MJD 57444.375 object DECaLS_38604_z
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081104_oi[S23,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082816_oi[S24,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082816_oi[S18,noinherit][2046,4094][real]: DECaLS_
The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T085930_oi[S22,noinherit][2046,4094][real]: DECaLS_
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T081104_oi[S23,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_954f856-z-20160228T082816_oi[S18,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
WARNING: The following header keyword is invalid or follows an unrecognized non-standard convention:
DECALS_DR9_95508ff-z-20160226T085930_oi[S22,noinherit][2046,4094][real]: DECaLS_ [astropy.io.fits.card]
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
OSError: FITSIO status = 104: could not open the named file
failed to find or open the following file: (ffopen)
/global/cscratch1/sd/desimpp/dr9e/images/decam/CP/V4.8.2/CP20170421/c4d_170422_0
52010_ooi_g_ls9.fits.fz

Something went wrong; please check the logfile.
ERROR: galaxy PGC2028578; please check the logfile.
Finished galaxy PGC2028578 in 0.105 minutes.