Started working on galaxy UGC04998 at Sat Feb  8 08:51:20 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 141.29875049999998 68.38319440000001 --width 585 --height 585 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T08:51:20.647018
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '141.29875049999998', '68.38319440000001', '--width', '585', '--height', '585', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 141.29875049999998 68.38319440000001 --width 585 --height 585 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 141.29875049999998 68.38319440000001
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 30149 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 30150 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 30151 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 30152 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 30153 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 30154 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 30155 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 30156 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T08:51:22.014778
Keeping 11 CCDs:
   ksb_190128_103155_ooi_r_ls9.fits.fz ksb_190128_103155_ooi_r_ls9-CCD3 r exptime 144.023 propid NOTBASS    seeing 1.18 MJD 58511.439 object 
   ksb_190111_110456_ooi_g_ls9.fits.fz ksb_190111_110456_ooi_g_ls9-CCD3 g exptime 123.643 propid NOTBASS    seeing 2.03 MJD 58494.462 object 
   ksb_180411_063614_ooi_g_ls9.fits.fz ksb_180411_063614_ooi_g_ls9-CCD3 g exptime 178.794 propid BASS       seeing 1.94 MJD 58219.275 object 44282
   ksb_190131_112335_ooi_r_ls9.fits.fz ksb_190131_112335_ooi_r_ls9-CCD3 r exptime 112.96 propid NOTBASS    seeing 1.50 MJD 58514.475 object 
   ksb_190127_065527_ooi_g_ls9.fits.fz ksb_190127_065527_ooi_g_ls9-CCD4 g exptime 212.212 propid NOTBASS    seeing 2.38 MJD 58510.289 object 
   ksb_190209_090746_ooi_r_ls9.fits.fz ksb_190209_090746_ooi_r_ls9-CCD4 r exptime 142.272 propid NOTBASS    seeing 1.54 MJD 58523.380 object 
   k4m_160125_090017_ooi_zd_ls9.fits.fz k4m_160125_090017_ooi_zd_ls9-CCD2 z exptime 116.103 propid 2016B-0453 seeing 1.11 MJD 57412.375 object MzLS_709451_z
   k4m_160320_061101_ooi_zd_ls9.fits.fz k4m_160320_061101_ooi_zd_ls9-CCD4 z exptime 91.104 propid 2016A-0453 seeing 0.83 MJD 57467.258 object MzLS_709920_z
   k4m_170522_050359_ooi_zd_ls9.fits.fz k4m_170522_050359_ooi_zd_ls9-CCD3 z exptime 184.104 propid 2016A-0453 seeing 1.39 MJD 57895.211 object MzLS_710386_z
   k4m_160320_061101_ooi_zd_ls9.fits.fz k4m_160320_061101_ooi_zd_ls9-CCD2 z exptime 91.104 propid 2016A-0453 seeing 0.83 MJD 57467.258 object MzLS_709920_z
   k4m_160127_083211_ooi_zd_ls9.fits.fz k4m_160127_083211_ooi_zd_ls9-CCD4 z exptime 107.103 propid 2016A-0453 seeing 0.99 MJD 57414.356 object MzLS_709450_z
Faked up EXPNUM 84940074
Faked up EXPNUM 84940074
Faked up EXPNUM 84940074
90prime get_fwhm: self.fwhm = 4.47
Faked up EXPNUM 85140220
Faked up EXPNUM 85140220
Faked up EXPNUM 85140220
90prime get_fwhm: self.fwhm = 3.31
WARNING: EXPNUM None!=160125090017 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_090017_ooi_zd_ls9.fits.fz header
mosaic get_fwhm: self.fwhm = 5.37
Faked up EXPNUM 85110236
Faked up EXPNUM 85110236
Faked up EXPNUM 85110236
90prime get_fwhm: self.fwhm = 2.61
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
90prime get_fwhm: self.fwhm = 3.39
WARNING: image median 0.017266002 is more than 1 sigma away from zero!
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
90prime get_fwhm: self.fwhm = 5.25
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy UGC04998; please check the logfile.
Finished galaxy UGC04998 in 0.073 minutes.
Started working on galaxy UGC04998 at Sat Feb  8 12:46:38 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 141.29875049999998 68.38319440000001 --width 585 --height 585 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T12:46:38.679245
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '141.29875049999998', '68.38319440000001', '--width', '585', '--height', '585', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 141.29875049999998 68.38319440000001 --width 585 --height 585 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 141.29875049999998 68.38319440000001
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 29699 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 29700 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 29701 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 29702 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 29703 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 29704 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 29705 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 29706 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T12:46:40.070503
Keeping 11 CCDs:
   ksb_190128_103155_ooi_r_ls9.fits.fz ksb_190128_103155_ooi_r_ls9-CCD3 r exptime 144.023 propid NOTBASS    seeing 1.18 MJD 58511.439 object 
   ksb_190111_110456_ooi_g_ls9.fits.fz ksb_190111_110456_ooi_g_ls9-CCD3 g exptime 123.643 propid NOTBASS    seeing 2.03 MJD 58494.462 object 
   ksb_180411_063614_ooi_g_ls9.fits.fz ksb_180411_063614_ooi_g_ls9-CCD3 g exptime 178.794 propid BASS       seeing 1.94 MJD 58219.275 object 44282
   ksb_190131_112335_ooi_r_ls9.fits.fz ksb_190131_112335_ooi_r_ls9-CCD3 r exptime 112.96 propid NOTBASS    seeing 1.50 MJD 58514.475 object 
   ksb_190127_065527_ooi_g_ls9.fits.fz ksb_190127_065527_ooi_g_ls9-CCD4 g exptime 212.212 propid NOTBASS    seeing 2.38 MJD 58510.289 object 
   ksb_190209_090746_ooi_r_ls9.fits.fz ksb_190209_090746_ooi_r_ls9-CCD4 r exptime 142.272 propid NOTBASS    seeing 1.54 MJD 58523.380 object 
   k4m_160125_090017_ooi_zd_ls9.fits.fz k4m_160125_090017_ooi_zd_ls9-CCD2 z exptime 116.103 propid 2016B-0453 seeing 1.11 MJD 57412.375 object MzLS_709451_z
   k4m_160320_061101_ooi_zd_ls9.fits.fz k4m_160320_061101_ooi_zd_ls9-CCD4 z exptime 91.104 propid 2016A-0453 seeing 0.83 MJD 57467.258 object MzLS_709920_z
   k4m_170522_050359_ooi_zd_ls9.fits.fz k4m_170522_050359_ooi_zd_ls9-CCD3 z exptime 184.104 propid 2016A-0453 seeing 1.39 MJD 57895.211 object MzLS_710386_z
   k4m_160320_061101_ooi_zd_ls9.fits.fz k4m_160320_061101_ooi_zd_ls9-CCD2 z exptime 91.104 propid 2016A-0453 seeing 0.83 MJD 57467.258 object MzLS_709920_z
   k4m_160127_083211_ooi_zd_ls9.fits.fz k4m_160127_083211_ooi_zd_ls9-CCD4 z exptime 107.103 propid 2016A-0453 seeing 0.99 MJD 57414.356 object MzLS_709450_z
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
90prime get_fwhm: self.fwhm = 3.39
WARNING: image median 0.017266002 is more than 1 sigma away from zero!
Faked up EXPNUM 84940074
Faked up EXPNUM 84940074
Faked up EXPNUM 84940074
90prime get_fwhm: self.fwhm = 4.47
WARNING: EXPNUM None!=160125090017 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_090017_ooi_zd_ls9.fits.fz header
mosaic get_fwhm: self.fwhm = 5.37
Faked up EXPNUM 85140220
Faked up EXPNUM 85140220
Faked up EXPNUM 85140220
90prime get_fwhm: self.fwhm = 3.31
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
90prime get_fwhm: self.fwhm = 5.25
Faked up EXPNUM 85110236
Faked up EXPNUM 85110236
Faked up EXPNUM 85110236
90prime get_fwhm: self.fwhm = 2.61
mosaic get_fwhm: self.fwhm = 3.21
mosaic get_fwhm: self.fwhm = 3.21
Faked up EXPNUM 82190088
Faked up EXPNUM 82190088
Faked up EXPNUM 82190088
90prime get_fwhm: self.fwhm = 4.27
mosaic get_fwhm: self.fwhm = 3.83
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy UGC04998; please check the logfile.
Finished galaxy UGC04998 in 0.074 minutes.
Started working on galaxy UGC04998 at Sun Feb  9 08:14:02 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 141.29875049999998 68.38319440000001 --width 585 --height 585 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-09T08:14:03.059108
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '141.29875049999998', '68.38319440000001', '--width', '585', '--height', '585', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 141.29875049999998 68.38319440000001 --width 585 --height 585 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 141.29875049999998 68.38319440000001
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 28030 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 28031 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 28032 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 28033 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 28034 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 28035 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 28036 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 28037 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 28038 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 28039 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 28040 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 28041 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 28042 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 28043 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 28044 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 28045 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 28046 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 28047 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072204 MB
Starting process 28048 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072204 MB
Starting process 28049 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071924 MB
Starting process 28050 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072076 MB
Starting process 28051 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 28052 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 28053 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 28054 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 28055 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 28056 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071484 MB
Starting process 28057 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071484 MB
Starting process 28058 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071484 MB
Starting process 28059 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071484 MB
Starting process 28060 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071484 MB
Starting process 28061 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071484 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-09T08:14:04.277702
Keeping 11 CCDs:
   ksb_190128_103155_ooi_r_ls9.fits.fz ksb_190128_103155_ooi_r_ls9-CCD3 r exptime 144.023 propid NOTBASS    seeing 1.18 MJD 58511.439 object 
   ksb_190111_110456_ooi_g_ls9.fits.fz ksb_190111_110456_ooi_g_ls9-CCD3 g exptime 123.643 propid NOTBASS    seeing 2.03 MJD 58494.462 object 
   ksb_180411_063614_ooi_g_ls9.fits.fz ksb_180411_063614_ooi_g_ls9-CCD3 g exptime 178.794 propid BASS       seeing 1.94 MJD 58219.275 object 44282
   ksb_190131_112335_ooi_r_ls9.fits.fz ksb_190131_112335_ooi_r_ls9-CCD3 r exptime 112.96 propid NOTBASS    seeing 1.50 MJD 58514.475 object 
   ksb_190127_065527_ooi_g_ls9.fits.fz ksb_190127_065527_ooi_g_ls9-CCD4 g exptime 212.212 propid NOTBASS    seeing 2.38 MJD 58510.289 object 
   ksb_190209_090746_ooi_r_ls9.fits.fz ksb_190209_090746_ooi_r_ls9-CCD4 r exptime 142.272 propid NOTBASS    seeing 1.54 MJD 58523.380 object 
   k4m_160125_090017_ooi_zd_ls9.fits.fz k4m_160125_090017_ooi_zd_ls9-CCD2 z exptime 116.103 propid 2016B-0453 seeing 1.11 MJD 57412.375 object MzLS_709451_z
   k4m_160320_061101_ooi_zd_ls9.fits.fz k4m_160320_061101_ooi_zd_ls9-CCD4 z exptime 91.104 propid 2016A-0453 seeing 0.83 MJD 57467.258 object MzLS_709920_z
   k4m_170522_050359_ooi_zd_ls9.fits.fz k4m_170522_050359_ooi_zd_ls9-CCD3 z exptime 184.104 propid 2016A-0453 seeing 1.39 MJD 57895.211 object MzLS_710386_z
   k4m_160320_061101_ooi_zd_ls9.fits.fz k4m_160320_061101_ooi_zd_ls9-CCD2 z exptime 91.104 propid 2016A-0453 seeing 0.83 MJD 57467.258 object MzLS_709920_z
   k4m_160127_083211_ooi_zd_ls9.fits.fz k4m_160127_083211_ooi_zd_ls9-CCD4 z exptime 107.103 propid 2016A-0453 seeing 0.99 MJD 57414.356 object MzLS_709450_z
WARNING: EXPNUM None!=160125090017 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_090017_ooi_zd_ls9.fits.fz header
mosaic get_fwhm: self.fwhm = 3.21
mosaic get_fwhm: self.fwhm = 3.21
Faked up EXPNUM 85140220
Faked up EXPNUM 85140220
Faked up EXPNUM 85140220
90prime get_fwhm: self.fwhm = 3.31
mosaic get_fwhm: self.fwhm = 5.37
Faked up EXPNUM 82190088
Faked up EXPNUM 82190088
Faked up EXPNUM 82190088
90prime get_fwhm: self.fwhm = 4.27
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
90prime get_fwhm: self.fwhm = 5.25
Faked up EXPNUM 84940074
Faked up EXPNUM 84940074
Faked up EXPNUM 84940074
90prime get_fwhm: self.fwhm = 4.47
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
90prime get_fwhm: self.fwhm = 3.39
WARNING: image median 0.017266002 is more than 1 sigma away from zero!
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy UGC04998; please check the logfile.
Finished galaxy UGC04998 in 0.056 minutes.
Started working on galaxy UGC04998 at Sun Feb  9 10:13:12 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 141.29875049999998 68.38319440000001 --width 585 --height 585 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-09T10:13:12.598456
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '141.29875049999998', '68.38319440000001', '--width', '585', '--height', '585', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 141.29875049999998 68.38319440000001 --width 585 --height 585 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 141.29875049999998 68.38319440000001
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 41931 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 41932 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 41933 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 41934 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 41935 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 41936 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 41937 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 41938 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 41939 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 41940 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 41941 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 41942 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 41943 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 41944 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 41945 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 41946 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 41947 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071196 MB
Starting process 41948 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072212 MB
Starting process 41949 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072212 MB
Starting process 41950 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071924 MB
Starting process 41951 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07192 MB
Starting process 41952 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071556 MB
Starting process 41953 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 41954 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 41955 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 41956 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 41957 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 41958 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 41959 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 41960 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 41961 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 41962 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-09T10:13:13.869042
Keeping 11 CCDs:
   ksb_190128_103155_ooi_r_ls9.fits.fz ksb_190128_103155_ooi_r_ls9-CCD3 r exptime 144.023 propid NOTBASS    seeing 1.18 MJD 58511.439 object 
   ksb_190111_110456_ooi_g_ls9.fits.fz ksb_190111_110456_ooi_g_ls9-CCD3 g exptime 123.643 propid NOTBASS    seeing 2.03 MJD 58494.462 object 
   ksb_180411_063614_ooi_g_ls9.fits.fz ksb_180411_063614_ooi_g_ls9-CCD3 g exptime 178.794 propid BASS       seeing 1.94 MJD 58219.275 object 44282
   ksb_190131_112335_ooi_r_ls9.fits.fz ksb_190131_112335_ooi_r_ls9-CCD3 r exptime 112.96 propid NOTBASS    seeing 1.50 MJD 58514.475 object 
   ksb_190127_065527_ooi_g_ls9.fits.fz ksb_190127_065527_ooi_g_ls9-CCD4 g exptime 212.212 propid NOTBASS    seeing 2.38 MJD 58510.289 object 
   ksb_190209_090746_ooi_r_ls9.fits.fz ksb_190209_090746_ooi_r_ls9-CCD4 r exptime 142.272 propid NOTBASS    seeing 1.54 MJD 58523.380 object 
   k4m_160125_090017_ooi_zd_ls9.fits.fz k4m_160125_090017_ooi_zd_ls9-CCD2 z exptime 116.103 propid 2016B-0453 seeing 1.11 MJD 57412.375 object MzLS_709451_z
   k4m_160320_061101_ooi_zd_ls9.fits.fz k4m_160320_061101_ooi_zd_ls9-CCD4 z exptime 91.104 propid 2016A-0453 seeing 0.83 MJD 57467.258 object MzLS_709920_z
   k4m_170522_050359_ooi_zd_ls9.fits.fz k4m_170522_050359_ooi_zd_ls9-CCD3 z exptime 184.104 propid 2016A-0453 seeing 1.39 MJD 57895.211 object MzLS_710386_z
   k4m_160320_061101_ooi_zd_ls9.fits.fz k4m_160320_061101_ooi_zd_ls9-CCD2 z exptime 91.104 propid 2016A-0453 seeing 0.83 MJD 57467.258 object MzLS_709920_z
   k4m_160127_083211_ooi_zd_ls9.fits.fz k4m_160127_083211_ooi_zd_ls9-CCD4 z exptime 107.103 propid 2016A-0453 seeing 0.99 MJD 57414.356 object MzLS_709450_z
WARNING: EXPNUM None!=160125090017 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_090017_ooi_zd_ls9.fits.fz header
mosaic get_fwhm: self.fwhm = 3.21
Faked up EXPNUM 82190088
Faked up EXPNUM 82190088
Faked up EXPNUM 82190088
90prime get_fwhm: self.fwhm = 4.27
Faked up EXPNUM 85140220
Faked up EXPNUM 85140220
Faked up EXPNUM 85140220
90prime get_fwhm: self.fwhm = 3.31
mosaic get_fwhm: self.fwhm = 3.21
Faked up EXPNUM 85110236
Faked up EXPNUM 85110236
Faked up EXPNUM 85110236
90prime get_fwhm: self.fwhm = 2.61
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
90prime get_fwhm: self.fwhm = 5.25
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
90prime get_fwhm: self.fwhm = 3.39
WARNING: image median 0.017266002 is more than 1 sigma away from zero!
Faked up EXPNUM 84940074
Faked up EXPNUM 84940074
Faked up EXPNUM 84940074
90prime get_fwhm: self.fwhm = 4.47
mosaic get_fwhm: self.fwhm = 3.83
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy UGC04998; please check the logfile.
Finished galaxy UGC04998 in 0.059 minutes.
Started working on galaxy UGC04998 at Mon Feb 10 06:15:52 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 141.29875049999998 68.38319440000001 --width 585 --height 585 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-10T06:15:53.080171
legacypipe git version: dr9e-pre1-9-g4a51be93
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '141.29875049999998', '68.38319440000001', '--width', '585', '--height', '585', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 141.29875049999998 68.38319440000001 --width 585 --height 585 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/141/UGC04998/UGC04998-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 141.29875049999998 68.38319440000001
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 14132 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 14133 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 14134 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 14135 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 14136 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 14137 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 14138 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 14139 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 14140 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 14141 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 14142 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 14143 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 14144 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 14145 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 14146 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 14147 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 14148 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071156 MB
Starting process 14149 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072168 MB
Starting process 14150 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072168 MB
Starting process 14151 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071876 MB
Starting process 14152 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071876 MB
Starting process 14153 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071508 MB
Starting process 14154 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071508 MB
Starting process 14155 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071508 MB
Starting process 14156 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071508 MB
Starting process 14157 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071508 MB
Starting process 14158 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071508 MB
Starting process 14159 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071508 MB
Starting process 14160 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071444 MB
Starting process 14161 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071444 MB
Starting process 14162 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071508 MB
Starting process 14163 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071444 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-10T06:15:54.344079
Keeping 11 CCDs:
   ksb_190128_103155_ooi_r_ls9.fits.fz ksb_190128_103155_ooi_r_ls9-CCD3 r exptime 144.023 propid NOTBASS    seeing 1.18 MJD 58511.439 object 
   ksb_190111_110456_ooi_g_ls9.fits.fz ksb_190111_110456_ooi_g_ls9-CCD3 g exptime 123.643 propid NOTBASS    seeing 2.03 MJD 58494.462 object 
   ksb_180411_063614_ooi_g_ls9.fits.fz ksb_180411_063614_ooi_g_ls9-CCD3 g exptime 178.794 propid BASS       seeing 1.94 MJD 58219.275 object 44282
   ksb_190131_112335_ooi_r_ls9.fits.fz ksb_190131_112335_ooi_r_ls9-CCD3 r exptime 112.96 propid NOTBASS    seeing 1.50 MJD 58514.475 object 
   ksb_190127_065527_ooi_g_ls9.fits.fz ksb_190127_065527_ooi_g_ls9-CCD4 g exptime 212.212 propid NOTBASS    seeing 2.38 MJD 58510.289 object 
   ksb_190209_090746_ooi_r_ls9.fits.fz ksb_190209_090746_ooi_r_ls9-CCD4 r exptime 142.272 propid NOTBASS    seeing 1.54 MJD 58523.380 object 
   k4m_160125_090017_ooi_zd_ls9.fits.fz k4m_160125_090017_ooi_zd_ls9-CCD2 z exptime 116.103 propid 2016B-0453 seeing 1.11 MJD 57412.375 object MzLS_709451_z
   k4m_160320_061101_ooi_zd_ls9.fits.fz k4m_160320_061101_ooi_zd_ls9-CCD4 z exptime 91.104 propid 2016A-0453 seeing 0.83 MJD 57467.258 object MzLS_709920_z
   k4m_170522_050359_ooi_zd_ls9.fits.fz k4m_170522_050359_ooi_zd_ls9-CCD3 z exptime 184.104 propid 2016A-0453 seeing 1.39 MJD 57895.211 object MzLS_710386_z
   k4m_160320_061101_ooi_zd_ls9.fits.fz k4m_160320_061101_ooi_zd_ls9-CCD2 z exptime 91.104 propid 2016A-0453 seeing 0.83 MJD 57467.258 object MzLS_709920_z
   k4m_160127_083211_ooi_zd_ls9.fits.fz k4m_160127_083211_ooi_zd_ls9-CCD4 z exptime 107.103 propid 2016A-0453 seeing 0.99 MJD 57414.356 object MzLS_709450_z
WARNING: EXPNUM None!=160125090017 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_090017_ooi_zd_ls9.fits.fz header
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
90prime get_fwhm: self.fwhm = 5.25
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
90prime get_fwhm: self.fwhm = 3.39
WARNING: image median 0.017266002 is more than 1 sigma away from zero!
mosaic get_fwhm: self.fwhm = 3.21
Faked up EXPNUM 82190088
Faked up EXPNUM 82190088
Faked up EXPNUM 82190088
90prime get_fwhm: self.fwhm = 4.27
Faked up EXPNUM 85140220
Faked up EXPNUM 85140220
Faked up EXPNUM 85140220
90prime get_fwhm: self.fwhm = 3.31
mosaic get_fwhm: self.fwhm = 3.21
Faked up EXPNUM 84940074
Faked up EXPNUM 84940074
Faked up EXPNUM 84940074
90prime get_fwhm: self.fwhm = 4.47
Faked up EXPNUM 85110236
Faked up EXPNUM 85110236
Faked up EXPNUM 85110236
90prime get_fwhm: self.fwhm = 2.61
mosaic get_fwhm: self.fwhm = 3.83
mosaic get_fwhm: self.fwhm = 5.37
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy UGC04998; please check the logfile.
Finished galaxy UGC04998 in 0.062 minutes.