Started working on galaxy PGC2793669 at Sat Feb  8 08:27:26 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.654589 68.8764535 --width 383 --height 383 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T08:27:27.189272
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '146.654589', '68.8764535', '--width', '383', '--height', '383', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.654589 68.8764535 --width 383 --height 383 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 146.654589 68.8764535
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 20354 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 20355 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 20356 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 20357 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 20358 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 20359 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 20360 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 20361 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T08:27:28.605858
Keeping 9 CCDs:
   ksb_190112_101522_ooi_g_ls9.fits.fz ksb_190112_101522_ooi_g_ls9-CCD2 g exptime 124.63 propid NOTBASS    seeing 1.48 MJD 58495.427 object 
   ksb_190128_105804_ooi_r_ls9.fits.fz ksb_190128_105804_ooi_r_ls9-CCD2 r exptime 163.914 propid NOTBASS    seeing 1.35 MJD 58511.457 object 
   ksb_180411_072612_ooi_g_ls9.fits.fz ksb_180411_072612_ooi_g_ls9-CCD2 g exptime 215.859 propid BASS       seeing 1.48 MJD 58219.310 object 45082
   ksb_190131_114353_ooi_r_ls9.fits.fz ksb_190131_114353_ooi_r_ls9-CCD2 r exptime 139.363 propid NOTBASS    seeing 1.45 MJD 58514.489 object 
   ksb_190128_063416_ooi_g_ls9.fits.fz ksb_190128_063416_ooi_g_ls9-CCD3 g exptime 110.595 propid NOTBASS    seeing 1.49 MJD 58511.274 object 
   ksb_190209_094735_ooi_r_ls9.fits.fz ksb_190209_094735_ooi_r_ls9-CCD3 r exptime 129.274 propid NOTBASS    seeing 1.56 MJD 58523.408 object 
   k4m_170208_080820_ooi_zd_ls9.fits.fz k4m_170208_080820_ooi_zd_ls9-CCD3 z exptime 124.104 propid 2016A-0453 seeing 1.20 MJD 57792.339 object MzLS_711764_z
   k4m_171028_104055_ooi_zd_ls9.fits.fz k4m_171028_104055_ooi_zd_ls9-CCD3 z exptime 135.306 propid 2016A-0453 seeing 1.01 MJD 58054.445 object MzLS_711310_z
   k4m_160125_092739_ooi_zd_ls9.fits.fz k4m_160125_092739_ooi_zd_ls9-CCD2 z exptime 128.103 propid 2016B-0453 seeing 1.13 MJD 57412.394 object MzLS_710853_z
mosaic get_fwhm: self.fwhm = 3.89
WARNING: EXPNUM None!=160125092739 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_092739_ooi_zd_ls9.fits.fz header
Faked up EXPNUM 85230163
Faked up EXPNUM 85230163
Faked up EXPNUM 85230163
90prime get_fwhm: self.fwhm = 3.45
Faked up EXPNUM 85110144
Faked up EXPNUM 85110144
Faked up EXPNUM 85110144
90prime get_fwhm: self.fwhm = 3.29
Faked up EXPNUM 85110243
Faked up EXPNUM 85110243
Faked up EXPNUM 85110243
90prime get_fwhm: self.fwhm = 2.97
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC2793669; please check the logfile.
Finished galaxy PGC2793669 in 0.066 minutes.
Started working on galaxy PGC2793669 at Sat Feb  8 08:50:58 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.654589 68.8764535 --width 383 --height 383 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T08:50:58.917946
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '146.654589', '68.8764535', '--width', '383', '--height', '383', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.654589 68.8764535 --width 383 --height 383 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 146.654589 68.8764535
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 53057 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 53058 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 53059 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 53060 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 53061 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 53062 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 53063 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 53064 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T08:51:00.272788
Keeping 9 CCDs:
   ksb_190112_101522_ooi_g_ls9.fits.fz ksb_190112_101522_ooi_g_ls9-CCD2 g exptime 124.63 propid NOTBASS    seeing 1.48 MJD 58495.427 object 
   ksb_190128_105804_ooi_r_ls9.fits.fz ksb_190128_105804_ooi_r_ls9-CCD2 r exptime 163.914 propid NOTBASS    seeing 1.35 MJD 58511.457 object 
   ksb_180411_072612_ooi_g_ls9.fits.fz ksb_180411_072612_ooi_g_ls9-CCD2 g exptime 215.859 propid BASS       seeing 1.48 MJD 58219.310 object 45082
   ksb_190131_114353_ooi_r_ls9.fits.fz ksb_190131_114353_ooi_r_ls9-CCD2 r exptime 139.363 propid NOTBASS    seeing 1.45 MJD 58514.489 object 
   ksb_190128_063416_ooi_g_ls9.fits.fz ksb_190128_063416_ooi_g_ls9-CCD3 g exptime 110.595 propid NOTBASS    seeing 1.49 MJD 58511.274 object 
   ksb_190209_094735_ooi_r_ls9.fits.fz ksb_190209_094735_ooi_r_ls9-CCD3 r exptime 129.274 propid NOTBASS    seeing 1.56 MJD 58523.408 object 
   k4m_170208_080820_ooi_zd_ls9.fits.fz k4m_170208_080820_ooi_zd_ls9-CCD3 z exptime 124.104 propid 2016A-0453 seeing 1.20 MJD 57792.339 object MzLS_711764_z
   k4m_171028_104055_ooi_zd_ls9.fits.fz k4m_171028_104055_ooi_zd_ls9-CCD3 z exptime 135.306 propid 2016A-0453 seeing 1.01 MJD 58054.445 object MzLS_711310_z
   k4m_160125_092739_ooi_zd_ls9.fits.fz k4m_160125_092739_ooi_zd_ls9-CCD2 z exptime 128.103 propid 2016B-0453 seeing 1.13 MJD 57412.394 object MzLS_710853_z
mosaic get_fwhm: self.fwhm = 3.89
WARNING: EXPNUM None!=160125092739 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_092739_ooi_zd_ls9.fits.fz header
Faked up EXPNUM 85110144
Faked up EXPNUM 85110144
Faked up EXPNUM 85110144
90prime get_fwhm: self.fwhm = 3.29
Faked up EXPNUM 85230163
Faked up EXPNUM 85230163
Faked up EXPNUM 85230163
90prime get_fwhm: self.fwhm = 3.45
Faked up EXPNUM 84950057
Faked up EXPNUM 84950057
Faked up EXPNUM 84950057
90prime get_fwhm: self.fwhm = 3.27
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
90prime get_fwhm: self.fwhm = 3.19
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
90prime get_fwhm: self.fwhm = 3.27
mosaic get_fwhm: self.fwhm = 4.61
Faked up EXPNUM 85110243
Faked up EXPNUM 85110243
Faked up EXPNUM 85110243
90prime get_fwhm: self.fwhm = 2.97
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC2793669; please check the logfile.
Finished galaxy PGC2793669 in 0.065 minutes.
Started working on galaxy PGC2793669 at Sat Feb  8 12:47:12 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.654589 68.8764535 --width 383 --height 383 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T12:47:12.988945
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '146.654589', '68.8764535', '--width', '383', '--height', '383', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.654589 68.8764535 --width 383 --height 383 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 146.654589 68.8764535
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 11006 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 11007 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 11008 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 11009 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 11010 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 11011 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 11012 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 11013 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T12:47:14.260891
Keeping 9 CCDs:
   ksb_190112_101522_ooi_g_ls9.fits.fz ksb_190112_101522_ooi_g_ls9-CCD2 g exptime 124.63 propid NOTBASS    seeing 1.48 MJD 58495.427 object 
   ksb_190128_105804_ooi_r_ls9.fits.fz ksb_190128_105804_ooi_r_ls9-CCD2 r exptime 163.914 propid NOTBASS    seeing 1.35 MJD 58511.457 object 
   ksb_180411_072612_ooi_g_ls9.fits.fz ksb_180411_072612_ooi_g_ls9-CCD2 g exptime 215.859 propid BASS       seeing 1.48 MJD 58219.310 object 45082
   ksb_190131_114353_ooi_r_ls9.fits.fz ksb_190131_114353_ooi_r_ls9-CCD2 r exptime 139.363 propid NOTBASS    seeing 1.45 MJD 58514.489 object 
   ksb_190128_063416_ooi_g_ls9.fits.fz ksb_190128_063416_ooi_g_ls9-CCD3 g exptime 110.595 propid NOTBASS    seeing 1.49 MJD 58511.274 object 
   ksb_190209_094735_ooi_r_ls9.fits.fz ksb_190209_094735_ooi_r_ls9-CCD3 r exptime 129.274 propid NOTBASS    seeing 1.56 MJD 58523.408 object 
   k4m_170208_080820_ooi_zd_ls9.fits.fz k4m_170208_080820_ooi_zd_ls9-CCD3 z exptime 124.104 propid 2016A-0453 seeing 1.20 MJD 57792.339 object MzLS_711764_z
   k4m_171028_104055_ooi_zd_ls9.fits.fz k4m_171028_104055_ooi_zd_ls9-CCD3 z exptime 135.306 propid 2016A-0453 seeing 1.01 MJD 58054.445 object MzLS_711310_z
   k4m_160125_092739_ooi_zd_ls9.fits.fz k4m_160125_092739_ooi_zd_ls9-CCD2 z exptime 128.103 propid 2016B-0453 seeing 1.13 MJD 57412.394 object MzLS_710853_z
mosaic get_fwhm: self.fwhm = 3.89
mosaic get_fwhm: self.fwhm = 4.61
WARNING: EXPNUM None!=160125092739 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_092739_ooi_zd_ls9.fits.fz header
Faked up EXPNUM 85110144
Faked up EXPNUM 85110144
Faked up EXPNUM 85110144
90prime get_fwhm: self.fwhm = 3.29
Faked up EXPNUM 85230163
Faked up EXPNUM 85230163
Faked up EXPNUM 85230163
90prime get_fwhm: self.fwhm = 3.45
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
90prime get_fwhm: self.fwhm = 3.19
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC2793669; please check the logfile.
Finished galaxy PGC2793669 in 0.064 minutes.
Started working on galaxy PGC2793669 at Sun Feb  9 09:50:29 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.654589 68.8764535 --width 383 --height 383 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-09T09:50:29.989653
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '146.654589', '68.8764535', '--width', '383', '--height', '383', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.654589 68.8764535 --width 383 --height 383 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 146.654589 68.8764535
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 35594 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 35595 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 35596 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 35597 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 35598 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 35599 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 35600 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 35601 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 35602 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 35603 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 35604 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 35605 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 35606 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 35607 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 35608 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 35609 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 35610 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071196 MB
Starting process 35611 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072208 MB
Starting process 35612 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072208 MB
Starting process 35613 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071916 MB
Starting process 35614 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071916 MB
Starting process 35615 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071552 MB
Starting process 35616 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 35617 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 35618 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 35619 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 35620 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 35621 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 35622 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 35623 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 35624 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Starting process 35625 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071488 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-09T09:50:31.224562
Keeping 9 CCDs:
   ksb_190112_101522_ooi_g_ls9.fits.fz ksb_190112_101522_ooi_g_ls9-CCD2 g exptime 124.63 propid NOTBASS    seeing 1.48 MJD 58495.427 object 
   ksb_190128_105804_ooi_r_ls9.fits.fz ksb_190128_105804_ooi_r_ls9-CCD2 r exptime 163.914 propid NOTBASS    seeing 1.35 MJD 58511.457 object 
   ksb_180411_072612_ooi_g_ls9.fits.fz ksb_180411_072612_ooi_g_ls9-CCD2 g exptime 215.859 propid BASS       seeing 1.48 MJD 58219.310 object 45082
   ksb_190131_114353_ooi_r_ls9.fits.fz ksb_190131_114353_ooi_r_ls9-CCD2 r exptime 139.363 propid NOTBASS    seeing 1.45 MJD 58514.489 object 
   ksb_190128_063416_ooi_g_ls9.fits.fz ksb_190128_063416_ooi_g_ls9-CCD3 g exptime 110.595 propid NOTBASS    seeing 1.49 MJD 58511.274 object 
   ksb_190209_094735_ooi_r_ls9.fits.fz ksb_190209_094735_ooi_r_ls9-CCD3 r exptime 129.274 propid NOTBASS    seeing 1.56 MJD 58523.408 object 
   k4m_170208_080820_ooi_zd_ls9.fits.fz k4m_170208_080820_ooi_zd_ls9-CCD3 z exptime 124.104 propid 2016A-0453 seeing 1.20 MJD 57792.339 object MzLS_711764_z
   k4m_171028_104055_ooi_zd_ls9.fits.fz k4m_171028_104055_ooi_zd_ls9-CCD3 z exptime 135.306 propid 2016A-0453 seeing 1.01 MJD 58054.445 object MzLS_711310_z
   k4m_160125_092739_ooi_zd_ls9.fits.fz k4m_160125_092739_ooi_zd_ls9-CCD2 z exptime 128.103 propid 2016B-0453 seeing 1.13 MJD 57412.394 object MzLS_710853_z
WARNING: EXPNUM None!=160125092739 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_092739_ooi_zd_ls9.fits.fz header
mosaic get_fwhm: self.fwhm = 4.61
Faked up EXPNUM 84950057
Faked up EXPNUM 84950057
Faked up EXPNUM 84950057
90prime get_fwhm: self.fwhm = 3.27
Faked up EXPNUM 85110144
Faked up EXPNUM 85110144
Faked up EXPNUM 85110144
90prime get_fwhm: self.fwhm = 3.29
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
90prime get_fwhm: self.fwhm = 3.27
mosaic get_fwhm: self.fwhm = 3.89
Faked up EXPNUM 85110243
Faked up EXPNUM 85110243
Faked up EXPNUM 85110243
90prime get_fwhm: self.fwhm = 2.97
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
90prime get_fwhm: self.fwhm = 3.19
Faked up EXPNUM 85230163
Faked up EXPNUM 85230163
Faked up EXPNUM 85230163
90prime get_fwhm: self.fwhm = 3.45
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC2793669; please check the logfile.
Finished galaxy PGC2793669 in 0.059 minutes.
Started working on galaxy PGC2793669 at Mon Feb 10 06:12:23 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.654589 68.8764535 --width 383 --height 383 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-10T06:12:24.073758
legacypipe git version: dr9e-pre1-9-g4a51be93
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '146.654589', '68.8764535', '--width', '383', '--height', '383', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 146.654589 68.8764535 --width 383 --height 383 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/146/PGC2793669/PGC2793669-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 146.654589 68.8764535
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 11333 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.07112 MB
Starting process 11334 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071128 MB
Starting process 11335 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071128 MB
Starting process 11336 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071128 MB
Starting process 11337 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071128 MB
Starting process 11338 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071132 MB
Starting process 11339 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071136 MB
Starting process 11340 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071136 MB
Starting process 11341 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.07114 MB
Starting process 11342 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.07114 MB
Starting process 11343 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071148 MB
Starting process 11344 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071148 MB
Starting process 11345 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071152 MB
Starting process 11346 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071152 MB
Starting process 11347 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071152 MB
Starting process 11348 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071152 MB
Starting process 11349 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 141 MB, maxrss: 0.071156 MB
Starting process 11350 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.072168 MB
Starting process 11351 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.072168 MB
Starting process 11352 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.07188 MB
Starting process 11353 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071872 MB
Starting process 11354 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071508 MB
Starting process 11355 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071444 MB
Starting process 11356 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071508 MB
Starting process 11357 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071444 MB
Starting process 11358 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071444 MB
Starting process 11359 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071444 MB
Starting process 11360 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071444 MB
Starting process 11361 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071444 MB
Starting process 11362 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071444 MB
Starting process 11363 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071444 MB
Starting process 11364 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 141 MB, maxrss: 0.071444 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-10T06:12:25.327287
Keeping 9 CCDs:
   ksb_190112_101522_ooi_g_ls9.fits.fz ksb_190112_101522_ooi_g_ls9-CCD2 g exptime 124.63 propid NOTBASS    seeing 1.48 MJD 58495.427 object 
   ksb_190128_105804_ooi_r_ls9.fits.fz ksb_190128_105804_ooi_r_ls9-CCD2 r exptime 163.914 propid NOTBASS    seeing 1.35 MJD 58511.457 object 
   ksb_180411_072612_ooi_g_ls9.fits.fz ksb_180411_072612_ooi_g_ls9-CCD2 g exptime 215.859 propid BASS       seeing 1.48 MJD 58219.310 object 45082
   ksb_190131_114353_ooi_r_ls9.fits.fz ksb_190131_114353_ooi_r_ls9-CCD2 r exptime 139.363 propid NOTBASS    seeing 1.45 MJD 58514.489 object 
   ksb_190128_063416_ooi_g_ls9.fits.fz ksb_190128_063416_ooi_g_ls9-CCD3 g exptime 110.595 propid NOTBASS    seeing 1.49 MJD 58511.274 object 
   ksb_190209_094735_ooi_r_ls9.fits.fz ksb_190209_094735_ooi_r_ls9-CCD3 r exptime 129.274 propid NOTBASS    seeing 1.56 MJD 58523.408 object 
   k4m_170208_080820_ooi_zd_ls9.fits.fz k4m_170208_080820_ooi_zd_ls9-CCD3 z exptime 124.104 propid 2016A-0453 seeing 1.20 MJD 57792.339 object MzLS_711764_z
   k4m_171028_104055_ooi_zd_ls9.fits.fz k4m_171028_104055_ooi_zd_ls9-CCD3 z exptime 135.306 propid 2016A-0453 seeing 1.01 MJD 58054.445 object MzLS_711310_z
   k4m_160125_092739_ooi_zd_ls9.fits.fz k4m_160125_092739_ooi_zd_ls9-CCD2 z exptime 128.103 propid 2016B-0453 seeing 1.13 MJD 57412.394 object MzLS_710853_z
WARNING: EXPNUM None!=160125092739 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_092739_ooi_zd_ls9.fits.fz header
mosaic get_fwhm: self.fwhm = 3.89
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
Faked up EXPNUM 82190101
90prime get_fwhm: self.fwhm = 3.27
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
Faked up EXPNUM 85140227
90prime get_fwhm: self.fwhm = 3.19
Faked up EXPNUM 85230163
Faked up EXPNUM 85230163
Faked up EXPNUM 85230163
90prime get_fwhm: self.fwhm = 3.45
Faked up EXPNUM 85110243
Faked up EXPNUM 85110243
Faked up EXPNUM 85110243
90prime get_fwhm: self.fwhm = 2.97
Faked up EXPNUM 84950057
Faked up EXPNUM 84950057
Faked up EXPNUM 84950057
90prime get_fwhm: self.fwhm = 3.27
Faked up EXPNUM 85110144
Faked up EXPNUM 85110144
Faked up EXPNUM 85110144
90prime get_fwhm: self.fwhm = 3.29
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC2793669; please check the logfile.
Finished galaxy PGC2793669 in 0.076 minutes.