Started working on galaxy PGC027061 at Sat Feb  8 08:14:55 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 142.94837549999997 69.03097220000001 --width 445 --height 445 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T08:14:56.557718
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '142.94837549999997', '69.03097220000001', '--width', '445', '--height', '445', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 142.94837549999997 69.03097220000001 --width 445 --height 445 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 142.94837549999997 69.03097220000001
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 24001 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 24002 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 24003 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 24004 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 24005 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 24006 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 24007 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 24008 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T08:14:57.893422
Keeping 10 CCDs:
   ksb_190127_065527_ooi_g_ls9.fits.fz ksb_190127_065527_ooi_g_ls9-CCD1 g exptime 212.212 propid NOTBASS    seeing 2.52 MJD 58510.289 object 
   ksb_190209_090746_ooi_r_ls9.fits.fz ksb_190209_090746_ooi_r_ls9-CCD1 r exptime 142.272 propid NOTBASS    seeing 1.62 MJD 58523.380 object 
   ksb_180411_065359_ooi_g_ls9.fits.fz ksb_180411_065359_ooi_g_ls9-CCD4 g exptime 220.374 propid BASS       seeing 1.61 MJD 58219.287 object 45072
   ksb_190131_112908_ooi_r_ls9.fits.fz ksb_190131_112908_ooi_r_ls9-CCD4 r exptime 117.431 propid NOTBASS    seeing 1.42 MJD 58514.479 object 
   ksb_190128_103857_ooi_r_ls9.fits.fz ksb_190128_103857_ooi_r_ls9-CCD1 r exptime 166.362 propid NOTBASS    seeing 1.40 MJD 58511.444 object 
   ksb_190111_125408_ooi_g_ls9.fits.fz ksb_190111_125408_ooi_g_ls9-CCD1 g exptime 185.5 propid NOTBASS    seeing 2.11 MJD 58494.538 object 
   k4m_160125_090527_ooi_zd_ls9.fits.fz k4m_160125_090527_ooi_zd_ls9-CCD3 z exptime 155.104 propid 2016B-0453 seeing 0.96 MJD 57412.379 object MzLS_712212_z
   k4m_160319_055039_ooi_zd_ls9.fits.fz k4m_160319_055039_ooi_zd_ls9-CCD2 z exptime 99.104 propid 2016A-0453 seeing 1.14 MJD 57466.244 object MzLS_711308_z
   k4m_160125_090250_ooi_zd_ls9.fits.fz k4m_160125_090250_ooi_zd_ls9-CCD4 z exptime 120.104 propid 2016B-0453 seeing 1.06 MJD 57412.377 object MzLS_710850_z
   k4m_160413_041750_ooi_zd_ls9.fits.fz k4m_160413_041750_ooi_zd_ls9-CCD1 z exptime 153.102 propid 2016A-0453 seeing 1.17 MJD 57491.179 object MzLS_711762_z
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
90prime get_fwhm: self.fwhm = 3.57
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
90prime get_fwhm: self.fwhm = 5.55
mosaic get_fwhm: self.fwhm = 4.41
Faked up EXPNUM 82190093
Faked up EXPNUM 82190093
Faked up EXPNUM 82190093
90prime get_fwhm: self.fwhm = 3.55
Faked up EXPNUM 85140222
Faked up EXPNUM 85140222
Faked up EXPNUM 85140222
90prime get_fwhm: self.fwhm = 3.13
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC027061; please check the logfile.
Finished galaxy PGC027061 in 0.065 minutes.
Started working on galaxy PGC027061 at Sat Feb  8 08:31:54 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 142.94837549999997 69.03097220000001 --width 445 --height 445 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T08:31:54.883135
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '142.94837549999997', '69.03097220000001', '--width', '445', '--height', '445', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 142.94837549999997 69.03097220000001 --width 445 --height 445 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 142.94837549999997 69.03097220000001
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 34546 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 34547 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 34548 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 34549 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 34550 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 34551 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 34552 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 34553 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T08:31:56.142769
Keeping 10 CCDs:
   ksb_190127_065527_ooi_g_ls9.fits.fz ksb_190127_065527_ooi_g_ls9-CCD1 g exptime 212.212 propid NOTBASS    seeing 2.52 MJD 58510.289 object 
   ksb_190209_090746_ooi_r_ls9.fits.fz ksb_190209_090746_ooi_r_ls9-CCD1 r exptime 142.272 propid NOTBASS    seeing 1.62 MJD 58523.380 object 
   ksb_180411_065359_ooi_g_ls9.fits.fz ksb_180411_065359_ooi_g_ls9-CCD4 g exptime 220.374 propid BASS       seeing 1.61 MJD 58219.287 object 45072
   ksb_190131_112908_ooi_r_ls9.fits.fz ksb_190131_112908_ooi_r_ls9-CCD4 r exptime 117.431 propid NOTBASS    seeing 1.42 MJD 58514.479 object 
   ksb_190128_103857_ooi_r_ls9.fits.fz ksb_190128_103857_ooi_r_ls9-CCD1 r exptime 166.362 propid NOTBASS    seeing 1.40 MJD 58511.444 object 
   ksb_190111_125408_ooi_g_ls9.fits.fz ksb_190111_125408_ooi_g_ls9-CCD1 g exptime 185.5 propid NOTBASS    seeing 2.11 MJD 58494.538 object 
   k4m_160125_090527_ooi_zd_ls9.fits.fz k4m_160125_090527_ooi_zd_ls9-CCD3 z exptime 155.104 propid 2016B-0453 seeing 0.96 MJD 57412.379 object MzLS_712212_z
   k4m_160319_055039_ooi_zd_ls9.fits.fz k4m_160319_055039_ooi_zd_ls9-CCD2 z exptime 99.104 propid 2016A-0453 seeing 1.14 MJD 57466.244 object MzLS_711308_z
   k4m_160125_090250_ooi_zd_ls9.fits.fz k4m_160125_090250_ooi_zd_ls9-CCD4 z exptime 120.104 propid 2016B-0453 seeing 1.06 MJD 57412.377 object MzLS_710850_z
   k4m_160413_041750_ooi_zd_ls9.fits.fz k4m_160413_041750_ooi_zd_ls9-CCD1 z exptime 153.102 propid 2016A-0453 seeing 1.17 MJD 57491.179 object MzLS_711762_z
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
90prime get_fwhm: self.fwhm = 3.57
mosaic get_fwhm: self.fwhm = 4.41
Faked up EXPNUM 82190093
Faked up EXPNUM 82190093
Faked up EXPNUM 82190093
90prime get_fwhm: self.fwhm = 3.55
Faked up EXPNUM 85140222
Faked up EXPNUM 85140222
Faked up EXPNUM 85140222
90prime get_fwhm: self.fwhm = 3.13
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
90prime get_fwhm: self.fwhm = 5.55
Faked up EXPNUM 85110238
Faked up EXPNUM 85110238
Faked up EXPNUM 85110238
90prime get_fwhm: self.fwhm = 3.09
Faked up EXPNUM 84940096
Faked up EXPNUM 84940096
Faked up EXPNUM 84940096
90prime get_fwhm: self.fwhm = 4.65
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC027061; please check the logfile.
Finished galaxy PGC027061 in 0.061 minutes.
Started working on galaxy PGC027061 at Sat Feb  8 12:56:22 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 142.94837549999997 69.03097220000001 --width 445 --height 445 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T12:56:22.570120
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '142.94837549999997', '69.03097220000001', '--width', '445', '--height', '445', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 142.94837549999997 69.03097220000001 --width 445 --height 445 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 142.94837549999997 69.03097220000001
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 12336 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 12337 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 12338 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 12339 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 12340 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 12341 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 12342 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 12343 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T12:56:23.866071
Keeping 10 CCDs:
   ksb_190127_065527_ooi_g_ls9.fits.fz ksb_190127_065527_ooi_g_ls9-CCD1 g exptime 212.212 propid NOTBASS    seeing 2.52 MJD 58510.289 object 
   ksb_190209_090746_ooi_r_ls9.fits.fz ksb_190209_090746_ooi_r_ls9-CCD1 r exptime 142.272 propid NOTBASS    seeing 1.62 MJD 58523.380 object 
   ksb_180411_065359_ooi_g_ls9.fits.fz ksb_180411_065359_ooi_g_ls9-CCD4 g exptime 220.374 propid BASS       seeing 1.61 MJD 58219.287 object 45072
   ksb_190131_112908_ooi_r_ls9.fits.fz ksb_190131_112908_ooi_r_ls9-CCD4 r exptime 117.431 propid NOTBASS    seeing 1.42 MJD 58514.479 object 
   ksb_190128_103857_ooi_r_ls9.fits.fz ksb_190128_103857_ooi_r_ls9-CCD1 r exptime 166.362 propid NOTBASS    seeing 1.40 MJD 58511.444 object 
   ksb_190111_125408_ooi_g_ls9.fits.fz ksb_190111_125408_ooi_g_ls9-CCD1 g exptime 185.5 propid NOTBASS    seeing 2.11 MJD 58494.538 object 
   k4m_160125_090527_ooi_zd_ls9.fits.fz k4m_160125_090527_ooi_zd_ls9-CCD3 z exptime 155.104 propid 2016B-0453 seeing 0.96 MJD 57412.379 object MzLS_712212_z
   k4m_160319_055039_ooi_zd_ls9.fits.fz k4m_160319_055039_ooi_zd_ls9-CCD2 z exptime 99.104 propid 2016A-0453 seeing 1.14 MJD 57466.244 object MzLS_711308_z
   k4m_160125_090250_ooi_zd_ls9.fits.fz k4m_160125_090250_ooi_zd_ls9-CCD4 z exptime 120.104 propid 2016B-0453 seeing 1.06 MJD 57412.377 object MzLS_710850_z
   k4m_160413_041750_ooi_zd_ls9.fits.fz k4m_160413_041750_ooi_zd_ls9-CCD1 z exptime 153.102 propid 2016A-0453 seeing 1.17 MJD 57491.179 object MzLS_711762_z
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
90prime get_fwhm: self.fwhm = 5.55
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
90prime get_fwhm: self.fwhm = 3.57
Faked up EXPNUM 85140222
Faked up EXPNUM 85140222
Faked up EXPNUM 85140222
90prime get_fwhm: self.fwhm = 3.13
Faked up EXPNUM 82190093
Faked up EXPNUM 82190093
Faked up EXPNUM 82190093
90prime get_fwhm: self.fwhm = 3.55
Faked up EXPNUM 84940096
Faked up EXPNUM 84940096
Faked up EXPNUM 84940096
90prime get_fwhm: self.fwhm = 4.65
Faked up EXPNUM 85110238
Faked up EXPNUM 85110238
Faked up EXPNUM 85110238
90prime get_fwhm: self.fwhm = 3.09
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC027061; please check the logfile.
Finished galaxy PGC027061 in 0.056 minutes.
Started working on galaxy PGC027061 at Sun Feb  9 09:57:52 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 142.94837549999997 69.03097220000001 --width 445 --height 445 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-09T09:57:53.452786
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '142.94837549999997', '69.03097220000001', '--width', '445', '--height', '445', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 142.94837549999997 69.03097220000001 --width 445 --height 445 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 142.94837549999997 69.03097220000001
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 37812 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 37813 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 37814 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 37815 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 37816 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 37817 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 37818 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 37819 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 37820 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 37821 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 37822 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 37823 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 37824 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 37825 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 37826 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 37827 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 37828 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 37829 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072204 MB
Starting process 37830 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072204 MB
Starting process 37831 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071912 MB
Starting process 37832 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071912 MB
Starting process 37833 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 37834 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07154 MB
Starting process 37835 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07154 MB
Starting process 37836 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07154 MB
Starting process 37837 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07154 MB
Starting process 37838 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07154 MB
Starting process 37839 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071476 MB
Starting process 37840 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071476 MB
Starting process 37841 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071476 MB
Starting process 37842 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07154 MB
Starting process 37843 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07148 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-09T09:57:54.745616
Keeping 10 CCDs:
   ksb_190127_065527_ooi_g_ls9.fits.fz ksb_190127_065527_ooi_g_ls9-CCD1 g exptime 212.212 propid NOTBASS    seeing 2.52 MJD 58510.289 object 
   ksb_190209_090746_ooi_r_ls9.fits.fz ksb_190209_090746_ooi_r_ls9-CCD1 r exptime 142.272 propid NOTBASS    seeing 1.62 MJD 58523.380 object 
   ksb_180411_065359_ooi_g_ls9.fits.fz ksb_180411_065359_ooi_g_ls9-CCD4 g exptime 220.374 propid BASS       seeing 1.61 MJD 58219.287 object 45072
   ksb_190131_112908_ooi_r_ls9.fits.fz ksb_190131_112908_ooi_r_ls9-CCD4 r exptime 117.431 propid NOTBASS    seeing 1.42 MJD 58514.479 object 
   ksb_190128_103857_ooi_r_ls9.fits.fz ksb_190128_103857_ooi_r_ls9-CCD1 r exptime 166.362 propid NOTBASS    seeing 1.40 MJD 58511.444 object 
   ksb_190111_125408_ooi_g_ls9.fits.fz ksb_190111_125408_ooi_g_ls9-CCD1 g exptime 185.5 propid NOTBASS    seeing 2.11 MJD 58494.538 object 
   k4m_160125_090527_ooi_zd_ls9.fits.fz k4m_160125_090527_ooi_zd_ls9-CCD3 z exptime 155.104 propid 2016B-0453 seeing 0.96 MJD 57412.379 object MzLS_712212_z
   k4m_160319_055039_ooi_zd_ls9.fits.fz k4m_160319_055039_ooi_zd_ls9-CCD2 z exptime 99.104 propid 2016A-0453 seeing 1.14 MJD 57466.244 object MzLS_711308_z
   k4m_160125_090250_ooi_zd_ls9.fits.fz k4m_160125_090250_ooi_zd_ls9-CCD4 z exptime 120.104 propid 2016B-0453 seeing 1.06 MJD 57412.377 object MzLS_710850_z
   k4m_160413_041750_ooi_zd_ls9.fits.fz k4m_160413_041750_ooi_zd_ls9-CCD1 z exptime 153.102 propid 2016A-0453 seeing 1.17 MJD 57491.179 object MzLS_711762_z
WARNING: EXPNUM None!=160125090527 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_090527_ooi_zd_ls9.fits.fz header
WARNING: EXPNUM None!=160125090250 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_090250_ooi_zd_ls9.fits.fz header
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
90prime get_fwhm: self.fwhm = 5.55
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
90prime get_fwhm: self.fwhm = 3.57
mosaic get_fwhm: self.fwhm = 4.41
Faked up EXPNUM 85110238
Faked up EXPNUM 85110238
Faked up EXPNUM 85110238
90prime get_fwhm: self.fwhm = 3.09
Faked up EXPNUM 85140222
Faked up EXPNUM 85140222
Faked up EXPNUM 85140222
90prime get_fwhm: self.fwhm = 3.13
Faked up EXPNUM 82190093
Faked up EXPNUM 82190093
Faked up EXPNUM 82190093
90prime get_fwhm: self.fwhm = 3.55
Faked up EXPNUM 84940096
Faked up EXPNUM 84940096
Faked up EXPNUM 84940096
90prime get_fwhm: self.fwhm = 4.65
mosaic get_fwhm: self.fwhm = 4.49
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC027061; please check the logfile.
Finished galaxy PGC027061 in 0.060 minutes.
Started working on galaxy PGC027061 at Mon Feb 10 06:13:29 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 142.94837549999997 69.03097220000001 --width 445 --height 445 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-10T06:13:29.633863
legacypipe git version: dr9e-pre1-9-g4a51be93
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '142.94837549999997', '69.03097220000001', '--width', '445', '--height', '445', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 142.94837549999997 69.03097220000001 --width 445 --height 445 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/142/PGC027061/PGC027061-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 142.94837549999997 69.03097220000001
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 12309 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071124 MB
Starting process 12310 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071124 MB
Starting process 12311 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071124 MB
Starting process 12312 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071124 MB
Starting process 12313 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071124 MB
Starting process 12314 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 12315 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 12316 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 12317 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 12318 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 12319 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 12320 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 12321 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 12322 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 12323 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 12324 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 12325 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 12326 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07216 MB
Starting process 12327 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07216 MB
Starting process 12328 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072096 MB
Starting process 12329 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072092 MB
Starting process 12330 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071508 MB
Starting process 12331 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB
Starting process 12332 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB
Starting process 12333 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB
Starting process 12334 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB
Starting process 12335 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB
Starting process 12336 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071504 MB
Starting process 12337 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071444 MB
Starting process 12338 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071444 MB
Starting process 12339 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071508 MB
Starting process 12340 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071508 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-10T06:13:30.886958
Keeping 10 CCDs:
   ksb_190127_065527_ooi_g_ls9.fits.fz ksb_190127_065527_ooi_g_ls9-CCD1 g exptime 212.212 propid NOTBASS    seeing 2.52 MJD 58510.289 object 
   ksb_190209_090746_ooi_r_ls9.fits.fz ksb_190209_090746_ooi_r_ls9-CCD1 r exptime 142.272 propid NOTBASS    seeing 1.62 MJD 58523.380 object 
   ksb_180411_065359_ooi_g_ls9.fits.fz ksb_180411_065359_ooi_g_ls9-CCD4 g exptime 220.374 propid BASS       seeing 1.61 MJD 58219.287 object 45072
   ksb_190131_112908_ooi_r_ls9.fits.fz ksb_190131_112908_ooi_r_ls9-CCD4 r exptime 117.431 propid NOTBASS    seeing 1.42 MJD 58514.479 object 
   ksb_190128_103857_ooi_r_ls9.fits.fz ksb_190128_103857_ooi_r_ls9-CCD1 r exptime 166.362 propid NOTBASS    seeing 1.40 MJD 58511.444 object 
   ksb_190111_125408_ooi_g_ls9.fits.fz ksb_190111_125408_ooi_g_ls9-CCD1 g exptime 185.5 propid NOTBASS    seeing 2.11 MJD 58494.538 object 
   k4m_160125_090527_ooi_zd_ls9.fits.fz k4m_160125_090527_ooi_zd_ls9-CCD3 z exptime 155.104 propid 2016B-0453 seeing 0.96 MJD 57412.379 object MzLS_712212_z
   k4m_160319_055039_ooi_zd_ls9.fits.fz k4m_160319_055039_ooi_zd_ls9-CCD2 z exptime 99.104 propid 2016A-0453 seeing 1.14 MJD 57466.244 object MzLS_711308_z
   k4m_160125_090250_ooi_zd_ls9.fits.fz k4m_160125_090250_ooi_zd_ls9-CCD4 z exptime 120.104 propid 2016B-0453 seeing 1.06 MJD 57412.377 object MzLS_710850_z
   k4m_160413_041750_ooi_zd_ls9.fits.fz k4m_160413_041750_ooi_zd_ls9-CCD1 z exptime 153.102 propid 2016A-0453 seeing 1.17 MJD 57491.179 object MzLS_711762_z
WARNING: EXPNUM None!=160125090527 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_090527_ooi_zd_ls9.fits.fz header
WARNING: EXPNUM None!=160125090250 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_090250_ooi_zd_ls9.fits.fz header
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
Faked up EXPNUM 85100122
90prime get_fwhm: self.fwhm = 5.55
mosaic get_fwhm: self.fwhm = 4.49
Faked up EXPNUM 85110238
Faked up EXPNUM 85110238
Faked up EXPNUM 85110238
90prime get_fwhm: self.fwhm = 3.09
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
Faked up EXPNUM 85230149
90prime get_fwhm: self.fwhm = 3.57
Faked up EXPNUM 85140222
Faked up EXPNUM 85140222
Faked up EXPNUM 85140222
90prime get_fwhm: self.fwhm = 3.13
Faked up EXPNUM 84940096
Faked up EXPNUM 84940096
Faked up EXPNUM 84940096
90prime get_fwhm: self.fwhm = 4.65
Faked up EXPNUM 82190093
Faked up EXPNUM 82190093
Faked up EXPNUM 82190093
90prime get_fwhm: self.fwhm = 3.55
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC027061; please check the logfile.
Finished galaxy PGC027061 in 0.064 minutes.