Started working on galaxy PGC2707508 at Sat Feb  8 08:45:38 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 143.24536799999998 67.8694247 --width 453 --height 453 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T08:45:39.446412
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '143.24536799999998', '67.8694247', '--width', '453', '--height', '453', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 143.24536799999998 67.8694247 --width 453 --height 453 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 143.24536799999998 67.8694247
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 23816 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 23817 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 23818 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 23819 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 23820 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 23821 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 23822 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 23823 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T08:45:40.807630
Keeping 12 CCDs:
   ksb_190112_101235_ooi_g_ls9.fits.fz ksb_190112_101235_ooi_g_ls9-CCD4 g exptime 126.55 propid NOTBASS    seeing 1.46 MJD 58495.425 object 
   ksb_190128_105043_ooi_r_ls9.fits.fz ksb_190128_105043_ooi_r_ls9-CCD4 r exptime 165.75 propid NOTBASS    seeing 1.10 MJD 58511.452 object 
   ksb_190131_112335_ooi_r_ls9.fits.fz ksb_190131_112335_ooi_r_ls9-CCD2 r exptime 112.96 propid NOTBASS    seeing 1.48 MJD 58514.475 object 
   ksb_180411_063614_ooi_g_ls9.fits.fz ksb_180411_063614_ooi_g_ls9-CCD2 g exptime 178.794 propid BASS       seeing 1.69 MJD 58219.275 object 44282
   ksb_190128_061521_ooi_g_ls9.fits.fz ksb_190128_061521_ooi_g_ls9-CCD3 g exptime 120.255 propid NOTBASS    seeing 1.28 MJD 58511.261 object 
   ksb_190209_091706_ooi_r_ls9.fits.fz ksb_190209_091706_ooi_r_ls9-CCD3 r exptime 141.748 propid NOTBASS    seeing 1.47 MJD 58523.387 object 
   k4m_180117_060239_ooi_zd_ls9.fits.fz k4m_180117_060239_ooi_zd_ls9-CCD1 z exptime 176.107 propid 2016A-0453 seeing 1.06 MJD 58135.252 object MzLS_709452_z
   k4m_160125_090839_ooi_zd_ls9.fits.fz k4m_160125_090839_ooi_zd_ls9-CCD1 z exptime 122.105 propid 2016B-0453 seeing 0.96 MJD 57412.381 object MzLS_709452_z
   k4m_160320_055053_ooi_zd_ls9.fits.fz k4m_160320_055053_ooi_zd_ls9-CCD2 z exptime 96.105 propid 2016A-0453 seeing 0.89 MJD 57467.244 object MzLS_708499_z
   k4m_171204_073646_ooi_zd_ls9.fits.fz k4m_171204_073646_ooi_zd_ls9-CCD4 z exptime 250.103 propid 2016A-0453 seeing 1.42 MJD 58091.317 object MzLS_708017_z
   k4m_160127_083745_ooi_zd_ls9.fits.fz k4m_160127_083745_ooi_zd_ls9-CCD4 z exptime 112.105 propid 2016A-0453 seeing 0.96 MJD 57414.360 object MzLS_708017_z
   k4m_160317_055713_ooi_zd_ls9.fits.fz k4m_160317_055713_ooi_zd_ls9-CCD1 z exptime 80.102 propid 2016A-0453 seeing 1.20 MJD 57464.248 object MzLS_708977_z
Faked up EXPNUM 85230152
Faked up EXPNUM 85230152
Faked up EXPNUM 85230152
90prime get_fwhm: self.fwhm = 3.25
Faked up EXPNUM 85110137
Faked up EXPNUM 85110137
Faked up EXPNUM 85110137
90prime get_fwhm: self.fwhm = 2.83
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
90prime get_fwhm: self.fwhm = 3.21
Faked up EXPNUM 85110241
Faked up EXPNUM 85110241
Faked up EXPNUM 85110241
90prime get_fwhm: self.fwhm = 2.43
WARNING: EXPNUM None!=160125090839 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_090839_ooi_zd_ls9.fits.fz header
mosaic get_fwhm: self.fwhm = 3.41
mosaic get_fwhm: self.fwhm = 4.07
mosaic get_fwhm: self.fwhm = 3.71
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC2707508; please check the logfile.
Finished galaxy PGC2707508 in 0.075 minutes.
Started working on galaxy PGC2707508 at Sat Feb  8 12:53:36 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 143.24536799999998 67.8694247 --width 453 --height 453 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-08T12:53:36.839997
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '143.24536799999998', '67.8694247', '--width', '453', '--height', '453', '--pixscale', '0.262', '--threads', '8', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 143.24536799999998 67.8694247 --width 453 --height 453 --pixscale 0.262 --threads 8 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 143.24536799999998 67.8694247
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 1418 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 1419 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 1420 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 1421 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 1422 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 1423 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 1424 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 1425 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071144 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-08T12:53:38.070823
Keeping 12 CCDs:
   ksb_190112_101235_ooi_g_ls9.fits.fz ksb_190112_101235_ooi_g_ls9-CCD4 g exptime 126.55 propid NOTBASS    seeing 1.46 MJD 58495.425 object 
   ksb_190128_105043_ooi_r_ls9.fits.fz ksb_190128_105043_ooi_r_ls9-CCD4 r exptime 165.75 propid NOTBASS    seeing 1.10 MJD 58511.452 object 
   ksb_190131_112335_ooi_r_ls9.fits.fz ksb_190131_112335_ooi_r_ls9-CCD2 r exptime 112.96 propid NOTBASS    seeing 1.48 MJD 58514.475 object 
   ksb_180411_063614_ooi_g_ls9.fits.fz ksb_180411_063614_ooi_g_ls9-CCD2 g exptime 178.794 propid BASS       seeing 1.69 MJD 58219.275 object 44282
   ksb_190128_061521_ooi_g_ls9.fits.fz ksb_190128_061521_ooi_g_ls9-CCD3 g exptime 120.255 propid NOTBASS    seeing 1.28 MJD 58511.261 object 
   ksb_190209_091706_ooi_r_ls9.fits.fz ksb_190209_091706_ooi_r_ls9-CCD3 r exptime 141.748 propid NOTBASS    seeing 1.47 MJD 58523.387 object 
   k4m_180117_060239_ooi_zd_ls9.fits.fz k4m_180117_060239_ooi_zd_ls9-CCD1 z exptime 176.107 propid 2016A-0453 seeing 1.06 MJD 58135.252 object MzLS_709452_z
   k4m_160125_090839_ooi_zd_ls9.fits.fz k4m_160125_090839_ooi_zd_ls9-CCD1 z exptime 122.105 propid 2016B-0453 seeing 0.96 MJD 57412.381 object MzLS_709452_z
   k4m_160320_055053_ooi_zd_ls9.fits.fz k4m_160320_055053_ooi_zd_ls9-CCD2 z exptime 96.105 propid 2016A-0453 seeing 0.89 MJD 57467.244 object MzLS_708499_z
   k4m_171204_073646_ooi_zd_ls9.fits.fz k4m_171204_073646_ooi_zd_ls9-CCD4 z exptime 250.103 propid 2016A-0453 seeing 1.42 MJD 58091.317 object MzLS_708017_z
   k4m_160127_083745_ooi_zd_ls9.fits.fz k4m_160127_083745_ooi_zd_ls9-CCD4 z exptime 112.105 propid 2016A-0453 seeing 0.96 MJD 57414.360 object MzLS_708017_z
   k4m_160317_055713_ooi_zd_ls9.fits.fz k4m_160317_055713_ooi_zd_ls9-CCD1 z exptime 80.102 propid 2016A-0453 seeing 1.20 MJD 57464.248 object MzLS_708977_z
mosaic get_fwhm: self.fwhm = 4.07
Faked up EXPNUM 85230152
Faked up EXPNUM 85230152
Faked up EXPNUM 85230152
90prime get_fwhm: self.fwhm = 3.25
Faked up EXPNUM 85110241
Faked up EXPNUM 85110241
Faked up EXPNUM 85110241
90prime get_fwhm: self.fwhm = 2.43
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
90prime get_fwhm: self.fwhm = 3.21
WARNING: EXPNUM None!=160125090839 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_090839_ooi_zd_ls9.fits.fz header
mosaic get_fwhm: self.fwhm = 3.41
Faked up EXPNUM 82190088
Faked up EXPNUM 82190088
Faked up EXPNUM 82190088
90prime get_fwhm: self.fwhm = 3.73
mosaic get_fwhm: self.fwhm = 3.71
Faked up EXPNUM 85140220
Faked up EXPNUM 85140220
Faked up EXPNUM 85140220
90prime get_fwhm: self.fwhm = 3.27
mosaic get_fwhm: self.fwhm = 5.47
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC2707508; please check the logfile.
Finished galaxy PGC2707508 in 0.066 minutes.
Started working on galaxy PGC2707508 at Sun Feb  9 08:25:28 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 143.24536799999998 67.8694247 --width 453 --height 453 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-09T08:25:29.337230
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '143.24536799999998', '67.8694247', '--width', '453', '--height', '453', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 143.24536799999998 67.8694247 --width 453 --height 453 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 143.24536799999998 67.8694247
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 29150 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 29151 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 29152 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 29153 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 29154 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 29155 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 29156 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071176 MB
Starting process 29157 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 29158 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 29159 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 29160 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 29161 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 29162 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 29163 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 29164 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 29165 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 29166 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 29167 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072204 MB
Starting process 29168 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072204 MB
Starting process 29169 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071912 MB
Starting process 29170 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071912 MB
Starting process 29171 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 29172 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 29173 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 29174 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 29175 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 29176 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 29177 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 29178 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071544 MB
Starting process 29179 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 29180 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 29181 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-09T08:25:30.607050
Keeping 12 CCDs:
   ksb_190112_101235_ooi_g_ls9.fits.fz ksb_190112_101235_ooi_g_ls9-CCD4 g exptime 126.55 propid NOTBASS    seeing 1.46 MJD 58495.425 object 
   ksb_190128_105043_ooi_r_ls9.fits.fz ksb_190128_105043_ooi_r_ls9-CCD4 r exptime 165.75 propid NOTBASS    seeing 1.10 MJD 58511.452 object 
   ksb_190131_112335_ooi_r_ls9.fits.fz ksb_190131_112335_ooi_r_ls9-CCD2 r exptime 112.96 propid NOTBASS    seeing 1.48 MJD 58514.475 object 
   ksb_180411_063614_ooi_g_ls9.fits.fz ksb_180411_063614_ooi_g_ls9-CCD2 g exptime 178.794 propid BASS       seeing 1.69 MJD 58219.275 object 44282
   ksb_190128_061521_ooi_g_ls9.fits.fz ksb_190128_061521_ooi_g_ls9-CCD3 g exptime 120.255 propid NOTBASS    seeing 1.28 MJD 58511.261 object 
   ksb_190209_091706_ooi_r_ls9.fits.fz ksb_190209_091706_ooi_r_ls9-CCD3 r exptime 141.748 propid NOTBASS    seeing 1.47 MJD 58523.387 object 
   k4m_180117_060239_ooi_zd_ls9.fits.fz k4m_180117_060239_ooi_zd_ls9-CCD1 z exptime 176.107 propid 2016A-0453 seeing 1.06 MJD 58135.252 object MzLS_709452_z
   k4m_160125_090839_ooi_zd_ls9.fits.fz k4m_160125_090839_ooi_zd_ls9-CCD1 z exptime 122.105 propid 2016B-0453 seeing 0.96 MJD 57412.381 object MzLS_709452_z
   k4m_160320_055053_ooi_zd_ls9.fits.fz k4m_160320_055053_ooi_zd_ls9-CCD2 z exptime 96.105 propid 2016A-0453 seeing 0.89 MJD 57467.244 object MzLS_708499_z
   k4m_171204_073646_ooi_zd_ls9.fits.fz k4m_171204_073646_ooi_zd_ls9-CCD4 z exptime 250.103 propid 2016A-0453 seeing 1.42 MJD 58091.317 object MzLS_708017_z
   k4m_160127_083745_ooi_zd_ls9.fits.fz k4m_160127_083745_ooi_zd_ls9-CCD4 z exptime 112.105 propid 2016A-0453 seeing 0.96 MJD 57414.360 object MzLS_708017_z
   k4m_160317_055713_ooi_zd_ls9.fits.fz k4m_160317_055713_ooi_zd_ls9-CCD1 z exptime 80.102 propid 2016A-0453 seeing 1.20 MJD 57464.248 object MzLS_708977_z
WARNING: EXPNUM None!=160125090839 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_090839_ooi_zd_ls9.fits.fz header
mosaic get_fwhm: self.fwhm = 4.07
Faked up EXPNUM 82190088
Faked up EXPNUM 82190088
Faked up EXPNUM 82190088
90prime get_fwhm: self.fwhm = 3.73
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
90prime get_fwhm: self.fwhm = 3.21
Faked up EXPNUM 85110241
Faked up EXPNUM 85110241
Faked up EXPNUM 85110241
90prime get_fwhm: self.fwhm = 2.43
Faked up EXPNUM 85110137
Faked up EXPNUM 85110137
Faked up EXPNUM 85110137
90prime get_fwhm: self.fwhm = 2.83
Faked up EXPNUM 85230152
Faked up EXPNUM 85230152
Faked up EXPNUM 85230152
90prime get_fwhm: self.fwhm = 3.25
Faked up EXPNUM 85140220
Faked up EXPNUM 85140220
Faked up EXPNUM 85140220
90prime get_fwhm: self.fwhm = 3.27
mosaic get_fwhm: self.fwhm = 3.41
mosaic get_fwhm: self.fwhm = 3.71
mosaic get_fwhm: self.fwhm = 5.47
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC2707508; please check the logfile.
Finished galaxy PGC2707508 in 0.058 minutes.
Started working on galaxy PGC2707508 at Sun Feb  9 09:58:13 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 143.24536799999998 67.8694247 --width 453 --height 453 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-09T09:58:13.889003
legacypipe git version: dr9e-pre1-7-g54c974a3
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '143.24536799999998', '67.8694247', '--width', '453', '--height', '453', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 143.24536799999998 67.8694247 --width 453 --height 453 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 143.24536799999998 67.8694247
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 38086 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071164 MB
Starting process 38087 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 38088 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071168 MB
Starting process 38089 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 38090 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071172 MB
Starting process 38091 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.07118 MB
Starting process 38092 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 38093 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 38094 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071184 MB
Starting process 38095 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071188 MB
Starting process 38096 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 38097 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 38098 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 38099 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 38100 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071192 MB
Starting process 38101 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071196 MB
Starting process 38102 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071196 MB
Starting process 38103 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072208 MB
Starting process 38104 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.072208 MB
Starting process 38105 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071916 MB
Starting process 38106 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071908 MB
Starting process 38107 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071552 MB
Starting process 38108 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 38109 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 38110 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 38111 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 38112 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 38113 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071484 MB
Starting process 38114 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 38115 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 38116 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071548 MB
Starting process 38117 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071552 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-09T09:58:15.174755
Keeping 12 CCDs:
   ksb_190112_101235_ooi_g_ls9.fits.fz ksb_190112_101235_ooi_g_ls9-CCD4 g exptime 126.55 propid NOTBASS    seeing 1.46 MJD 58495.425 object 
   ksb_190128_105043_ooi_r_ls9.fits.fz ksb_190128_105043_ooi_r_ls9-CCD4 r exptime 165.75 propid NOTBASS    seeing 1.10 MJD 58511.452 object 
   ksb_190131_112335_ooi_r_ls9.fits.fz ksb_190131_112335_ooi_r_ls9-CCD2 r exptime 112.96 propid NOTBASS    seeing 1.48 MJD 58514.475 object 
   ksb_180411_063614_ooi_g_ls9.fits.fz ksb_180411_063614_ooi_g_ls9-CCD2 g exptime 178.794 propid BASS       seeing 1.69 MJD 58219.275 object 44282
   ksb_190128_061521_ooi_g_ls9.fits.fz ksb_190128_061521_ooi_g_ls9-CCD3 g exptime 120.255 propid NOTBASS    seeing 1.28 MJD 58511.261 object 
   ksb_190209_091706_ooi_r_ls9.fits.fz ksb_190209_091706_ooi_r_ls9-CCD3 r exptime 141.748 propid NOTBASS    seeing 1.47 MJD 58523.387 object 
   k4m_180117_060239_ooi_zd_ls9.fits.fz k4m_180117_060239_ooi_zd_ls9-CCD1 z exptime 176.107 propid 2016A-0453 seeing 1.06 MJD 58135.252 object MzLS_709452_z
   k4m_160125_090839_ooi_zd_ls9.fits.fz k4m_160125_090839_ooi_zd_ls9-CCD1 z exptime 122.105 propid 2016B-0453 seeing 0.96 MJD 57412.381 object MzLS_709452_z
   k4m_160320_055053_ooi_zd_ls9.fits.fz k4m_160320_055053_ooi_zd_ls9-CCD2 z exptime 96.105 propid 2016A-0453 seeing 0.89 MJD 57467.244 object MzLS_708499_z
   k4m_171204_073646_ooi_zd_ls9.fits.fz k4m_171204_073646_ooi_zd_ls9-CCD4 z exptime 250.103 propid 2016A-0453 seeing 1.42 MJD 58091.317 object MzLS_708017_z
   k4m_160127_083745_ooi_zd_ls9.fits.fz k4m_160127_083745_ooi_zd_ls9-CCD4 z exptime 112.105 propid 2016A-0453 seeing 0.96 MJD 57414.360 object MzLS_708017_z
   k4m_160317_055713_ooi_zd_ls9.fits.fz k4m_160317_055713_ooi_zd_ls9-CCD1 z exptime 80.102 propid 2016A-0453 seeing 1.20 MJD 57464.248 object MzLS_708977_z
WARNING: EXPNUM None!=160125090839 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_090839_ooi_zd_ls9.fits.fz header
mosaic get_fwhm: self.fwhm = 4.07
mosaic get_fwhm: self.fwhm = 3.41
Faked up EXPNUM 85110137
Faked up EXPNUM 85110137
Faked up EXPNUM 85110137
90prime get_fwhm: self.fwhm = 2.83
Faked up EXPNUM 85110241
Faked up EXPNUM 85110241
Faked up EXPNUM 85110241
90prime get_fwhm: self.fwhm = 2.43
Faked up EXPNUM 85230152
Faked up EXPNUM 85230152
Faked up EXPNUM 85230152
90prime get_fwhm: self.fwhm = 3.25
mosaic get_fwhm: self.fwhm = 3.71
Faked up EXPNUM 85140220
Faked up EXPNUM 85140220
Faked up EXPNUM 85140220
90prime get_fwhm: self.fwhm = 3.27
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
90prime get_fwhm: self.fwhm = 3.21
mosaic get_fwhm: self.fwhm = 5.47
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC2707508; please check the logfile.
Finished galaxy PGC2707508 in 0.058 minutes.
Started working on galaxy PGC2707508 at Mon Feb 10 06:13:50 2020
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 143.24536799999998 67.8694247 --width 453 --height 453 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-%%(stage)s.p --unwise-coadds 

runbrick.py starting at 2020-02-10T06:13:50.755974
legacypipe git version: dr9e-pre1-9-g4a51be93
Command-line args: ['/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py', '--radec', '143.24536799999998', '67.8694247', '--width', '453', '--height', '453', '--pixscale', '0.262', '--threads', '32', '--outdir', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508', '--survey-dir', '/global/cscratch1/sd/desimpp/dr9e', '--run', '90prime-mosaic', '--largegalaxy-preburner', '--write-stage', 'tims', '--write-stage', 'srcs', '--skip-calibs', '--checkpoint', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-checkpoint.p', '--pickle', '/global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-%%(stage)s.p', '--unwise-coadds']
python /global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py --radec 143.24536799999998 67.8694247 --width 453 --height 453 --pixscale 0.262 --threads 32 --outdir /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508 --survey-dir /global/cscratch1/sd/desimpp/dr9e --run 90prime-mosaic --largegalaxy-preburner --write-stage tims --write-stage srcs --skip-calibs --checkpoint /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-checkpoint.p --pickle /global/projecta/projectdirs/desi/users/ioannis/LSLGA-data-dr9e/143/PGC2707508/PGC2707508-runbrick-%%(stage)s.p --unwise-coadds

WARNING: no WISE sky background maps in /global/cscratch1/sd/desimpp/dr9e/calib/wise/modelsky
Parsed RA,Dec 143.24536799999998 67.8694247
WARNING: CacheMissingWarning: Not clearing data cache - cache inaccessible due to OSError: [Errno 30] Read-only file system: '/home/legacyhalos/.astropy/cache/download/py3/lock' [astropy.utils.data]
Starting process 12583 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 12584 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 12585 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 12586 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 12587 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071128 MB
Starting process 12588 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071132 MB
Starting process 12589 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071136 MB
Starting process 12590 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 12591 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 12592 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.07114 MB
Starting process 12593 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071144 MB
Starting process 12594 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 12595 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 12596 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 12597 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071148 MB
Starting process 12598 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 69 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 12599 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 71 MB, VmData: 140 MB, maxrss: 0.071152 MB
Starting process 12600 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072168 MB
Starting process 12601 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.072168 MB
Starting process 12602 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071876 MB
Starting process 12603 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071876 MB
Starting process 12604 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071156 MB
Starting process 12605 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071512 MB
Starting process 12606 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071516 MB
Starting process 12607 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071512 MB
Starting process 12608 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071156 MB
Starting process 12609 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071512 MB
Starting process 12610 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071512 MB
Starting process 12611 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071512 MB
Starting process 12612 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071512 MB
Starting process 12613 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071512 MB
Starting process 12614 Wall: -0.00 s, CPU: -0.00 s, VmPeak: 665 MB, VmSize: 665 MB, VmRSS: 70 MB, VmData: 140 MB, maxrss: 0.071512 MB
Runstage writecat
Runstage wise_forced
Runstage coadds
Runstage fitblobs
Runstage srcs
Runstage largegalaxies
Runstage halos
Runstage outliers
Runstage refs
Runstage tims
Running stage tims
Running stage tims at 2020-02-10T06:13:52.035345
Keeping 12 CCDs:
   ksb_190112_101235_ooi_g_ls9.fits.fz ksb_190112_101235_ooi_g_ls9-CCD4 g exptime 126.55 propid NOTBASS    seeing 1.46 MJD 58495.425 object 
   ksb_190128_105043_ooi_r_ls9.fits.fz ksb_190128_105043_ooi_r_ls9-CCD4 r exptime 165.75 propid NOTBASS    seeing 1.10 MJD 58511.452 object 
   ksb_190131_112335_ooi_r_ls9.fits.fz ksb_190131_112335_ooi_r_ls9-CCD2 r exptime 112.96 propid NOTBASS    seeing 1.48 MJD 58514.475 object 
   ksb_180411_063614_ooi_g_ls9.fits.fz ksb_180411_063614_ooi_g_ls9-CCD2 g exptime 178.794 propid BASS       seeing 1.69 MJD 58219.275 object 44282
   ksb_190128_061521_ooi_g_ls9.fits.fz ksb_190128_061521_ooi_g_ls9-CCD3 g exptime 120.255 propid NOTBASS    seeing 1.28 MJD 58511.261 object 
   ksb_190209_091706_ooi_r_ls9.fits.fz ksb_190209_091706_ooi_r_ls9-CCD3 r exptime 141.748 propid NOTBASS    seeing 1.47 MJD 58523.387 object 
   k4m_180117_060239_ooi_zd_ls9.fits.fz k4m_180117_060239_ooi_zd_ls9-CCD1 z exptime 176.107 propid 2016A-0453 seeing 1.06 MJD 58135.252 object MzLS_709452_z
   k4m_160125_090839_ooi_zd_ls9.fits.fz k4m_160125_090839_ooi_zd_ls9-CCD1 z exptime 122.105 propid 2016B-0453 seeing 0.96 MJD 57412.381 object MzLS_709452_z
   k4m_160320_055053_ooi_zd_ls9.fits.fz k4m_160320_055053_ooi_zd_ls9-CCD2 z exptime 96.105 propid 2016A-0453 seeing 0.89 MJD 57467.244 object MzLS_708499_z
   k4m_171204_073646_ooi_zd_ls9.fits.fz k4m_171204_073646_ooi_zd_ls9-CCD4 z exptime 250.103 propid 2016A-0453 seeing 1.42 MJD 58091.317 object MzLS_708017_z
   k4m_160127_083745_ooi_zd_ls9.fits.fz k4m_160127_083745_ooi_zd_ls9-CCD4 z exptime 112.105 propid 2016A-0453 seeing 0.96 MJD 57414.360 object MzLS_708017_z
   k4m_160317_055713_ooi_zd_ls9.fits.fz k4m_160317_055713_ooi_zd_ls9-CCD1 z exptime 80.102 propid 2016A-0453 seeing 1.20 MJD 57464.248 object MzLS_708977_z
WARNING: EXPNUM None!=160125090839 in /global/cscratch1/sd/desimpp/dr9e/images/mosaic/CP/V4.3/CP20160124/k4m_160125_090839_ooi_zd_ls9.fits.fz header
Faked up EXPNUM 85110241
Faked up EXPNUM 85110241
Faked up EXPNUM 85110241
90prime get_fwhm: self.fwhm = 2.43
mosaic get_fwhm: self.fwhm = 4.07
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
Faked up EXPNUM 84950056
90prime get_fwhm: self.fwhm = 3.21
Faked up EXPNUM 85110137
Faked up EXPNUM 85110137
Faked up EXPNUM 85110137
90prime get_fwhm: self.fwhm = 2.83
Faked up EXPNUM 85230152
Faked up EXPNUM 85230152
Faked up EXPNUM 85230152
90prime get_fwhm: self.fwhm = 3.25
Faked up EXPNUM 85140220
Faked up EXPNUM 85140220
Faked up EXPNUM 85140220
90prime get_fwhm: self.fwhm = 3.27
Faked up EXPNUM 82190088
Faked up EXPNUM 82190088
Faked up EXPNUM 82190088
90prime get_fwhm: self.fwhm = 3.73
mosaic get_fwhm: self.fwhm = 4.63
mosaic get_fwhm: self.fwhm = 3.71
mosaic get_fwhm: self.fwhm = 5.47
Traceback (most recent call last):
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3605, in <module>
    sys.exit(main())
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3575, in main
    run_brick(opt.brick, survey, **kwargs)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3161, in run_brick
    initial_args=initargs, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 113, in runstage
    write=write, initial_args=initial_args, **kwargs)
  [Previous line repeated 6 more times]
  File "/usr/local/lib/python/astrometry/util/stages.py", line 124, in runstage
    R = stagefunc(stage, **Px)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 3150, in mystagefunc
    R = stagefunc(stage, mp=mp, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 40, in __call__
    rtn = super(CallGlobalTime, self).__call__(stage, **kwargs)
  File "/usr/local/lib/python/astrometry/util/stages.py", line 32, in __call__
    return func(*self.args, **kwa)
  File "/global/homes/i/ioannis/repos/git/legacypipe/py/legacypipe/runbrick.py", line 271, in stage_tims
    tims = list(mp.map(read_one_tim, args))
  File "/usr/local/lib/python/astrometry/util/multiproc.py", line 89, in map
    return self.pool.map(f, args, cs)
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 268, in map
    return self._map_async(func, iterable, mapstar, chunksize).get()
  File "/opt/conda/lib/python3.7/multiprocessing/pool.py", line 657, in get
    raise self._value
AssertionError
Something went wrong; please check the logfile.
ERROR: galaxy PGC2707508; please check the logfile.
Finished galaxy PGC2707508 in 0.066 minutes.