2017-06-01 19:58:39,224 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 19:58:26 2017-06-01 19:58:39,225 INFO repeater task os150nb_lastensq starting 3.2 seconds late 2017-06-01 19:58:40,466 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 19:58:40,466 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 19:58:26 2017-06-01 20:00:50,758 INFO repeater task os150bb_10pings starting 5.0 seconds late 2017-06-01 20:00:50,928 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 20:00:50,929 INFO repeater task os150nb_10pings starting 5.1 seconds late 2017-06-01 20:00:51,020 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 20:03:26,329 INFO repeater task posmv_hcorrstats starting 0.5 seconds late 2017-06-01 20:03:27,271 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 20:03:27,271 INFO repeater task posmv_plotposmv starting 1.5 seconds late 2017-06-01 20:03:27,887 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 20:03:37,902 INFO repeater task os150bb_lastensq starting 1.9 seconds late 2017-06-01 20:03:39,602 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 20:03:39,602 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 20:03:26 2017-06-01 20:03:39,603 INFO repeater task os150nb_lastensq starting 3.6 seconds late 2017-06-01 20:03:40,751 INFO repeater task os150nb_lastensq ended after 1.1 seconds 2017-06-01 20:03:40,752 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 20:03:26 2017-06-01 20:06:06,028 INFO repeater task os150bb_10pings starting 0.2 seconds late 2017-06-01 20:06:06,183 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 20:06:06,184 INFO repeater task os150nb_10pings starting 0.4 seconds late 2017-06-01 20:06:06,275 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 20:08:26,572 INFO repeater task os150bb_quick starting 0.8 seconds late 2017-06-01 20:08:29,783 INFO repeater task os150bb_quick ended after 3.2 seconds 2017-06-01 20:08:29,783 INFO repeater task os150bb_3dayplots starting 4.0 seconds late 2017-06-01 20:08:36,035 INFO repeater task os150bb_3dayplots ended after 6.3 seconds 2017-06-01 20:08:36,036 INFO repeater task os150bb_tsstats starting 10.3 seconds late 2017-06-01 20:08:36,608 INFO repeater task os150bb_tsstats ended after 0.6 seconds 2017-06-01 20:08:36,608 INFO repeater task os150nb_lastensq starting 0.6 seconds late 2017-06-01 20:08:37,803 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 20:08:37,803 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 20:08:26 2017-06-01 20:08:37,804 INFO repeater task os150nb_quick starting 12.0 seconds late 2017-06-01 20:08:40,850 INFO repeater task os150nb_quick ended after 3.0 seconds 2017-06-01 20:08:40,850 INFO repeater task os150nb_3dayplots starting 15.1 seconds late 2017-06-01 20:08:47,146 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-06-01 20:08:47,146 INFO repeater task os150nb_3dayplots ended after 6.3 seconds 2017-06-01 20:08:47,146 INFO repeater task os150nb_tsstats starting 21.4 seconds late 2017-06-01 20:08:47,727 INFO repeater task os150nb_tsstats ended after 0.6 seconds 2017-06-01 20:08:47,727 INFO repeater task os150bb_beamplots starting 21.9 seconds late 2017-06-01 20:08:48,652 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 20:08:48,652 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 20:08:48,652 INFO repeater task os150bb_beamplots starting 22.9 seconds late 2017-06-01 20:08:49,574 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 20:08:49,574 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 20:08:49,575 INFO repeater task os150nb_beamplots starting 23.8 seconds late 2017-06-01 20:08:50,490 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 20:08:50,490 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 20:08:50,490 INFO repeater task os150nb_beamplots starting 24.7 seconds late 2017-06-01 20:08:51,412 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 20:08:51,412 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 20:08:51,412 INFO repeater task posmv_hcorrstats starting 25.6 seconds late 2017-06-01 20:08:52,277 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 20:08:52,278 INFO repeater task posmv_plotposmv starting 26.5 seconds late 2017-06-01 20:08:52,873 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 20:08:57,878 INFO repeater task os150bb_lastensq starting 21.9 seconds late 2017-06-01 20:08:59,565 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 20:08:59,565 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 20:08:26 2017-06-01 20:11:29,875 INFO repeater task os150bb_10pings starting 4.1 seconds late 2017-06-01 20:11:30,025 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 20:11:30,027 INFO repeater task os150nb_10pings starting 4.2 seconds late 2017-06-01 20:11:30,121 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 20:13:30,378 INFO repeater task posmv_hcorrstats starting 4.6 seconds late 2017-06-01 20:13:31,314 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 20:13:31,315 INFO repeater task posmv_plotposmv starting 5.5 seconds late 2017-06-01 20:13:31,934 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 20:13:36,940 INFO repeater task os150bb_lastensq starting 0.9 seconds late 2017-06-01 20:13:38,565 INFO repeater task os150bb_lastensq ended after 1.6 seconds 2017-06-01 20:13:38,565 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 20:13:26 2017-06-01 20:13:38,566 INFO repeater task os150nb_lastensq starting 2.6 seconds late 2017-06-01 20:13:39,727 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 20:13:39,727 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 20:13:26 2017-06-01 20:16:50,105 INFO repeater task os150bb_10pings starting 4.3 seconds late 2017-06-01 20:16:50,267 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 20:16:50,268 INFO repeater task os150nb_10pings starting 4.5 seconds late 2017-06-01 20:16:50,364 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 20:18:30,552 INFO repeater task posmv_hcorrstats starting 4.8 seconds late 2017-06-01 20:18:31,379 INFO repeater task posmv_hcorrstats ended after 0.8 seconds 2017-06-01 20:18:31,379 INFO repeater task posmv_plotposmv starting 5.6 seconds late 2017-06-01 20:18:31,976 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 20:18:36,982 INFO repeater task os150bb_lastensq starting 1.0 seconds late 2017-06-01 20:18:38,723 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 20:18:38,723 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 20:18:26 2017-06-01 20:18:38,724 INFO repeater task os150nb_lastensq starting 2.7 seconds late 2017-06-01 20:18:39,859 INFO repeater task os150nb_lastensq ended after 1.1 seconds 2017-06-01 20:18:39,859 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 20:18:26 2017-06-01 20:22:10,294 INFO repeater task os150bb_10pings starting 4.5 seconds late 2017-06-01 20:22:10,464 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 20:22:10,464 INFO repeater task os150nb_10pings starting 4.7 seconds late 2017-06-01 20:22:10,557 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 20:23:30,722 INFO repeater task os150bb_quick starting 4.9 seconds late 2017-06-01 20:23:33,925 INFO repeater task os150bb_quick ended after 3.2 seconds 2017-06-01 20:23:33,926 INFO repeater task os150bb_3dayplots starting 8.1 seconds late 2017-06-01 20:23:40,222 INFO repeater task os150bb_3dayplots ended after 6.3 seconds 2017-06-01 20:23:40,222 INFO repeater task os150bb_tsstats starting 14.4 seconds late 2017-06-01 20:23:40,784 INFO repeater task os150bb_tsstats ended after 0.6 seconds 2017-06-01 20:23:40,785 INFO repeater task os150nb_lastensq starting 4.8 seconds late 2017-06-01 20:23:41,910 INFO repeater task os150nb_lastensq ended after 1.1 seconds 2017-06-01 20:23:41,910 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 20:23:26 2017-06-01 20:23:41,911 INFO repeater task os150nb_quick starting 16.1 seconds late 2017-06-01 20:23:45,009 INFO repeater task os150nb_quick ended after 3.1 seconds 2017-06-01 20:23:45,009 INFO repeater task os150nb_3dayplots starting 19.2 seconds late 2017-06-01 20:23:51,352 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-06-01 20:23:51,352 INFO repeater task os150nb_3dayplots ended after 6.3 seconds 2017-06-01 20:23:51,352 INFO repeater task os150nb_tsstats starting 25.6 seconds late 2017-06-01 20:23:51,927 INFO repeater task os150nb_tsstats ended after 0.6 seconds 2017-06-01 20:23:51,928 INFO repeater task os150bb_beamplots starting 26.1 seconds late 2017-06-01 20:23:52,823 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 20:23:52,823 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 20:23:52,824 INFO repeater task os150bb_beamplots starting 27.0 seconds late 2017-06-01 20:23:53,740 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 20:23:53,740 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 20:23:53,740 INFO repeater task os150nb_beamplots starting 28.0 seconds late 2017-06-01 20:23:54,684 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 20:23:54,684 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 20:23:54,684 INFO repeater task os150nb_beamplots starting 28.9 seconds late 2017-06-01 20:23:55,635 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 20:23:55,635 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2017-06-01 20:23:55,636 INFO repeater task posmv_hcorrstats starting 29.9 seconds late 2017-06-01 20:23:56,468 INFO repeater task posmv_hcorrstats ended after 0.8 seconds 2017-06-01 20:23:56,469 INFO repeater task posmv_plotposmv starting 30.7 seconds late 2017-06-01 20:23:57,090 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 20:24:02,095 INFO repeater task os150bb_lastensq starting 26.1 seconds late 2017-06-01 20:24:03,808 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 20:24:03,809 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 20:23:26 2017-06-01 20:27:29,223 INFO repeater task os150bb_10pings starting 3.4 seconds late 2017-06-01 20:27:29,378 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 20:27:29,379 INFO repeater task os150nb_10pings starting 3.6 seconds late 2017-06-01 20:27:29,472 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 20:28:29,599 INFO repeater task posmv_hcorrstats starting 3.8 seconds late 2017-06-01 20:28:30,527 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 20:28:30,528 INFO repeater task posmv_plotposmv starting 4.7 seconds late 2017-06-01 20:28:31,130 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 20:28:36,136 INFO repeater task os150bb_lastensq starting 0.1 seconds late 2017-06-01 20:28:37,804 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 20:28:37,804 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 20:28:26 2017-06-01 20:28:37,805 INFO repeater task os150nb_lastensq starting 1.8 seconds late 2017-06-01 20:28:38,989 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 20:28:38,989 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 20:28:26 2017-06-01 20:32:49,374 INFO repeater task os150bb_10pings starting 3.6 seconds late 2017-06-01 20:32:49,496 INFO repeater task os150bb_10pings ended after 0.1 seconds 2017-06-01 20:32:49,497 INFO repeater task os150nb_10pings starting 3.7 seconds late 2017-06-01 20:32:49,591 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 20:33:29,631 INFO repeater task posmv_hcorrstats starting 3.8 seconds late 2017-06-01 20:33:30,503 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 20:33:30,504 INFO repeater task posmv_plotposmv starting 4.7 seconds late 2017-06-01 20:33:31,113 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 20:33:36,119 INFO repeater task os150bb_lastensq starting 0.1 seconds late 2017-06-01 20:33:38,017 INFO repeater task os150bb_lastensq ended after 1.9 seconds 2017-06-01 20:33:38,017 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 20:33:26 2017-06-01 20:33:38,018 INFO repeater task os150nb_lastensq starting 2.0 seconds late 2017-06-01 20:33:39,199 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 20:33:39,199 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 20:33:26 2017-06-01 20:38:09,750 INFO repeater task os150bb_10pings starting 4.0 seconds late 2017-06-01 20:38:09,905 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 20:38:09,905 INFO repeater task os150nb_10pings starting 4.1 seconds late 2017-06-01 20:38:10,001 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 20:38:30,043 INFO repeater task os150bb_quick starting 4.3 seconds late 2017-06-01 20:38:33,262 INFO repeater task os150bb_quick ended after 3.2 seconds 2017-06-01 20:38:33,262 INFO repeater task os150bb_3dayplots starting 7.5 seconds late 2017-06-01 20:38:39,935 INFO repeater task os150bb_3dayplots ended after 6.7 seconds 2017-06-01 20:38:39,936 INFO repeater task os150bb_tsstats starting 14.2 seconds late 2017-06-01 20:38:40,509 INFO repeater task os150bb_tsstats ended after 0.6 seconds 2017-06-01 20:38:40,509 INFO repeater task os150nb_lastensq starting 4.5 seconds late 2017-06-01 20:38:41,757 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 20:38:41,757 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 20:38:26 2017-06-01 20:38:41,757 INFO repeater task os150nb_quick starting 16.0 seconds late 2017-06-01 20:38:44,849 INFO repeater task os150nb_quick ended after 3.1 seconds 2017-06-01 20:38:44,850 INFO repeater task os150nb_3dayplots starting 19.1 seconds late 2017-06-01 20:38:51,825 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-06-01 20:38:51,825 INFO repeater task os150nb_3dayplots ended after 7.0 seconds 2017-06-01 20:38:51,825 INFO repeater task os150nb_tsstats starting 26.0 seconds late 2017-06-01 20:38:52,410 INFO repeater task os150nb_tsstats ended after 0.6 seconds 2017-06-01 20:38:52,410 INFO repeater task os150bb_beamplots starting 26.6 seconds late 2017-06-01 20:38:53,392 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 20:38:53,392 INFO repeater task os150bb_beamplots ended after 1.0 seconds 2017-06-01 20:38:53,392 INFO repeater task os150bb_beamplots starting 27.6 seconds late 2017-06-01 20:38:54,307 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 20:38:54,307 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 20:38:54,307 INFO repeater task os150nb_beamplots starting 28.5 seconds late 2017-06-01 20:38:55,257 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 20:38:55,257 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 20:38:55,257 INFO repeater task os150nb_beamplots starting 29.5 seconds late 2017-06-01 20:38:56,204 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 20:38:56,204 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 20:38:56,204 INFO repeater task posmv_hcorrstats starting 30.4 seconds late 2017-06-01 20:38:57,093 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 20:38:57,093 INFO repeater task posmv_plotposmv starting 31.3 seconds late 2017-06-01 20:38:57,723 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 20:39:02,729 INFO repeater task os150bb_lastensq starting 26.7 seconds late 2017-06-01 20:39:04,562 INFO repeater task os150bb_lastensq ended after 1.8 seconds 2017-06-01 20:39:04,563 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 20:38:26 2017-06-01 20:43:30,129 INFO repeater task os150bb_10pings starting 4.3 seconds late 2017-06-01 20:43:30,228 INFO repeater task os150bb_10pings ended after 0.1 seconds 2017-06-01 20:43:30,228 INFO repeater task os150nb_10pings starting 4.4 seconds late 2017-06-01 20:43:30,322 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 20:43:30,323 INFO repeater task posmv_hcorrstats starting 4.5 seconds late 2017-06-01 20:43:31,179 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 20:43:31,179 INFO repeater task posmv_plotposmv starting 5.4 seconds late 2017-06-01 20:43:31,829 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 20:43:36,831 INFO repeater task os150bb_lastensq starting 0.8 seconds late 2017-06-01 20:43:38,542 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 20:43:38,542 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 20:43:26 2017-06-01 20:43:38,543 INFO repeater task os150nb_lastensq starting 2.5 seconds late 2017-06-01 20:43:39,736 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 20:43:39,736 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 20:43:26 2017-06-01 20:48:30,310 INFO repeater task posmv_hcorrstats starting 4.5 seconds late 2017-06-01 20:48:31,253 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 20:48:31,253 INFO repeater task posmv_plotposmv starting 5.5 seconds late 2017-06-01 20:48:31,856 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 20:48:36,862 INFO repeater task os150bb_lastensq starting 0.9 seconds late 2017-06-01 20:48:38,474 INFO repeater task os150bb_lastensq ended after 1.6 seconds 2017-06-01 20:48:38,474 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 20:48:26 2017-06-01 20:48:38,475 INFO repeater task os150nb_lastensq starting 2.5 seconds late 2017-06-01 20:48:39,619 INFO repeater task os150nb_lastensq ended after 1.1 seconds 2017-06-01 20:48:39,619 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 20:48:26 2017-06-01 20:48:49,635 INFO repeater task os150bb_10pings starting 3.9 seconds late 2017-06-01 20:48:49,803 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 20:48:49,804 INFO repeater task os150nb_10pings starting 4.0 seconds late 2017-06-01 20:48:49,897 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 20:53:30,383 INFO repeater task os150bb_quick starting 4.6 seconds late 2017-06-01 20:53:33,554 INFO repeater task os150bb_quick ended after 3.2 seconds 2017-06-01 20:53:33,555 INFO repeater task os150bb_3dayplots starting 7.8 seconds late 2017-06-01 20:53:39,797 INFO repeater task os150bb_3dayplots ended after 6.2 seconds 2017-06-01 20:53:39,797 INFO repeater task os150bb_tsstats starting 14.0 seconds late 2017-06-01 20:53:40,339 INFO repeater task os150bb_tsstats ended after 0.5 seconds 2017-06-01 20:53:40,339 INFO repeater task os150nb_lastensq starting 4.3 seconds late 2017-06-01 20:53:41,518 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 20:53:41,518 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 20:53:26 2017-06-01 20:53:41,518 INFO repeater task os150nb_quick starting 15.7 seconds late 2017-06-01 20:53:44,573 INFO repeater task os150nb_quick ended after 3.1 seconds 2017-06-01 20:53:44,574 INFO repeater task os150nb_3dayplots starting 18.8 seconds late 2017-06-01 20:53:50,852 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-06-01 20:53:50,852 INFO repeater task os150nb_3dayplots ended after 6.3 seconds 2017-06-01 20:53:50,852 INFO repeater task os150nb_tsstats starting 25.1 seconds late 2017-06-01 20:53:51,386 INFO repeater task os150nb_tsstats ended after 0.5 seconds 2017-06-01 20:53:51,386 INFO repeater task os150bb_beamplots starting 25.6 seconds late 2017-06-01 20:53:52,300 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 20:53:52,300 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 20:53:52,300 INFO repeater task os150bb_beamplots starting 26.5 seconds late 2017-06-01 20:53:53,198 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 20:53:53,199 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 20:53:53,199 INFO repeater task os150nb_beamplots starting 27.4 seconds late 2017-06-01 20:53:54,099 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 20:53:54,099 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 20:53:54,099 INFO repeater task os150nb_beamplots starting 28.3 seconds late 2017-06-01 20:53:55,057 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 20:53:55,057 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2017-06-01 20:53:55,058 INFO repeater task posmv_hcorrstats starting 29.3 seconds late 2017-06-01 20:53:55,903 INFO repeater task posmv_hcorrstats ended after 0.8 seconds 2017-06-01 20:53:55,903 INFO repeater task posmv_plotposmv starting 30.1 seconds late 2017-06-01 20:53:56,525 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 20:54:01,530 INFO repeater task os150bb_lastensq starting 25.5 seconds late 2017-06-01 20:54:03,253 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 20:54:03,253 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 20:53:26 2017-06-01 20:54:08,257 INFO repeater task os150bb_10pings starting 2.5 seconds late 2017-06-01 20:54:08,409 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 20:54:08,410 INFO repeater task os150nb_10pings starting 2.6 seconds late 2017-06-01 20:54:08,505 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 20:58:29,102 INFO repeater task posmv_hcorrstats starting 3.3 seconds late 2017-06-01 20:58:30,093 INFO repeater task posmv_hcorrstats ended after 1.0 seconds 2017-06-01 20:58:30,094 INFO repeater task posmv_plotposmv starting 4.3 seconds late 2017-06-01 20:58:30,714 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 20:58:40,730 INFO repeater task os150bb_lastensq starting 4.7 seconds late 2017-06-01 20:58:42,405 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 20:58:42,405 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 20:58:26 2017-06-01 20:58:42,406 INFO repeater task os150nb_lastensq starting 6.4 seconds late 2017-06-01 20:58:43,603 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 20:58:43,603 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 20:58:26 2017-06-01 20:59:28,709 INFO repeater task os150bb_10pings starting 2.9 seconds late 2017-06-01 20:59:28,862 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 20:59:28,863 INFO repeater task os150nb_10pings starting 3.1 seconds late 2017-06-01 20:59:28,958 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 21:03:29,392 INFO repeater task posmv_hcorrstats starting 3.6 seconds late 2017-06-01 21:03:30,247 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 21:03:30,248 INFO repeater task posmv_plotposmv starting 4.5 seconds late 2017-06-01 21:03:30,869 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 21:03:40,881 INFO repeater task os150bb_lastensq starting 4.9 seconds late 2017-06-01 21:03:42,648 INFO repeater task os150bb_lastensq ended after 1.8 seconds 2017-06-01 21:03:42,648 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 21:03:26 2017-06-01 21:03:42,649 INFO repeater task os150nb_lastensq starting 6.6 seconds late 2017-06-01 21:03:43,834 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 21:03:43,835 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 21:03:26 2017-06-01 21:04:48,935 INFO repeater task os150bb_10pings starting 3.2 seconds late 2017-06-01 21:04:49,026 INFO repeater task os150bb_10pings ended after 0.1 seconds 2017-06-01 21:04:49,027 INFO repeater task os150nb_10pings starting 3.2 seconds late 2017-06-01 21:04:49,121 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 21:08:29,539 INFO repeater task os150bb_quick starting 3.8 seconds late 2017-06-01 21:08:32,676 INFO repeater task os150bb_quick ended after 3.1 seconds 2017-06-01 21:08:32,676 INFO repeater task os150bb_3dayplots starting 6.9 seconds late 2017-06-01 21:08:38,898 INFO repeater task os150bb_3dayplots ended after 6.2 seconds 2017-06-01 21:08:38,898 INFO repeater task os150bb_tsstats starting 13.1 seconds late 2017-06-01 21:08:39,477 INFO repeater task os150bb_tsstats ended after 0.6 seconds 2017-06-01 21:08:39,477 INFO repeater task os150nb_lastensq starting 3.5 seconds late 2017-06-01 21:08:40,663 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 21:08:40,663 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 21:08:26 2017-06-01 21:08:40,664 INFO repeater task os150nb_quick starting 14.9 seconds late 2017-06-01 21:08:43,770 INFO repeater task os150nb_quick ended after 3.1 seconds 2017-06-01 21:08:43,770 INFO repeater task os150nb_3dayplots starting 18.0 seconds late 2017-06-01 21:08:50,089 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-06-01 21:08:50,089 INFO repeater task os150nb_3dayplots ended after 6.3 seconds 2017-06-01 21:08:50,089 INFO repeater task os150nb_tsstats starting 24.3 seconds late 2017-06-01 21:08:50,674 INFO repeater task os150nb_tsstats ended after 0.6 seconds 2017-06-01 21:08:50,674 INFO repeater task os150bb_beamplots starting 24.9 seconds late 2017-06-01 21:08:51,630 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 21:08:51,630 INFO repeater task os150bb_beamplots ended after 1.0 seconds 2017-06-01 21:08:51,630 INFO repeater task os150bb_beamplots starting 25.8 seconds late 2017-06-01 21:08:52,542 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 21:08:52,543 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 21:08:52,543 INFO repeater task os150nb_beamplots starting 26.8 seconds late 2017-06-01 21:08:53,455 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 21:08:53,455 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 21:08:53,455 INFO repeater task os150nb_beamplots starting 27.7 seconds late 2017-06-01 21:08:54,376 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 21:08:54,376 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 21:08:54,376 INFO repeater task posmv_hcorrstats starting 28.6 seconds late 2017-06-01 21:08:55,276 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 21:08:55,276 INFO repeater task posmv_plotposmv starting 29.5 seconds late 2017-06-01 21:08:55,930 INFO repeater task posmv_plotposmv ended after 0.7 seconds 2017-06-01 21:09:00,936 INFO repeater task os150bb_lastensq starting 24.9 seconds late 2017-06-01 21:09:02,569 INFO repeater task os150bb_lastensq ended after 1.6 seconds 2017-06-01 21:09:02,569 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 21:08:26 2017-06-01 21:10:07,676 INFO repeater task os150bb_10pings starting 1.9 seconds late 2017-06-01 21:10:07,839 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 21:10:07,839 INFO repeater task os150nb_10pings starting 2.1 seconds late 2017-06-01 21:10:07,935 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 21:13:28,341 INFO repeater task posmv_hcorrstats starting 2.6 seconds late 2017-06-01 21:13:29,299 INFO repeater task posmv_hcorrstats ended after 1.0 seconds 2017-06-01 21:13:29,299 INFO repeater task posmv_plotposmv starting 3.5 seconds late 2017-06-01 21:13:29,918 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 21:13:39,934 INFO repeater task os150bb_lastensq starting 3.9 seconds late 2017-06-01 21:13:41,787 INFO repeater task os150bb_lastensq ended after 1.9 seconds 2017-06-01 21:13:41,788 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 21:13:26 2017-06-01 21:13:41,788 INFO repeater task os150nb_lastensq starting 5.8 seconds late 2017-06-01 21:13:42,971 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 21:13:42,971 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 21:13:26 2017-06-01 21:15:28,160 INFO repeater task os150bb_10pings starting 2.4 seconds late 2017-06-01 21:15:28,315 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 21:15:28,315 INFO repeater task os150nb_10pings starting 2.5 seconds late 2017-06-01 21:15:28,414 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 21:18:28,789 INFO repeater task posmv_hcorrstats starting 3.0 seconds late 2017-06-01 21:18:29,694 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 21:18:29,694 INFO repeater task posmv_plotposmv starting 3.9 seconds late 2017-06-01 21:18:30,312 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 21:18:40,322 INFO repeater task os150bb_lastensq starting 4.3 seconds late 2017-06-01 21:18:42,007 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 21:18:42,008 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 21:18:26 2017-06-01 21:18:42,009 INFO repeater task os150nb_lastensq starting 6.0 seconds late 2017-06-01 21:18:43,213 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 21:18:43,213 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 21:18:26 2017-06-01 21:20:48,432 INFO repeater task os150bb_10pings starting 2.7 seconds late 2017-06-01 21:20:48,528 INFO repeater task os150bb_10pings ended after 0.1 seconds 2017-06-01 21:20:48,529 INFO repeater task os150nb_10pings starting 2.7 seconds late 2017-06-01 21:20:48,623 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 21:23:28,941 INFO repeater task os150bb_quick starting 3.2 seconds late 2017-06-01 21:23:32,061 INFO repeater task os150bb_quick ended after 3.1 seconds 2017-06-01 21:23:32,061 INFO repeater task os150bb_3dayplots starting 6.3 seconds late 2017-06-01 21:23:38,316 INFO repeater task os150bb_3dayplots ended after 6.3 seconds 2017-06-01 21:23:38,317 INFO repeater task os150bb_tsstats starting 12.5 seconds late 2017-06-01 21:23:38,862 INFO repeater task os150bb_tsstats ended after 0.5 seconds 2017-06-01 21:23:38,862 INFO repeater task os150nb_lastensq starting 2.9 seconds late 2017-06-01 21:23:40,026 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 21:23:40,026 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 21:23:26 2017-06-01 21:23:40,026 INFO repeater task os150nb_quick starting 14.2 seconds late 2017-06-01 21:23:43,098 INFO repeater task os150nb_quick ended after 3.1 seconds 2017-06-01 21:23:43,099 INFO repeater task os150nb_3dayplots starting 17.3 seconds late 2017-06-01 21:23:49,402 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-06-01 21:23:49,402 INFO repeater task os150nb_3dayplots ended after 6.3 seconds 2017-06-01 21:23:49,403 INFO repeater task os150nb_tsstats starting 23.6 seconds late 2017-06-01 21:23:49,965 INFO repeater task os150nb_tsstats ended after 0.6 seconds 2017-06-01 21:23:49,965 INFO repeater task os150bb_beamplots starting 24.2 seconds late 2017-06-01 21:23:50,854 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 21:23:50,854 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 21:23:50,855 INFO repeater task os150bb_beamplots starting 25.1 seconds late 2017-06-01 21:23:51,765 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 21:23:51,765 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 21:23:51,765 INFO repeater task os150nb_beamplots starting 26.0 seconds late 2017-06-01 21:23:52,691 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 21:23:52,691 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 21:23:52,691 INFO repeater task os150nb_beamplots starting 26.9 seconds late 2017-06-01 21:23:53,594 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 21:23:53,594 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 21:23:53,594 INFO repeater task posmv_hcorrstats starting 27.8 seconds late 2017-06-01 21:23:54,456 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 21:23:54,456 INFO repeater task posmv_plotposmv starting 28.7 seconds late 2017-06-01 21:23:55,068 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 21:24:00,074 INFO repeater task os150bb_lastensq starting 24.1 seconds late 2017-06-01 21:24:01,793 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 21:24:01,794 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 21:23:26 2017-06-01 21:26:07,038 INFO repeater task os150bb_10pings starting 1.3 seconds late 2017-06-01 21:26:07,189 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 21:26:07,190 INFO repeater task os150nb_10pings starting 1.4 seconds late 2017-06-01 21:26:07,287 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 21:28:27,579 INFO repeater task posmv_hcorrstats starting 1.8 seconds late 2017-06-01 21:28:28,494 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 21:28:28,495 INFO repeater task posmv_plotposmv starting 2.7 seconds late 2017-06-01 21:28:29,108 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 21:28:39,126 INFO repeater task os150bb_lastensq starting 3.1 seconds late 2017-06-01 21:28:40,890 INFO repeater task os150bb_lastensq ended after 1.8 seconds 2017-06-01 21:28:40,890 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 21:28:26 2017-06-01 21:28:40,891 INFO repeater task os150nb_lastensq starting 4.9 seconds late 2017-06-01 21:28:42,047 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 21:28:42,047 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 21:28:26 2017-06-01 21:31:27,345 INFO repeater task os150bb_10pings starting 1.6 seconds late 2017-06-01 21:31:27,438 INFO repeater task os150bb_10pings ended after 0.1 seconds 2017-06-01 21:31:27,439 INFO repeater task os150nb_10pings starting 1.7 seconds late 2017-06-01 21:31:27,530 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 21:33:27,813 INFO repeater task posmv_hcorrstats starting 2.0 seconds late 2017-06-01 21:33:28,768 INFO repeater task posmv_hcorrstats ended after 1.0 seconds 2017-06-01 21:33:28,768 INFO repeater task posmv_plotposmv starting 3.0 seconds late 2017-06-01 21:33:29,391 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 21:33:39,402 INFO repeater task os150bb_lastensq starting 3.4 seconds late 2017-06-01 21:33:41,132 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 21:33:41,133 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 21:33:26 2017-06-01 21:33:41,133 INFO repeater task os150nb_lastensq starting 5.1 seconds late 2017-06-01 21:33:42,435 INFO repeater task os150nb_lastensq ended after 1.3 seconds 2017-06-01 21:33:42,435 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 21:33:26 2017-06-01 21:36:47,743 INFO repeater task os150bb_10pings starting 2.0 seconds late 2017-06-01 21:36:47,919 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 21:36:47,919 INFO repeater task os150nb_10pings starting 2.1 seconds late 2017-06-01 21:36:48,019 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 21:38:28,199 INFO repeater task os150bb_quick starting 2.4 seconds late 2017-06-01 21:38:31,480 INFO repeater task os150bb_quick ended after 3.3 seconds 2017-06-01 21:38:31,481 INFO repeater task os150bb_3dayplots starting 5.7 seconds late 2017-06-01 21:38:37,770 INFO repeater task os150bb_3dayplots ended after 6.3 seconds 2017-06-01 21:38:37,770 INFO repeater task os150bb_tsstats starting 12.0 seconds late 2017-06-01 21:38:38,312 INFO repeater task os150bb_tsstats ended after 0.5 seconds 2017-06-01 21:38:38,312 INFO repeater task os150nb_lastensq starting 2.3 seconds late 2017-06-01 21:38:39,492 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 21:38:39,492 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 21:38:26 2017-06-01 21:38:39,493 INFO repeater task os150nb_quick starting 13.7 seconds late 2017-06-01 21:38:42,604 INFO repeater task os150nb_quick ended after 3.1 seconds 2017-06-01 21:38:42,604 INFO repeater task os150nb_3dayplots starting 16.8 seconds late 2017-06-01 21:38:48,956 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-06-01 21:38:48,957 INFO repeater task os150nb_3dayplots ended after 6.4 seconds 2017-06-01 21:38:48,957 INFO repeater task os150nb_tsstats starting 23.2 seconds late 2017-06-01 21:38:49,509 INFO repeater task os150nb_tsstats ended after 0.6 seconds 2017-06-01 21:38:49,509 INFO repeater task os150bb_beamplots starting 23.7 seconds late 2017-06-01 21:38:50,406 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 21:38:50,406 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 21:38:50,407 INFO repeater task os150bb_beamplots starting 24.6 seconds late 2017-06-01 21:38:51,324 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 21:38:51,324 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 21:38:51,324 INFO repeater task os150nb_beamplots starting 25.5 seconds late 2017-06-01 21:38:52,224 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 21:38:52,224 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 21:38:52,224 INFO repeater task os150nb_beamplots starting 26.4 seconds late 2017-06-01 21:38:53,133 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 21:38:53,133 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 21:38:53,133 INFO repeater task posmv_hcorrstats starting 27.4 seconds late 2017-06-01 21:38:53,993 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 21:38:53,994 INFO repeater task posmv_plotposmv starting 28.2 seconds late 2017-06-01 21:38:54,601 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 21:38:59,606 INFO repeater task os150bb_lastensq starting 23.6 seconds late 2017-06-01 21:39:01,232 INFO repeater task os150bb_lastensq ended after 1.6 seconds 2017-06-01 21:39:01,232 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 21:38:26 2017-06-01 21:42:06,618 INFO repeater task os150bb_10pings starting 0.8 seconds late 2017-06-01 21:42:06,785 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 21:42:06,786 INFO repeater task os150nb_10pings starting 1.0 seconds late 2017-06-01 21:42:06,879 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 21:43:27,034 INFO repeater task posmv_hcorrstats starting 1.3 seconds late 2017-06-01 21:43:27,973 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 21:43:27,974 INFO repeater task posmv_plotposmv starting 2.2 seconds late 2017-06-01 21:43:28,580 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 21:43:38,598 INFO repeater task os150bb_lastensq starting 2.6 seconds late 2017-06-01 21:43:40,326 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 21:43:40,327 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 21:43:26 2017-06-01 21:43:40,328 INFO repeater task os150nb_lastensq starting 4.3 seconds late 2017-06-01 21:43:41,483 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 21:43:41,483 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 21:43:26 2017-06-01 21:47:26,963 INFO repeater task os150bb_10pings starting 1.2 seconds late 2017-06-01 21:47:27,145 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 21:47:27,146 INFO repeater task os150nb_10pings starting 1.4 seconds late 2017-06-01 21:47:27,241 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 21:48:27,380 INFO repeater task posmv_hcorrstats starting 1.6 seconds late 2017-06-01 21:48:28,281 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 21:48:28,282 INFO repeater task posmv_plotposmv starting 2.5 seconds late 2017-06-01 21:48:28,898 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 21:48:38,912 INFO repeater task os150bb_lastensq starting 2.9 seconds late 2017-06-01 21:48:40,657 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 21:48:40,657 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 21:48:26 2017-06-01 21:48:40,658 INFO repeater task os150nb_lastensq starting 4.7 seconds late 2017-06-01 21:48:41,809 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 21:48:41,809 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 21:48:26 2017-06-01 21:52:47,262 INFO repeater task os150bb_10pings starting 1.5 seconds late 2017-06-01 21:52:47,432 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 21:52:47,433 INFO repeater task os150nb_10pings starting 1.7 seconds late 2017-06-01 21:52:47,526 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 21:53:27,606 INFO repeater task os150bb_quick starting 1.8 seconds late 2017-06-01 21:53:30,822 INFO repeater task os150bb_quick ended after 3.2 seconds 2017-06-01 21:53:30,823 INFO repeater task os150bb_3dayplots starting 5.0 seconds late 2017-06-01 21:53:37,040 INFO repeater task os150bb_3dayplots ended after 6.2 seconds 2017-06-01 21:53:37,041 INFO repeater task os150bb_tsstats starting 11.3 seconds late 2017-06-01 21:53:37,574 INFO repeater task os150bb_tsstats ended after 0.5 seconds 2017-06-01 21:53:37,575 INFO repeater task os150nb_lastensq starting 1.6 seconds late 2017-06-01 21:53:38,818 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 21:53:38,818 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 21:53:26 2017-06-01 21:53:38,819 INFO repeater task os150nb_quick starting 13.0 seconds late 2017-06-01 21:53:41,886 INFO repeater task os150nb_quick ended after 3.1 seconds 2017-06-01 21:53:41,887 INFO repeater task os150nb_3dayplots starting 16.1 seconds late 2017-06-01 21:53:48,231 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-06-01 21:53:48,231 INFO repeater task os150nb_3dayplots ended after 6.3 seconds 2017-06-01 21:53:48,231 INFO repeater task os150nb_tsstats starting 22.5 seconds late 2017-06-01 21:53:48,784 INFO repeater task os150nb_tsstats ended after 0.6 seconds 2017-06-01 21:53:48,785 INFO repeater task os150bb_beamplots starting 23.0 seconds late 2017-06-01 21:53:49,685 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 21:53:49,685 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 21:53:49,685 INFO repeater task os150bb_beamplots starting 23.9 seconds late 2017-06-01 21:53:50,629 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 21:53:50,630 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 21:53:50,630 INFO repeater task os150nb_beamplots starting 24.8 seconds late 2017-06-01 21:53:51,597 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 21:53:51,597 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2017-06-01 21:53:51,597 INFO repeater task os150nb_beamplots starting 25.8 seconds late 2017-06-01 21:53:52,519 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 21:53:52,519 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 21:53:52,520 INFO repeater task posmv_hcorrstats starting 26.7 seconds late 2017-06-01 21:53:53,381 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 21:53:53,382 INFO repeater task posmv_plotposmv starting 27.6 seconds late 2017-06-01 21:53:53,980 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 21:53:58,986 INFO repeater task os150bb_lastensq starting 23.0 seconds late 2017-06-01 21:54:00,696 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 21:54:00,697 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 21:53:26 2017-06-01 21:58:06,256 INFO repeater task os150bb_10pings starting 0.5 seconds late 2017-06-01 21:58:06,430 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 21:58:06,431 INFO repeater task os150nb_10pings starting 0.7 seconds late 2017-06-01 21:58:06,523 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 21:58:26,567 INFO repeater task posmv_hcorrstats starting 0.8 seconds late 2017-06-01 21:58:27,506 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 21:58:27,506 INFO repeater task posmv_plotposmv starting 1.7 seconds late 2017-06-01 21:58:28,120 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 21:58:38,127 INFO repeater task os150bb_lastensq starting 2.1 seconds late 2017-06-01 21:58:39,771 INFO repeater task os150bb_lastensq ended after 1.6 seconds 2017-06-01 21:58:39,771 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 21:58:26 2017-06-01 21:58:39,772 INFO repeater task os150nb_lastensq starting 3.8 seconds late 2017-06-01 21:58:40,950 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 21:58:40,950 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 21:58:26 2017-06-01 22:03:26,572 INFO repeater task os150bb_10pings starting 0.8 seconds late 2017-06-01 22:03:26,705 INFO repeater task os150bb_10pings ended after 0.1 seconds 2017-06-01 22:03:26,706 INFO repeater task os150nb_10pings starting 0.9 seconds late 2017-06-01 22:03:26,800 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 22:03:26,801 INFO repeater task posmv_hcorrstats starting 1.0 seconds late 2017-06-01 22:03:27,673 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 22:03:27,673 INFO repeater task posmv_plotposmv starting 1.9 seconds late 2017-06-01 22:03:28,264 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 22:03:38,280 INFO repeater task os150bb_lastensq starting 2.3 seconds late 2017-06-01 22:03:39,988 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 22:03:39,988 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 22:03:26 2017-06-01 22:03:39,989 INFO repeater task os150nb_lastensq starting 4.0 seconds late 2017-06-01 22:03:41,188 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 22:03:41,188 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 22:03:26 2017-06-01 22:08:26,746 INFO repeater task os150bb_quick starting 1.0 seconds late 2017-06-01 22:08:30,054 INFO repeater task os150bb_quick ended after 3.3 seconds 2017-06-01 22:08:30,054 INFO repeater task os150bb_3dayplots starting 4.3 seconds late 2017-06-01 22:08:36,277 INFO repeater task os150bb_3dayplots ended after 6.2 seconds 2017-06-01 22:08:36,277 INFO repeater task os150bb_tsstats starting 10.5 seconds late 2017-06-01 22:08:36,823 INFO repeater task os150bb_tsstats ended after 0.5 seconds 2017-06-01 22:08:36,823 INFO repeater task os150nb_lastensq starting 0.8 seconds late 2017-06-01 22:08:37,972 INFO repeater task os150nb_lastensq ended after 1.1 seconds 2017-06-01 22:08:37,972 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 22:08:26 2017-06-01 22:08:37,972 INFO repeater task os150nb_quick starting 12.2 seconds late 2017-06-01 22:08:41,108 INFO repeater task os150nb_quick ended after 3.1 seconds 2017-06-01 22:08:41,109 INFO repeater task os150nb_3dayplots starting 15.3 seconds late 2017-06-01 22:08:47,424 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-06-01 22:08:47,424 INFO repeater task os150nb_3dayplots ended after 6.3 seconds 2017-06-01 22:08:47,425 INFO repeater task os150nb_tsstats starting 21.6 seconds late 2017-06-01 22:08:48,007 INFO repeater task os150nb_tsstats ended after 0.6 seconds 2017-06-01 22:08:48,008 INFO repeater task os150bb_beamplots starting 22.2 seconds late 2017-06-01 22:08:48,929 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 22:08:48,929 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 22:08:48,930 INFO repeater task os150bb_beamplots starting 23.1 seconds late 2017-06-01 22:08:49,874 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 22:08:49,874 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 22:08:49,874 INFO repeater task os150nb_beamplots starting 24.1 seconds late 2017-06-01 22:08:50,772 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 22:08:50,772 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 22:08:50,773 INFO repeater task os150nb_beamplots starting 25.0 seconds late 2017-06-01 22:08:51,696 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 22:08:51,696 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 22:08:51,696 INFO repeater task posmv_hcorrstats starting 25.9 seconds late 2017-06-01 22:08:52,554 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 22:08:52,554 INFO repeater task posmv_plotposmv starting 26.8 seconds late 2017-06-01 22:08:53,156 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 22:08:58,162 INFO repeater task os150bb_lastensq starting 22.2 seconds late 2017-06-01 22:08:59,845 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 22:08:59,845 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 22:08:26 2017-06-01 22:08:59,845 INFO repeater task os150bb_10pings starting 14.1 seconds late 2017-06-01 22:08:59,939 INFO repeater task os150bb_10pings ended after 0.1 seconds 2017-06-01 22:08:59,940 INFO repeater task os150nb_10pings starting 14.2 seconds late 2017-06-01 22:09:00,034 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 22:13:30,589 INFO repeater task posmv_hcorrstats starting 4.8 seconds late 2017-06-01 22:13:31,456 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 22:13:31,457 INFO repeater task posmv_plotposmv starting 5.7 seconds late 2017-06-01 22:13:32,075 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 22:13:37,080 INFO repeater task os150bb_lastensq starting 1.1 seconds late 2017-06-01 22:13:38,741 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 22:13:38,742 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 22:13:26 2017-06-01 22:13:38,742 INFO repeater task os150nb_lastensq starting 2.7 seconds late 2017-06-01 22:13:39,932 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 22:13:39,933 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 22:13:26 2017-06-01 22:14:09,990 INFO repeater task os150bb_10pings starting 4.2 seconds late 2017-06-01 22:14:10,159 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 22:14:10,160 INFO repeater task os150nb_10pings starting 4.4 seconds late 2017-06-01 22:14:10,254 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 22:18:30,761 INFO repeater task posmv_hcorrstats starting 5.0 seconds late 2017-06-01 22:18:31,651 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 22:18:31,652 INFO repeater task posmv_plotposmv starting 5.9 seconds late 2017-06-01 22:18:32,268 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 22:18:37,273 INFO repeater task os150bb_lastensq starting 1.3 seconds late 2017-06-01 22:18:39,035 INFO repeater task os150bb_lastensq ended after 1.8 seconds 2017-06-01 22:18:39,035 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 22:18:26 2017-06-01 22:18:39,035 INFO repeater task os150nb_lastensq starting 3.0 seconds late 2017-06-01 22:18:40,227 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 22:18:40,227 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 22:18:26 2017-06-01 22:19:30,339 INFO repeater task os150bb_10pings starting 4.6 seconds late 2017-06-01 22:19:30,519 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 22:19:30,520 INFO repeater task os150nb_10pings starting 4.7 seconds late 2017-06-01 22:19:30,616 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 22:23:26,141 INFO repeater task os150bb_quick starting 0.4 seconds late 2017-06-01 22:23:29,358 INFO repeater task os150bb_quick ended after 3.2 seconds 2017-06-01 22:23:29,358 INFO repeater task os150bb_3dayplots starting 3.6 seconds late 2017-06-01 22:23:35,604 INFO repeater task os150bb_3dayplots ended after 6.2 seconds 2017-06-01 22:23:35,604 INFO repeater task os150bb_tsstats starting 9.8 seconds late 2017-06-01 22:23:36,140 INFO repeater task os150bb_tsstats ended after 0.5 seconds 2017-06-01 22:23:36,140 INFO repeater task os150nb_lastensq starting 0.1 seconds late 2017-06-01 22:23:37,306 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 22:23:37,307 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 22:23:26 2017-06-01 22:23:37,307 INFO repeater task os150nb_quick starting 11.5 seconds late 2017-06-01 22:23:40,420 INFO repeater task os150nb_quick ended after 3.1 seconds 2017-06-01 22:23:40,420 INFO repeater task os150nb_3dayplots starting 14.6 seconds late 2017-06-01 22:23:46,803 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-06-01 22:23:46,803 INFO repeater task os150nb_3dayplots ended after 6.4 seconds 2017-06-01 22:23:46,803 INFO repeater task os150nb_tsstats starting 21.0 seconds late 2017-06-01 22:23:47,354 INFO repeater task os150nb_tsstats ended after 0.6 seconds 2017-06-01 22:23:47,354 INFO repeater task os150bb_beamplots starting 21.6 seconds late 2017-06-01 22:23:48,259 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 22:23:48,259 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 22:23:48,260 INFO repeater task os150bb_beamplots starting 22.5 seconds late 2017-06-01 22:23:49,172 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 22:23:49,173 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 22:23:49,173 INFO repeater task os150nb_beamplots starting 23.4 seconds late 2017-06-01 22:23:50,074 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 22:23:50,074 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 22:23:50,074 INFO repeater task os150nb_beamplots starting 24.3 seconds late 2017-06-01 22:23:50,989 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 22:23:50,989 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 22:23:50,990 INFO repeater task posmv_hcorrstats starting 25.2 seconds late 2017-06-01 22:23:51,820 INFO repeater task posmv_hcorrstats ended after 0.8 seconds 2017-06-01 22:23:51,820 INFO repeater task posmv_plotposmv starting 26.0 seconds late 2017-06-01 22:23:52,448 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 22:23:57,454 INFO repeater task os150bb_lastensq starting 21.5 seconds late 2017-06-01 22:23:59,069 INFO repeater task os150bb_lastensq ended after 1.6 seconds 2017-06-01 22:23:59,069 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 22:23:26 2017-06-01 22:24:49,167 INFO repeater task os150bb_10pings starting 3.4 seconds late 2017-06-01 22:24:49,326 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 22:24:49,327 INFO repeater task os150nb_10pings starting 3.5 seconds late 2017-06-01 22:24:49,420 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 22:28:29,913 INFO repeater task posmv_hcorrstats starting 4.1 seconds late 2017-06-01 22:28:30,840 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 22:28:30,840 INFO repeater task posmv_plotposmv starting 5.1 seconds late 2017-06-01 22:28:31,444 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 22:28:36,450 INFO repeater task os150bb_lastensq starting 0.5 seconds late 2017-06-01 22:28:38,208 INFO repeater task os150bb_lastensq ended after 1.8 seconds 2017-06-01 22:28:38,208 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 22:28:26 2017-06-01 22:28:38,209 INFO repeater task os150nb_lastensq starting 2.2 seconds late 2017-06-01 22:28:39,403 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 22:28:39,403 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 22:28:26 2017-06-01 22:30:09,538 INFO repeater task os150bb_10pings starting 3.8 seconds late 2017-06-01 22:30:09,696 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 22:30:09,696 INFO repeater task os150nb_10pings starting 3.9 seconds late 2017-06-01 22:30:09,791 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 22:33:30,156 INFO repeater task posmv_hcorrstats starting 4.4 seconds late 2017-06-01 22:33:31,054 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 22:33:31,055 INFO repeater task posmv_plotposmv starting 5.3 seconds late 2017-06-01 22:33:31,662 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 22:33:36,667 INFO repeater task os150bb_lastensq starting 0.7 seconds late 2017-06-01 22:33:38,345 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 22:33:38,346 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 22:33:26 2017-06-01 22:33:38,346 INFO repeater task os150nb_lastensq starting 2.3 seconds late 2017-06-01 22:33:39,485 INFO repeater task os150nb_lastensq ended after 1.1 seconds 2017-06-01 22:33:39,485 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 22:33:26 2017-06-01 22:35:29,717 INFO repeater task os150bb_10pings starting 3.9 seconds late 2017-06-01 22:35:29,878 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 22:35:29,879 INFO repeater task os150nb_10pings starting 4.1 seconds late 2017-06-01 22:35:29,970 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 22:38:30,324 INFO repeater task os150bb_quick starting 4.5 seconds late 2017-06-01 22:38:33,610 INFO repeater task os150bb_quick ended after 3.3 seconds 2017-06-01 22:38:33,610 INFO repeater task os150bb_3dayplots starting 7.8 seconds late 2017-06-01 22:38:39,864 INFO repeater task os150bb_3dayplots ended after 6.3 seconds 2017-06-01 22:38:39,865 INFO repeater task os150bb_tsstats starting 14.1 seconds late 2017-06-01 22:38:40,435 INFO repeater task os150bb_tsstats ended after 0.6 seconds 2017-06-01 22:38:40,435 INFO repeater task os150nb_lastensq starting 4.4 seconds late 2017-06-01 22:38:41,596 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 22:38:41,596 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 22:38:26 2017-06-01 22:38:41,597 INFO repeater task os150nb_quick starting 15.8 seconds late 2017-06-01 22:38:44,691 INFO repeater task os150nb_quick ended after 3.1 seconds 2017-06-01 22:38:44,692 INFO repeater task os150nb_3dayplots starting 18.9 seconds late 2017-06-01 22:38:51,015 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-06-01 22:38:51,015 INFO repeater task os150nb_3dayplots ended after 6.3 seconds 2017-06-01 22:38:51,015 INFO repeater task os150nb_tsstats starting 25.2 seconds late 2017-06-01 22:38:51,567 INFO repeater task os150nb_tsstats ended after 0.6 seconds 2017-06-01 22:38:51,568 INFO repeater task os150bb_beamplots starting 25.8 seconds late 2017-06-01 22:38:52,485 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 22:38:52,485 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 22:38:52,486 INFO repeater task os150bb_beamplots starting 26.7 seconds late 2017-06-01 22:38:53,413 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 22:38:53,413 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 22:38:53,413 INFO repeater task os150nb_beamplots starting 27.6 seconds late 2017-06-01 22:38:54,365 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 22:38:54,365 INFO repeater task os150nb_beamplots ended after 1.0 seconds 2017-06-01 22:38:54,365 INFO repeater task os150nb_beamplots starting 28.6 seconds late 2017-06-01 22:38:55,286 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 22:38:55,286 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 22:38:55,286 INFO repeater task posmv_hcorrstats starting 29.5 seconds late 2017-06-01 22:38:56,102 INFO repeater task posmv_hcorrstats ended after 0.8 seconds 2017-06-01 22:38:56,103 INFO repeater task posmv_plotposmv starting 30.3 seconds late 2017-06-01 22:38:56,716 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 22:39:01,720 INFO repeater task os150bb_lastensq starting 25.7 seconds late 2017-06-01 22:39:03,317 INFO repeater task os150bb_lastensq ended after 1.6 seconds 2017-06-01 22:39:03,317 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 22:38:26 2017-06-01 22:40:48,534 INFO repeater task os150bb_10pings starting 2.8 seconds late 2017-06-01 22:40:48,703 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 22:40:48,704 INFO repeater task os150nb_10pings starting 2.9 seconds late 2017-06-01 22:40:48,797 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 22:43:29,122 INFO repeater task posmv_hcorrstats starting 3.3 seconds late 2017-06-01 22:43:30,006 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 22:43:30,006 INFO repeater task posmv_plotposmv starting 4.2 seconds late 2017-06-01 22:43:30,652 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 22:43:40,665 INFO repeater task os150bb_lastensq starting 4.7 seconds late 2017-06-01 22:43:42,457 INFO repeater task os150bb_lastensq ended after 1.8 seconds 2017-06-01 22:43:42,457 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 22:43:26 2017-06-01 22:43:42,458 INFO repeater task os150nb_lastensq starting 6.5 seconds late 2017-06-01 22:43:43,675 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 22:43:43,676 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 22:43:26 2017-06-01 22:46:08,924 INFO repeater task os150bb_10pings starting 3.1 seconds late 2017-06-01 22:46:09,020 INFO repeater task os150bb_10pings ended after 0.1 seconds 2017-06-01 22:46:09,020 INFO repeater task os150nb_10pings starting 3.2 seconds late 2017-06-01 22:46:09,116 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 22:48:29,325 INFO repeater task posmv_hcorrstats starting 3.5 seconds late 2017-06-01 22:48:30,198 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 22:48:30,198 INFO repeater task posmv_plotposmv starting 4.4 seconds late 2017-06-01 22:48:30,806 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 22:48:40,824 INFO repeater task os150bb_lastensq starting 4.8 seconds late 2017-06-01 22:48:42,460 INFO repeater task os150bb_lastensq ended after 1.6 seconds 2017-06-01 22:48:42,460 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 22:48:26 2017-06-01 22:48:42,460 INFO repeater task os150nb_lastensq starting 6.5 seconds late 2017-06-01 22:48:43,586 INFO repeater task os150nb_lastensq ended after 1.1 seconds 2017-06-01 22:48:43,586 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 22:48:26 2017-06-01 22:51:28,912 INFO repeater task os150bb_10pings starting 3.1 seconds late 2017-06-01 22:51:29,054 INFO repeater task os150bb_10pings ended after 0.1 seconds 2017-06-01 22:51:29,055 INFO repeater task os150nb_10pings starting 3.3 seconds late 2017-06-01 22:51:29,148 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 22:53:29,405 INFO repeater task os150bb_quick starting 3.6 seconds late 2017-06-01 22:53:32,540 INFO repeater task os150bb_quick ended after 3.1 seconds 2017-06-01 22:53:32,540 INFO repeater task os150bb_3dayplots starting 6.8 seconds late 2017-06-01 22:53:38,795 INFO repeater task os150bb_3dayplots ended after 6.3 seconds 2017-06-01 22:53:38,795 INFO repeater task os150bb_tsstats starting 13.0 seconds late 2017-06-01 22:53:39,338 INFO repeater task os150bb_tsstats ended after 0.5 seconds 2017-06-01 22:53:39,338 INFO repeater task os150nb_lastensq starting 3.3 seconds late 2017-06-01 22:53:40,480 INFO repeater task os150nb_lastensq ended after 1.1 seconds 2017-06-01 22:53:40,481 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 22:53:26 2017-06-01 22:53:40,481 INFO repeater task os150nb_quick starting 14.7 seconds late 2017-06-01 22:53:43,534 INFO repeater task os150nb_quick ended after 3.1 seconds 2017-06-01 22:53:43,534 INFO repeater task os150nb_3dayplots starting 17.8 seconds late 2017-06-01 22:53:49,854 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-06-01 22:53:49,855 INFO repeater task os150nb_3dayplots ended after 6.3 seconds 2017-06-01 22:53:49,855 INFO repeater task os150nb_tsstats starting 24.1 seconds late 2017-06-01 22:53:50,398 INFO repeater task os150nb_tsstats ended after 0.5 seconds 2017-06-01 22:53:50,398 INFO repeater task os150bb_beamplots starting 24.6 seconds late 2017-06-01 22:53:51,376 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 22:53:51,376 INFO repeater task os150bb_beamplots ended after 1.0 seconds 2017-06-01 22:53:51,376 INFO repeater task os150bb_beamplots starting 25.6 seconds late 2017-06-01 22:53:52,324 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 22:53:52,324 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 22:53:52,325 INFO repeater task os150nb_beamplots starting 26.5 seconds late 2017-06-01 22:53:53,252 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 22:53:53,252 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 22:53:53,252 INFO repeater task os150nb_beamplots starting 27.5 seconds late 2017-06-01 22:53:54,158 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 22:53:54,158 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 22:53:54,159 INFO repeater task posmv_hcorrstats starting 28.4 seconds late 2017-06-01 22:53:54,989 INFO repeater task posmv_hcorrstats ended after 0.8 seconds 2017-06-01 22:53:54,990 INFO repeater task posmv_plotposmv starting 29.2 seconds late 2017-06-01 22:53:55,599 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 22:54:00,605 INFO repeater task os150bb_lastensq starting 24.6 seconds late 2017-06-01 22:54:02,378 INFO repeater task os150bb_lastensq ended after 1.8 seconds 2017-06-01 22:54:02,378 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 22:53:26 2017-06-01 22:56:47,711 INFO repeater task os150bb_10pings starting 1.9 seconds late 2017-06-01 22:56:47,873 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 22:56:47,874 INFO repeater task os150nb_10pings starting 2.1 seconds late 2017-06-01 22:56:47,966 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 22:58:28,152 INFO repeater task posmv_hcorrstats starting 2.4 seconds late 2017-06-01 22:58:29,162 INFO repeater task posmv_hcorrstats ended after 1.0 seconds 2017-06-01 22:58:29,162 INFO repeater task posmv_plotposmv starting 3.4 seconds late 2017-06-01 22:58:29,797 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 22:58:39,815 INFO repeater task os150bb_lastensq starting 3.8 seconds late 2017-06-01 22:58:41,587 INFO repeater task os150bb_lastensq ended after 1.8 seconds 2017-06-01 22:58:41,587 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 22:58:26 2017-06-01 22:58:41,588 INFO repeater task os150nb_lastensq starting 5.6 seconds late 2017-06-01 22:58:42,806 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 22:58:42,806 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 22:58:26 2017-06-01 23:02:08,158 INFO repeater task os150bb_10pings starting 2.4 seconds late 2017-06-01 23:02:08,253 INFO repeater task os150bb_10pings ended after 0.1 seconds 2017-06-01 23:02:08,254 INFO repeater task os150nb_10pings starting 2.5 seconds late 2017-06-01 23:02:08,351 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 23:03:28,540 INFO repeater task posmv_hcorrstats starting 2.8 seconds late 2017-06-01 23:03:29,503 INFO repeater task posmv_hcorrstats ended after 1.0 seconds 2017-06-01 23:03:29,503 INFO repeater task posmv_plotposmv starting 3.7 seconds late 2017-06-01 23:03:30,137 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 23:03:40,150 INFO repeater task os150bb_lastensq starting 4.2 seconds late 2017-06-01 23:03:41,977 INFO repeater task os150bb_lastensq ended after 1.8 seconds 2017-06-01 23:03:41,977 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 23:03:26 2017-06-01 23:03:41,978 INFO repeater task os150nb_lastensq starting 6.0 seconds late 2017-06-01 23:03:43,226 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 23:03:43,226 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 23:03:26 2017-06-01 23:07:28,687 INFO repeater task os150bb_10pings starting 2.9 seconds late 2017-06-01 23:07:28,860 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 23:07:28,861 INFO repeater task os150nb_10pings starting 3.1 seconds late 2017-06-01 23:07:28,959 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 23:08:29,108 INFO repeater task os150bb_quick starting 3.3 seconds late 2017-06-01 23:08:32,351 INFO repeater task os150bb_quick ended after 3.2 seconds 2017-06-01 23:08:32,352 INFO repeater task os150bb_3dayplots starting 6.6 seconds late 2017-06-01 23:08:39,184 INFO repeater task os150bb_3dayplots ended after 6.8 seconds 2017-06-01 23:08:39,184 INFO repeater task os150bb_tsstats starting 13.4 seconds late 2017-06-01 23:08:39,779 INFO repeater task os150bb_tsstats ended after 0.6 seconds 2017-06-01 23:08:39,779 INFO repeater task os150nb_lastensq starting 3.8 seconds late 2017-06-01 23:08:41,017 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 23:08:41,018 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 23:08:26 2017-06-01 23:08:41,018 INFO repeater task os150nb_quick starting 15.2 seconds late 2017-06-01 23:08:44,274 INFO repeater task os150nb_quick ended after 3.3 seconds 2017-06-01 23:08:44,275 INFO repeater task os150nb_3dayplots starting 18.5 seconds late 2017-06-01 23:08:51,261 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-06-01 23:08:51,261 INFO repeater task os150nb_3dayplots ended after 7.0 seconds 2017-06-01 23:08:51,261 INFO repeater task os150nb_tsstats starting 25.5 seconds late 2017-06-01 23:08:51,807 INFO repeater task os150nb_tsstats ended after 0.5 seconds 2017-06-01 23:08:51,807 INFO repeater task os150bb_beamplots starting 26.0 seconds late 2017-06-01 23:08:52,788 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 23:08:52,788 INFO repeater task os150bb_beamplots ended after 1.0 seconds 2017-06-01 23:08:52,788 INFO repeater task os150bb_beamplots starting 27.0 seconds late 2017-06-01 23:08:53,733 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 23:08:53,733 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 23:08:53,734 INFO repeater task os150nb_beamplots starting 28.0 seconds late 2017-06-01 23:08:54,630 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 23:08:54,630 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 23:08:54,630 INFO repeater task os150nb_beamplots starting 28.8 seconds late 2017-06-01 23:08:55,554 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 23:08:55,554 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 23:08:55,555 INFO repeater task posmv_hcorrstats starting 29.8 seconds late 2017-06-01 23:08:56,420 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 23:08:56,421 INFO repeater task posmv_plotposmv starting 30.6 seconds late 2017-06-01 23:08:57,088 INFO repeater task posmv_plotposmv ended after 0.7 seconds 2017-06-01 23:09:02,093 INFO repeater task os150bb_lastensq starting 26.1 seconds late 2017-06-01 23:09:03,950 INFO repeater task os150bb_lastensq ended after 1.9 seconds 2017-06-01 23:09:03,950 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 23:08:26 2017-06-01 23:12:49,435 INFO repeater task os150bb_10pings starting 3.7 seconds late 2017-06-01 23:12:49,527 INFO repeater task os150bb_10pings ended after 0.1 seconds 2017-06-01 23:12:49,528 INFO repeater task os150nb_10pings starting 3.7 seconds late 2017-06-01 23:12:49,619 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 23:13:29,707 INFO repeater task posmv_hcorrstats starting 3.9 seconds late 2017-06-01 23:13:30,636 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 23:13:30,637 INFO repeater task posmv_plotposmv starting 4.9 seconds late 2017-06-01 23:13:31,296 INFO repeater task posmv_plotposmv ended after 0.7 seconds 2017-06-01 23:13:36,302 INFO repeater task os150bb_lastensq starting 0.3 seconds late 2017-06-01 23:13:38,101 INFO repeater task os150bb_lastensq ended after 1.8 seconds 2017-06-01 23:13:38,101 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 23:13:26 2017-06-01 23:13:38,102 INFO repeater task os150nb_lastensq starting 2.1 seconds late 2017-06-01 23:13:39,286 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 23:13:39,286 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 23:13:26 2017-06-01 23:18:09,846 INFO repeater task os150bb_10pings starting 4.1 seconds late 2017-06-01 23:18:09,942 INFO repeater task os150bb_10pings ended after 0.1 seconds 2017-06-01 23:18:09,942 INFO repeater task os150nb_10pings starting 4.2 seconds late 2017-06-01 23:18:10,037 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 23:18:30,075 INFO repeater task posmv_hcorrstats starting 4.3 seconds late 2017-06-01 23:18:31,050 INFO repeater task posmv_hcorrstats ended after 1.0 seconds 2017-06-01 23:18:31,051 INFO repeater task posmv_plotposmv starting 5.3 seconds late 2017-06-01 23:18:31,677 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 23:18:36,683 INFO repeater task os150bb_lastensq starting 0.7 seconds late 2017-06-01 23:18:38,415 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 23:18:38,415 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 23:18:26 2017-06-01 23:18:38,416 INFO repeater task os150nb_lastensq starting 2.4 seconds late 2017-06-01 23:18:39,691 INFO repeater task os150nb_lastensq ended after 1.3 seconds 2017-06-01 23:18:39,691 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 23:18:26 2017-06-01 23:23:30,311 INFO repeater task os150bb_10pings starting 4.5 seconds late 2017-06-01 23:23:30,480 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 23:23:30,481 INFO repeater task os150bb_quick starting 4.7 seconds late 2017-06-01 23:23:33,535 INFO repeater task os150bb_quick ended after 3.1 seconds 2017-06-01 23:23:33,535 INFO repeater task os150bb_3dayplots starting 7.8 seconds late 2017-06-01 23:23:39,824 INFO repeater task os150bb_3dayplots ended after 6.3 seconds 2017-06-01 23:23:39,825 INFO repeater task os150bb_tsstats starting 14.0 seconds late 2017-06-01 23:23:40,368 INFO repeater task os150bb_tsstats ended after 0.5 seconds 2017-06-01 23:23:40,368 INFO repeater task os150nb_lastensq starting 4.4 seconds late 2017-06-01 23:23:41,542 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 23:23:41,542 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 23:23:26 2017-06-01 23:23:41,542 INFO repeater task os150nb_10pings starting 15.8 seconds late 2017-06-01 23:23:41,635 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 23:23:41,635 INFO repeater task os150nb_quick starting 15.9 seconds late 2017-06-01 23:23:44,721 INFO repeater task os150nb_quick ended after 3.1 seconds 2017-06-01 23:23:44,721 INFO repeater task os150nb_3dayplots starting 18.9 seconds late 2017-06-01 23:23:51,085 INFO repeater task os150nb_3dayplots output: /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:319: RuntimeWarning: All-NaN slice encountered warnings.warn("All-NaN slice encountered", RuntimeWarning) /usr/lib/python2.7/dist-packages/numpy/lib/nanfunctions.py:220: RuntimeWarning: All-NaN axis encountered warnings.warn("All-NaN axis encountered", RuntimeWarning) 2017-06-01 23:23:51,085 INFO repeater task os150nb_3dayplots ended after 6.4 seconds 2017-06-01 23:23:51,085 INFO repeater task os150nb_tsstats starting 25.3 seconds late 2017-06-01 23:23:51,628 INFO repeater task os150nb_tsstats ended after 0.5 seconds 2017-06-01 23:23:51,628 INFO repeater task os150bb_beamplots starting 25.8 seconds late 2017-06-01 23:23:52,538 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 23:23:52,538 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 23:23:52,539 INFO repeater task os150bb_beamplots starting 26.8 seconds late 2017-06-01 23:23:53,464 INFO repeater task os150bb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 23:23:53,464 INFO repeater task os150bb_beamplots ended after 0.9 seconds 2017-06-01 23:23:53,464 INFO repeater task os150nb_beamplots starting 27.7 seconds late 2017-06-01 23:23:54,379 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 23:23:54,379 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 23:23:54,379 INFO repeater task os150nb_beamplots starting 28.6 seconds late 2017-06-01 23:23:55,321 INFO repeater task os150nb_beamplots output: Starting procsetup scriptpath is /usr/local/currents/bin 2017-06-01 23:23:55,321 INFO repeater task os150nb_beamplots ended after 0.9 seconds 2017-06-01 23:23:55,321 INFO repeater task posmv_hcorrstats starting 29.5 seconds late 2017-06-01 23:23:56,164 INFO repeater task posmv_hcorrstats ended after 0.8 seconds 2017-06-01 23:23:56,164 INFO repeater task posmv_plotposmv starting 30.4 seconds late 2017-06-01 23:23:56,762 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 23:24:01,768 INFO repeater task os150bb_lastensq starting 25.8 seconds late 2017-06-01 23:24:03,487 INFO repeater task os150bb_lastensq ended after 1.7 seconds 2017-06-01 23:24:03,488 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 23:23:26 2017-06-01 23:28:29,121 INFO repeater task posmv_hcorrstats starting 3.3 seconds late 2017-06-01 23:28:29,974 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 23:28:29,974 INFO repeater task posmv_plotposmv starting 4.2 seconds late 2017-06-01 23:28:30,614 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 23:28:40,627 INFO repeater task os150bb_lastensq starting 4.6 seconds late 2017-06-01 23:28:42,381 INFO repeater task os150bb_lastensq ended after 1.8 seconds 2017-06-01 23:28:42,381 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 23:28:26 2017-06-01 23:28:42,382 INFO repeater task os150nb_lastensq starting 6.4 seconds late 2017-06-01 23:28:43,653 INFO repeater task os150nb_lastensq ended after 1.3 seconds 2017-06-01 23:28:43,654 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 23:28:26 2017-06-01 23:28:48,661 INFO repeater task os150bb_10pings starting 2.9 seconds late 2017-06-01 23:28:48,821 INFO repeater task os150bb_10pings ended after 0.2 seconds 2017-06-01 23:28:48,821 INFO repeater task os150nb_10pings starting 3.0 seconds late 2017-06-01 23:28:48,916 INFO repeater task os150nb_10pings ended after 0.1 seconds 2017-06-01 23:33:29,538 INFO repeater task posmv_hcorrstats starting 3.8 seconds late 2017-06-01 23:33:30,461 INFO repeater task posmv_hcorrstats ended after 0.9 seconds 2017-06-01 23:33:30,462 INFO repeater task posmv_plotposmv starting 4.7 seconds late 2017-06-01 23:33:31,066 INFO repeater task posmv_plotposmv ended after 0.6 seconds 2017-06-01 23:33:36,072 INFO repeater task os150bb_lastensq starting 0.1 seconds late 2017-06-01 23:33:37,856 INFO repeater task os150bb_lastensq ended after 1.8 seconds 2017-06-01 23:33:37,856 INFO repeater task os150bb_lastensq: Time from /home/data/NF1704_B/proc/os150bb/load/wrote_ens_time: 2017/06/01 23:33:26 2017-06-01 23:33:37,857 INFO repeater task os150nb_lastensq starting 1.9 seconds late 2017-06-01 23:33:39,101 INFO repeater task os150nb_lastensq ended after 1.2 seconds 2017-06-01 23:33:39,102 INFO repeater task os150nb_lastensq: Time from /home/data/NF1704_B/proc/os150nb/load/wrote_ens_time: 2017/06/01 23:33:26 2017-06-01 23:34:09,161 INFO repeater task os150bb_10pings starting 3.4 seconds late 2017-06-01 23:34:09,263 INFO repeater task os150bb_10pings ended after 0.1 seconds 2017-06-01 23:34:09,264 INFO repeater task os150nb_10pings starting 3.5 seconds late