1-50 of 2033 results (13ms)
2019-08-17 §
10:56 <arturo> force-reboot tools-worker-1006. Is completely stuck
2019-08-15 §
15:32 <jeh> upgraded jobutils debian package to 1.38 T229551
09:22 <arturo> restart maintain-kubeusers service in tools-k8s-master-01 because some tools were missing their namespaces
2019-08-13 §
22:00 <bstorm_> truncated exim paniclog on tools-sgecron-01 because it was being spammy
13:41 <jeh> Set icingia downtime for toolschecker labs showmount T229448
2019-08-12 §
16:08 <phamhi> updated prometheus-node-exporter from 0.14.0~git20170523-1 to 0.17.0+ds-3 in tools-worker-[1030-1040] nodes (T230147)
2019-08-08 §
19:25 <jeh> restarting tools-sgewebgrid-lighttpd-0915 T230157
2019-08-07 §
19:07 <bd808> Disassociated SUL and Phabricator accounts from user Lophi (T229713)
2019-08-06 §
16:18 <arturo> add phamhi as user/projectadmin (T228942) and delete hpham
15:58 <arturo> add hpham as user/projectadmin (T228942)
13:43 <jeh> disabling puppet on tools-checker-03 while testing nginx timeouts T221301
2019-08-05 §
22:49 <bstorm_> launching tools-worker-1040
20:36 <andrewbogott> rebooting oom tools-worker-1026
16:10 <jeh> `tools-k8s-master-01: systemctl restart maintain-kubeusers` T229846
09:39 <arturo> `root@tools-checker-03:~# toolscheckerctl restart` again (T229787)
09:30 <arturo> `root@tools-checker-03:~# toolscheckerctl restart` (T229787)
2019-08-02 §
14:00 <andrewbogott_> rebooting tools-worker-1022 as it is unresponsive
2019-07-31 §
18:07 <bstorm_> drained tools-worker-1015/05/03/17 to rebalance load
17:41 <bstorm_> drained tools-worker-1025 and 1026 to rebalance load
17:32 <bstorm_> drained tools-worker-1028 to rebalance load
17:29 <bstorm_> drained tools-worker-1008 to rebalance load
17:23 <bstorm_> drained tools-worker-1021 to rebalance load
17:17 <bstorm_> drained tools-worker-1007 to rebalance load
17:07 <bstorm_> drained tools-worker-1004 to rebalance load
16:27 <andrewbogott> moving tools-static-12 to cloudvirt1018
15:33 <bstorm_> T228573 spinning up 5 worker nodes for kubernetes cluster (tools-worker-1035-9)
2019-07-27 §
23:00 <zhuyifei1999_> a past probably related ticket: T194859
22:57 <zhuyifei1999_> maintain-kubeusers seems stuck. Traceback: https://phabricator.wikimedia.org/P8812, core dump: /root/core.17898. Restarting
2019-07-26 §
17:39 <bstorm_> restarted maintain-kubeusers because it was suspiciously tardy and quiet
17:14 <bstorm_> drained tools-worker-1013.tools.eqiad.wmflabs to rebalance load
17:09 <bstorm_> draining tools-worker-1020.tools.eqiad.wmflabs to rebalance load
16:32 <bstorm_> created tools-worker-1034 - T228573
15:57 <bstorm_> created tools-worker-1032 and 1033 - T228573
15:54 <bstorm_> created tools-worker-1031 - T228573
2019-07-25 §
22:01 <bstorm_> T228573 created tools-worker-1030
21:22 <jeh> rebooting tools-worker-1016 unresponsive
2019-07-24 §
10:14 <arturo> reallocating tools-puppetmaster-01 from cloudvirt1027 to cloudvirt1028 (T227539)
10:12 <arturo> reallocating tools-docker-registry-04 from cloudvirt1027 to cloudvirt1028 (T227539)
2019-07-22 §
18:39 <bstorm_> repooled tools-sgeexec-0905 after reboot
18:33 <bstorm_> depooled tools-sgeexec-0905 because it's acting kind of weird and not responding to prometheus
18:32 <bstorm_> repooled tools-sgewebgrid-lighttpd-0902 after restarting the grid-exec service
18:28 <bstorm_> depooled tools-sgewebgrid-lighttpd-0902 to find out why it is behaving weird
17:55 <bstorm_> draining tools-worker-1023 since it is having issues
17:38 <bstorm_> Adding the prometheus servers to the ferm rules via wikitech hiera for kubelet stats T228573
2019-07-20 §
19:52 <andrewbogott> rebooting tools-worker-1023
2019-07-17 §
20:23 <andrewbogott> migrating tools-sgegrid-shadow to cloudvirt1014
2019-07-15 §
14:50 <bstorm_> cleared error state from tools-sgeexec-0911 which went offline after error from job 5190035
2019-06-25 §
09:30 <arturo> detected puppet issue in all VMs: T226480
2019-06-24 §
17:42 <andrewbogott> moving tools-sgeexec-0905 to cloudvirt1015
2019-06-17 §
14:07 <andrewbogott> moving tools-sgewebgrid-lighttpd-0903 to cloudvirt1015