151-200 of 1735 results (11ms)
2019-02-22 §
15:13 <gtirloni> shutdown tools-puppetmaster-01
2019-02-21 §
09:59 <gtirloni> upgraded all packages in all stretch nodes
00:12 <zhuyifei1999_> forcing puppet run on tools-k8s-master-01
00:08 <zhuyifei1999_> running /usr/local/bin/git-sync-upstream on tools-puppetmaster-01 to speed puppet changes up
2019-02-20 §
23:30 <zhuyifei1999_> begin rebuilding all docker images T178601 T193646 T215683
23:25 <zhuyifei1999_> upgraded toollabs-webservice on tools-bastion-02 to 0.44 (newly-built version)
23:19 <zhuyifei1999_> this was built for stretch. hopefully it works for all distros
23:17 <zhuyifei1999_> begin build new tools-webservice package T178601 T193646 T215683
21:57 <andrewbogott> moving tools-static-13 to a new virt host
21:34 <andrewbogott> moving the tools-static IP from tools-static-13 to tools-static-12
19:17 <andrewbogott> moving tools-bastion-02 to labvirt1004
16:56 <andrewbogott> moving tools-paws-worker-1003
15:53 <andrewbogott> moving tools-worker-1017, tools-worker-1027, tools-worker-1028
15:03 <andrewbogott> moving tools-exec-1413 and tools-exec-1442
2019-02-19 §
01:49 <bd808> Revoked Toolforge project membership for user DannyS712 (T215092)
2019-02-18 §
20:45 <gtirloni> upgraded and rebooted tools-sgebastion-07 (login-stretch)
20:22 <gtirloni> enabled toolsdb monitoring in Icinga
20:03 <gtirloni> pointed tools-db.eqiad.wmflabs to 172.16.7.153
18:50 <chicocvenancio> moving paws back to toolsdb T216208
13:47 <arturo> rebooting tools-sgebastion-07 to try fixing general slowness
2019-02-17 §
22:23 <zhuyifei1999_> uncordon tools-worker-1010.tools.eqiad.wmflabs
22:11 <zhuyifei1999_> rebooting tools-worker-1010.tools.eqiad.wmflabs
22:10 <zhuyifei1999_> draining tools-worker-1010.tools.eqiad.wmflabs, `docker ps` is hanging. no idea why. also other weirdness like ContainerCreating forever
2019-02-16 §
05:00 <zhuyifei1999_> fixed by restarting flannel. another puppet run simply started kubelet
04:58 <zhuyifei1999_> puppet logs: https://phabricator.wikimedia.org/P8097. Docker is failing with 'Failed to load environment files: No such file or directory'
04:52 <zhuyifei1999_> copied the resolv.conf from tools-k8s-master-01, removing secondary DNS to make sure puppet fixes that, and starting puppet
04:48 <zhuyifei1999_> that host's resolv.conf is badly broken https://phabricator.wikimedia.org/P8096. The last Puppet run was at Thu Feb 14 15:21:09 UTC 2019 (2247 minutes ago)
04:44 <zhuyifei1999_> puppet is also failing bad here 'Error: Could not request certificate: getaddrinfo: Name or service not known'
04:43 <zhuyifei1999_> this one has logs full of 'Can't contact LDAP server'
04:41 <zhuyifei1999_> nslcd also broken on tools-worker-1005
04:34 <zhuyifei1999_> uncordon tools-worker-1014.tools.eqiad.wmflabs
04:33 <zhuyifei1999_> the issue was, /var/run/nslcd/socket was somehow a directory, AFAICT
04:31 <zhuyifei1999_> then started nslcd vis systemctl and `id zhuyifei1999` returns correct stuffs
04:30 <zhuyifei1999_> `nslcd -nd` complains about 'nslcd: bind() to /var/run/nslcd/socket failed: Address already in use'. SIGTERMed a background nslcd, `rmdir /var/run/nslcd/socket`, and `nslcd -nd` seemingly starts to work
04:23 <zhuyifei1999_> drained tools-worker-1014.tools.eqiad.wmflabs
04:16 <zhuyifei1999_> logs: https://phabricator.wikimedia.org/P8095
04:14 <zhuyifei1999_> restarting nslcd on tools-worker-1014 in an attempt to fix that, service failed to start, looking into logs
04:12 <zhuyifei1999_> restarting nscd on tools-worker-1014 in an attempt to fix seemingly-not-attached-to-LDAP
2019-02-14 §
21:57 <bd808> Deleted old tools-proxy-02 instance
21:57 <bd808> Deleted old tools-proxy-01 instance
21:56 <bd808> Deleted old tools-package-builder-01 instance
20:57 <andrewbogott> rebooting tools-worker-1005
20:34 <andrewbogott> moving tools-exec-1409, tools-exec-1410, tools-exec-1414, tools-exec-1419
19:55 <andrewbogott> moving tools-webgrid-generic-1401 and tools-webgrid-lighttpd-1419
19:33 <andrewbogott> moving tools-checker-01 to labvirt1003
19:25 <andrewbogott> moving tools-elastic-02 to labvirt1003
19:11 <andrewbogott> moving tools-k8s-etcd-01 to labvirt1002
18:37 <andrewbogott> moving tools-exec-1418, tools-exec-1424 to labvirt1003
18:34 <andrewbogott> moving tools-webgrid-lighttpd-1404, tools-webgrid-lighttpd-1406, tools-webgrid-lighttpd-1410 to labvirt1002
17:35 <arturo> T215154 tools-sgebastion-07 now running systemd 239 and starts enforcing user limits