1-50 of 963 results (15ms)
2017-11-20 §
20:34<chasemp>backup crons tools-cron-01:/var/spool/cron# cp -Rp crontabs/ /root/20112017/
00:52<andrewbogott>cherry-picking https://gerrit.wikimedia.org/r/#/c/392172/ onto the tools puppetmaster
2017-11-17 §
21:33<valhallasw`cloud>also g-w'ed those files, and sent emails to all the affected users
21:17<valhallasw`cloud>chmod o-w'ed a bunch of files reported by Dispenser; writing emails to the owners about this
2017-11-16 §
17:40<chasemp>tools-clushmaster-01:~$ clush -w @all 'sudo puppet agent --enable && sudo puppet agent --test && sudo unattended-upgrades -d'
16:50<bd808>Force upgraded nginx on tools-elastic-*
16:37<chasemp>reboot tools-checker-01
15:17<chasemp>disable puppet
2017-11-15 §
22:48<madhuvishy>Rebooted tools-paws-worker-1017
15:53<chasemp>reboot bastion-03
15:48<chasemp>kill tools.powow on bastion-03 for hammering IO and making bastion unusable
2017-11-07 §
01:21<bd808>Removed all non-directory files from /home (via labstore1004 direct access)
2017-11-06 §
18:30<bd808>Load on tools-bastion-03 down to 0.72 from 17.47 after killing a bunch of local processes that should have been running on the job grid instead
2017-11-05 §
23:48<bd808>Cleaned up 2 huge /tmp files left by tools.croptool (~6.5G)
23:44<bd808>Cleaned up 109 files owned by tools.rezabot on tools-webgrid-lighttpd-1428 with `sudo find /tmp -user tools.rezabot -exec rm {} \+`
23:37<bd808>Cleaned up 955 files owned by tools.wsexport on tools-webgrid-lighttpd-1428 with `sudo find /tmp -user tools.wsexport -exec rm {} \+`
2017-11-03 §
21:19<bd808>Deployed misctools 1.26 (T156174)
2017-11-02 §
16:15<bd808>Restarted nslcd on tools-bastion-03
2017-11-01 §
07:11<madhuvishy>Clear nscd cache across all projects post labsdb dns switchover T179464
07:11<madhuvishy>Clear nscd cache across all projects post labsdb dns switchover
2017-10-31 §
16:50<bd808>tools-bastion-03 (tools-login, login.tools) is overloaded
2017-10-30 §
17:35<madhuvishy>Clear dns caches across tools hosts `sudo nscd -i hosts`
16:08<arturo>depool tools-exec-1401.tools.eqiad.wmflabs
2017-10-24 §
18:09<madhuvishy>Disable puppet on tools-package-builder-01 temporarily (T178920)
13:22<chasemp>start admin webservice
13:22<chasemp>stop admin webservice
2017-10-23 §
14:49<chasemp>wall message and scheduled reboot in 5m for bastion-03
2017-10-18 §
21:36<chasemp>stop basebot -- it is going crazy and spamming email w/ failing to log to error.log. Need to figure out how to notify but it's clearly in a failure loop.
14:04<chasemp>add strephit creds to elasticsearch per T178310
2017-10-12 §
16:57<bd808>Rebuilding all Kubernetes Docker images to include toollabs-webservice 0.38
16:53<bd808>Upgraded toollabs-webservice to 0.38
2017-10-06 §
15:33<bd808>Upgrade jobutils to 1.25 (T177614)
00:27<bd808>Updated misctools to 1.24
2017-10-05 §
22:47<bd808>Updated misctools to 1.23
22:42<bd808>Updated jobutils to 1.23
15:46<chasemp>tools-bastion-03 has tons of local tools running long lived NFS intensive processes. I'm rebooting rather than playing whackamole.
2017-10-03 §
19:30<bd808>`kubectl --namespace=prod delete pod --all` on tools-paws-master-01
2017-10-01 §
21:46<madhuvishy>Cold migrating tools-clushmaster-01 from labvirt1015 to labvirt1017
2017-09-29 §
19:49<andrewbogott>migration tools-clushmaster-01 to labvirt1015
2017-09-25 §
15:14<andrewbogott>rebooting tools-paws-worker-1006 since I can't access it
14:57<chasemp>OS_TENANT_NAME=tools openstack server reboot 2c0cf363-c7c3-42ad-94bd-e586f2492321 (unresponsive)
2017-09-20 §
16:52<madhuvishy>apt-get install --only-upgrade apache2; service apache2 restart on tools-puppetmaster-01
2017-09-19 §
15:22<chasemp>tools-clushmaster-01:~$ clush -f 5 -g all 'sudo puppet agent --test'
13:39<chasemp>bastion-03 someone dropped 8.6G in /tmp which is /not/ seemingly on a temp file system
13:25<chasemp>wall Bastion disk is full and needs attention and reboot in 60
2017-09-18 §
18:02<bd808>Updated PHP5.6 images for Kubernetes (T172358)
2017-09-13 §
15:34<bd808>Running inbound message purge via clush to @tools-exec
15:15<bd808>Running outbound message purge via clush to @tools-exec
13:57<bd808>apt-get install nginx-common on tools-static-1[01]
13:31<bd808>static down due to apparent nginx package upgrade/config change