1-50 of 89798 results (84ms)
2018-01-16 §
10:13<godog>start cassandra-a on restbase1018 - T184100
09:56<moritzm>upgrading canary app servers to HHVM 3.18.7
09:50<marostegui>Stop replication in sync db1089 - db1105:3311 - T162807
09:43<marostegui@tin>Synchronized wmf-config/db-eqiad.php: Depool db1105:3311 - T162807 (duration: 01m 12s)
09:38<_joe_>started refreshLinks additional jobs for commonswiki,ruwiki
09:30<oblivian@neodymium>conftool action : set/weight=10; selector: cluster=api_appserver,name=mw13(39|4[12]).eqiad.wmnet
09:24<marostegui@tin>Synchronized wmf-config/db-eqiad.php: Repool db1099:3311 - T162807 (duration: 01m 12s)
09:24<oblivian@neodymium>conftool action : set/pooled=yes:weight=1; selector: cluster=api_appserver,name=mw13(39|4[12]).eqiad.wmnet
09:16<moritzm>installing libxml2 security updates on mw* servers (so that it gets picked up along the HHVM 3.18.7 rollout)
09:12<moritzm>installing krb5 security updates (we're just using rev deps)
09:08<jynus>upgrade and reboot db1031 after switchover
08:49<moritzm>rearmed key holder on sarin
08:45<moritzm>rebooting sarin for kernel security update
08:38<marostegui>Stop replication in sync db1089 - db1099:3311 - T162807
08:36<marostegui@tin>Synchronized wmf-config/db-eqiad.php: Repool db1066, depool db1099:3311 - T162807 (duration: 01m 12s)
08:28<jynus>master x1 eqiad failover has finished
08:19<jynus@tin>Synchronized wmf-config/db-eqiad.php: Promote db1055 as the new x1 master (duration: 00m 49s)
08:17<jynus>setting db1031 (x1 master) as read only
08:11<jynus>start x1 eqiad master failover
08:02<jynus@tin>Synchronized wmf-config/db-eqiad.php: Repool db1055 and db1056 after maintenance (duration: 00m 49s)
07:42<jynus>moving replication topology of x1 replicas
07:34<marostegui>Deploy schema change on dbstore1001 (s8) - T174569
07:30<marostegui>Stop replication in sync db1066 and db1089 - T162807
07:30<jynus>upgrade and reboot db1056
07:29<marostegui@tin>Synchronized wmf-config/db-eqiad.php: Depool db1066 and db1089 - T162807 (duration: 01m 13s)
07:29<oblivian@neodymium>conftool action : set/weight=25; selector: name=mw1340.eqiad.wmnet
07:17<jynus>upgrade and reboot db1055
07:14<jynus@tin>Synchronized wmf-config/db-eqiad.php: Depool db1055 and db1056 for maintenance (duration: 01m 12s)
07:03<marostegui>Deploy schema change on dbstore1002 (s8) - T174569
06:32<marostegui>Deploy schema change on db1092 - T174569
06:24<marostegui>Upgrade kernel on db1092
06:23<marostegui@tin>Synchronized wmf-config/db-eqiad.php: Depool db1092 - T174569 (duration: 01m 32s)
02:24<l10nupdate@tin>scap sync-l10n completed (1.31.0-wmf.15) (duration: 06m 10s)
2018-01-15 §
23:40<demon@tin>Synchronized wmf-config/InitialiseSettings.php: turn educationprogram back on for cs.wikipedia -- turns out there was no consensus and a patch should never have been written 😡 (duration: 01m 13s)
18:50<_joe_>pooled mw1340 as an api appserver
18:43<oblivian@puppetmaster1001>conftool action : set/pooled=active; selector: name=mw1338.eqiad.wmnet
18:42<oblivian@neodymium>conftool action : set/pooled=active; selector: name=mw1338.eqiad.wmnet
18:34<oblivian@neodymium>conftool action : set/pooled=active; selector: name=mw1338.eqiad.wmnet
18:01<moritzm>uploading HHVM 3.18.7 (3.18.5+dfsg-1+wmf3) for jessie-wikimedia to apt.wikimedia.org
17:44<moritzm>updating HHVM in deployment-prep to HHVM 3.18.7
17:08<godog>bootstrap cassandra-c on restbase1017
16:53<jynus>upgrade and restart db2018
16:53<marostegui@tin>Synchronized wmf-config/db-eqiad.php: Repool db1067 and db1089 - T162807 (duration: 01m 12s)
16:49<jynus>finished codfw s3 master switchover
16:49<jynus@tin>Synchronized wmf-config/db-codfw.php: Switchover s3 codfw master from db2018 to db2036 (duration: 01m 12s)
16:41<_joe_>restarting hhvm on mw1227, threads stuck in HPHP::jit::enterTCImpl
16:31<marostegui>Force WB on db2033 - T184888
16:24<jynus>restarting db2036 to set as master
16:20<jynus>starting codfw s3 master switchover
15:55<marostegui>Stop replication in sync db1067 and db1089 - T162807