Tech:Server admin log

2020-02-24

 * 01:02 paladox: restarted mariadb - db4 as ran out of space

2020-02-23

 * 22:27 paladox: root@mw4:/srv/mediawiki/w/maintenance# sudo -u www-data php resetUserTokens.php --wiki=onepiecebountyrushwiki
 * 21:15 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php updateArticleCount.php --wiki=hispanowiki --update
 * 21:14 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php updateArticleCount.php --wiki=ucroniaswiki --update
 * 20:57 paladox: root@cloud2:/home/paladox# swapoff -a
 * 20:56 paladox: root@cloud1:/home/paladox# swapoff -a
 * 20:50 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php updateArticleCount.php --wiki=hispanowiki --update

2020-02-22

 * 02:55 paladox: drop grafana db on db4
 * 02:49 paladox: apt-get install puppet-agent mariadb-client-10.1 mariadb-client-core-10.1 mariadb-common - misc2
 * 02:43 paladox: remove php/nginx from misc2
 * 01:45 paladox: drop database piwik on db5

2020-02-21

 * 23:31 paladox: migrating matomo to new infra
 * 21:11 paladox: root@mw1:/srv/mediawiki/w/extensions/ManageWiki/maintenance# sudo -u www-data php populateNamespacesWithDefaults.php --wiki=unicodesubsetswiki --overwrite
 * 21:10 paladox: MariaDB [mhglobal]> delete from mw_namespaces where ns_dbname = 'unicodesubsetswiki';
 * 21:09 paladox: root@mw1:/srv/mediawiki/w/extensions/ManageWiki/maintenance# sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki=unicodesubsetswiki --overwrite
 * 10:04 Reception123: RENAME browndustwiki to braveninewiki (db + script)

2020-02-20

 * 01:02 paladox: CHANGE MASTER TO MASTER_HOST='db6.miraheze.org', MASTER_USER='replica', MASTER_PASSWORD='xxxx', MASTER_LOG_FILE='mysql-bin.000004', MASTER_LOG_POS=1044075691; - db6
 * 00:51 paladox: GRANT REPLICATION SLAVE ON *.* TO 'replica'@'%' IDENTIFIED BY 'xxxx'; - db6

2020-02-18

 * 20:54 paladox: "ip addr del fe80::216:3cff:fe25:9a18/64 dev ens3" - db4
 * 20:54 paladox: "ip -6 address add 2a00:d880:5:d6c::2/64 dev ens3" - db4
 * 20:25 paladox: added www-data to ssl-cert on mw[23] lizardfs6
 * 20:19 paladox: fixed perms on /etc/ssl/private on lizardfs6
 * 19:50 paladox: restart redis to see if it will fix the 503s
 * 19:33 paladox: restart php7.3-fpm on mw* and lizardfs6
 * 19:32 paladox: restart php7.3-fpm on mw1
 * 19:11 paladox: enable puppet on jobrunner1
 * 16:44 paladox: upgrade mariadb-client on misc1
 * 16:43 paladox: upgrade grafana on misc1
 * 15:10 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10784;
 * 15:10 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10783;
 * 15:10 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10782;
 * 15:09 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10781;
 * 15:08 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10780;
 * 15:08 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10778;
 * 15:07 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10777;
 * 15:07 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10776;
 * 15:04 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10775;
 * 15:04 paladox: MariaDB [metawiki]> update cw_requests set cw_status = 'approved' where cw_id = 10765;

2020-02-17

 * 16:46 paladox: restart nginx on cp4
 * 16:24 paladox: ALTER TABLE /*_*/oathauth_users DROP COLUMN secret, DROP COLUMN scratch_tokens;
 * 16:12 paladox: renaming kujumwiki to archlinuxfamilyorgwiki per T5241
 * 16:04 paladox: renaming endeavourosdewiki to archlinuxfamilywiki per T5242
 * 15:39 paladox: sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki=pcsdscotlandwiki --overwrite
 * 12:50 Reception123: disabled PlavorSeol on Phabricator
 * 12:04 Reception123: renamed ungamewiki to libertygamewiki (db + script)

2020-02-16

 * 23:52 paladox: MariaDB [allthetropeswiki]> truncate objectcache;
 * 22:56 Southparkfan: set read_only=1 on db6
 * 20:31 Reception123: re-inserted cw_requests from db6 after misclick
 * 19:29 Reception123: removed spam from cw_requests (inappropriate comment)
 * 14:49 paladox: [14:43:09] <+paladox>	!log hacked lizardfs6 Database.php (added persitents mode and removed)
 * 14:44 paladox: hacked lizardfs6 Database.php (added persitents mode and removed)
 * 14:43 paladox: [14:41:46] <+paladox>	!log reboot lizardfs6
 * 14:31 paladox: restart php on mw[123]

2020-02-15

 * 19:48 paladox: depooled mw[4567]
 * 18:41 paladox: repool cp4

2020-02-14

 * 20:28 paladox: backing up phabricator db
 * 20:04 Southparkfan: install atop/iotop/iperf (with fw rule for latter) on db4/db6
 * 19:34 paladox: [19:11:31] <+paladox>	!log set gluster file system to read only on lizardfs6
 * 19:26 Southparkfan: added firewall rule on db4 to allow traffic from db6 (v4/v6) to port 3306
 * 19:26 Southparkfan: dynamically set server_id=4 on db4 and server_id=6 on db6
 * 19:26 Southparkfan: dumping db4's databases to db6 in /home/dbcopy
 * 18:52 Southparkfan: disable puppet on db[456]
 * 14:45 paladox: tried upgrading phabricator to latest on master, failed and rollbacked
 * 10:59 Reception123: umount /mnt/mediawiki-static ; mount /mnt/mediawiki-static on mw3
 * 02:35 paladox: stopped parsoid, removed /srv/parsoid from misc4
 * 02:33 paladox: apt-get upgrade - misc4
 * 02:25 paladox: increase puppet2 disk to 15G
 * 01:42 Southparkfan: decommissioned misc3
 * 01:33 Southparkfan: shutdown misc3 - testing before permanent decom
 * 01:27 paladox: drop phabricator_* on db4
 * 01:23 Southparkfan: decommission cp2
 * 00:56 John: soft depooling of misc1 as ns2 via DNS entry change
 * 00:53 Southparkfan: shutdown cp2 nginx for transferring nginx logs to cp7:/root

2020-02-13

 * 22:05 paladox|UKInEU: install parted on phab1
 * 22:02 paladox|UKInEU: increase phab1 disk to 20g
 * 21:31 paladox|UKInEU: increased phab1 cores to 2
 * 20:45 paladox|UKInEU: install mariadb-client-10.1 on misc4 (to backup the db)
 * 20:43 paladox|UKInEU: stop phd & nginx on misc4. Dump phabricator db, migrate to db6 and migrate phabricator to phab1
 * 20:15 paladox|UKInEU: restart php7.3-fpm on mw3
 * 18:33 paladox|UKInEU: restarted nginx on mw3
 * 01:17 paladox|UKInEU: installed rsync on gluster1
 * 00:04 paladox|UKInEU: reboot test1

2020-02-12

 * 23:44 paladox|UKInEU: lizardfs6 not 67
 * 23:43 paladox|UKInEU: apt-get upgrade - lizardfs67
 * 22:34 paladox|UKInEU: beginning migration from lizardfs6 to gluster1
 * 20:42 Zppix: create more various dummy dbs to prevent issues with wiki creations
 * 16:10 Reception123: move toxicfandomsandhatedomswiki from db4 to db5
 * 15:48 Reception123: moved zhdelwiki from db4 to db5
 * 15:32 Reception123: PURGE BINARY LOGS BEFORE '2020-02-12 12:00:00'; on db4

2020-02-11

 * 23:13 Zppix: created various dummy dbs using sql.php on mw1 to prevent creation of new wikis with existing hostnames to prevent potential issues
 * 01:51 paladox|UKInEU: reboot cp2
 * 01:33 paladox|UKInEU: MariaDB [mhglobal]> update mw_namespaces set ns_additional = '{"wgNamespacesToPostIn":true,"wgVisualEditorAvailableNamespaces":true}' where ns_dbname = 'civclassicwiki' and ns_namespace_name = '';
 * 01:01 paladox|UKInEU: rm /srv/mediawiki/w/cache/managewiki/** on mw[123], lizardfs6

2020-02-10

 * 19:42 Reception123: MariaDB [(none)]> DROP DATABASE isabellafan2wiki;
 * 19:42 Reception123: deleted isabellafan2wiki
 * 02:22 paladox|UKInEU: probe gluster1 from lizardfs(gluster) but do not add it as a brick

2020-02-08

 * 15:29 paladox|UKInEU: rename diaroleplayswiki to sparkleswiki per https://phabricator.miraheze.org/T5188
 * 15:27 paladox|UKInEU: MariaDB [shiropediawiki]> delete from nl_newsletters where nl_id = 5;
 * 15:26 paladox|UKInEU: MariaDB [shiropediawiki]> delete from nl_newsletters where nl_id = 6;
 * 15:20 paladox|UKInEU: renaming idiotpediawiki to nocyclopediawiki per https://phabricator.miraheze.org/T5208
 * 15:15 paladox|UKInEU: delete & drop ewrikiawiki per https://phabricator.miraheze.org/T5213
 * 15:14 paladox|UKInEU: drop cdwwik, gameshowswiki, glaucuswiki, receptionflamewarswiki, stargazerwikiwiki, thechasewikiwiki, toslawwiki
 * 10:14 Reception123: PURGE BINARY LOGS BEFORE '2020-02-08 08:00';

2020-02-07

 * 19:22 Zppix: create mw[67]wiki using sql.php
 * 19:21 Zppix: created cloud1 and cloud2 dbs using sql.php

2020-02-06

 * 01:41 paladox|UKInEU: reboot mw3

2020-02-05

 * 19:49 paladox|UKInEU: upgrade php on lizardfs6
 * 19:47 paladox|UKInEU: upgrade php on mw1
 * 19:46 paladox|UKInEU: upgrade php on mw2
 * 19:43 paladox|UKInEU: upgrade php on mw3
 * 19:39 paladox|UKInEU: restart php7.3-fpm on mw3
 * 19:31 Zppix: initsitestats and rebuildrc for ungamewiki after import

2020-02-04

 * 20:47 Southparkfan: locked phab auth config
 * 20:41 Southparkfan: unlocked phab auth config (temp)

2020-02-03

 * 17:28 paladox|UKInEU: root@mw1:/var/log/mediawiki# rm *.log.*
 * 17:27 paladox|UKInEU: rm /var/log/mediawiki/*.log*
 * 17:26 paladox|UKInEU: rm /var/log/mediawiki/*.log

2020-02-02

 * 15:50 paladox|UKInEU: > MatomoAnalytics::addSite( 'loginwiki' );
 * 15:50 paladox|UKInEU: > MatomoAnalytics::deleteSite( 'loginwiki' );
 * 15:48 paladox|UKInEU: > MatomoAnalytics::addSite( 'foxmaskeventwiki' );
 * 15:48 paladox|UKInEU: > MatomoAnalytics::deleteSite( 'foxmaskeventwiki' );
 * 15:48 paladox|UKInEU: > MatomoAnalytics::addSite( 'davshiqwiki' );
 * 15:48 paladox|UKInEU: > MatomoAnalytics::deleteSite( 'davshiqwiki' );
 * 14:03 paladox|UKInEU: remove hack from cron - db5
 * 04:34 paladox|UKInEU: hack /var/spool/cron/crontabs/root and run /home/paladox/purge-binary.sh every 1 minute.
 * 00:38 paladox|UKInEU: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php rebuildall.php --wiki=csydeswiki
 * 00:02 paladox|UKInEU: drop database piwik on db5; create databse db5 & import from backup. To see if this will reduce the disk space it's using from ~75gb.

2020-02-01

 * 23:19 paladox|UKInEU: drop database uncyclomirrorwiki on db4 as migrated to db5
 * 22:29 paladox|UKInEU: root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=metawiki --type=gzip
 * 22:23 paladox|UKInEU: root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=toxicfandomsandhatedomswiki --type=gzip
 * 22:18 paladox|UKInEU: root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=uncyclomirrorwiki --type=gzip
 * 22:14 paladox|UKInEU: root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=altversewiki --type=gzip
 * 12:26 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki Tundra "Just Amelia" --logwiki=metawiki

2020-01-31

 * 16:48 paladox|UKInEU: MariaDB [gmcwiki]> INSERT INTO content_models (model_id,model_name) VALUES (4,'Scribunto');
 * 16:47 paladox|UKInEU: MariaDB [gmcwiki]> INSERT INTO content_models (model_id,model_name) VALUES (2,'css');
 * 16:47 paladox|UKInEU: MariaDB [gmcwiki]> INSERT INTO content_models (model_id,model_name) VALUES (3,'javascript');
 * 16:47 paladox|UKInEU: MariaDB [gmcwiki]> INSERT INTO content_models (model_id,model_name) VALUES (5,'json');
 * 16:46 paladox|UKInEU: MariaDB [gmcwiki]> INSERT INTO content_models (model_id,model_name) VALUES (1,'wikitext');
 * 16:43 paladox|UKInEU: MariaDB [gmcwiki]> INSERT INTO slot_roles (role_id,role_name) VALUES (1,'main');
 * 16:37 paladox|UKInEU: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php cleanupUsersWithNoId.php --wiki=gmcwiki --force -p gmcwiki

2020-01-30

 * 18:02 paladox: apt-get upgrade - mw[123] (gluster, php and puppet-agent + few other packages)
 * 17:55 paladox: apt-get upgrade - lizardfs6 (gluster, php and puppet-agent + few other packages)

2020-01-28

 * 23:42 paladox: upgrade puppet-agent & salt on misc1
 * 23:41 paladox: upgrade php - misc1
 * 23:36 paladox: upgrade grafana - misc1
 * 22:00 paladox: root@mw1:/srv/mediawiki/w/maintenance/storage# sudo -u www-data php compressOld.php --wiki=ungamewiki --type=gzip
 * 21:58 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/guiasdobrasil.com.br "', null ) WHERE wiki_dbname = 'guiaslocaiswiki';
 * 19:32 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php nukeNS.php --wiki=thefinalrumblewiki --all --ns=3001 --delete
 * 13:13 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki Toshinori "Agent Mace" --logwiki=metawiki
 * 00:53 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=files
 * 00:52 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=subcats
 * 00:52 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=pages

2020-01-27

 * 22:59 paladox: sync; echo 1 > /proc/sys/vm/drop_caches (lizardfs6 & db4)
 * 22:50 paladox: sudo /sbin/sysctl -w vm.drop_caches=3 on lizardfs6 and db4 temporarily
 * 22:28 Zppix: zppix@mw1:~$ sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --report=1 --no-local-users --username-prefix "백괴게임" --wiki ungamewiki /home/zppix/uncyclogame-full-2019-05-15.xml (T5148)
 * 20:02 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php deleteBatch.php --wiki=thefinalrumblewiki list (Deleting Message_Wall:%)
 * 19:57 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php deleteBatch.php --wiki=thefinalrumblewiki list2 (Deleting User_blog_comment:%)
 * 19:55 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php deleteBatch.php --wiki=thefinalrumblewiki list (Deleting User_blog:%)
 * 19:51 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php nukeNS.php --wiki=thefinalrumblewiki --all --ns=3000 --delete
 * 17:28 paladox: root@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance# sudo -u www-data php migrateAccount.php --wiki=wikifinanzawiki --userlist=./userlist --homewiki=wikifinanzawiki --attachmissing --auto
 * 15:26 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki ZapperGlaze49 "TheChisatoFan" --logwiki=metawiki

2020-01-26

 * 22:15 paladox: revert hack on mw[123] & lizardfs6
 * 20:12 paladox: hacked LS on mw[123] & lizardfs6 to switch off wgUseInstantCommons temporarily
 * 17:51 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=pages/subpages/files
 * 17:41 paladox: root@mw2:/srv/mediawiki/w/maintenance# sudo -u www-data php refreshLinks.php --wiki=ucroniaswiki
 * 17:41 paladox: root@mw2:/srv/mediawiki/w/maintenance# sudo -u www-data php recountCategories.php --wiki=ucroniaswiki --mode=pages
 * 16:48 paladox: revert hack on mw[123] & lizardfs6
 * 15:18 paladox: hacked LS on mw[23] & lizardfs6 to switch off wgUseInstantCommons temporarily
 * 15:17 paladox: hacked LS on mw1 to switch off wgUseInstantCommons temporarily
 * 15:06 paladox: restart php7.3-fpm on mw1 and mw3
 * 15:05 paladox: restart php7.3-fpm on mw2 and lizardfs6
 * 06:37 Reception123: deleted and dropped gmcwiki (T5150)

2020-01-25

 * 18:40 Zppix: sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki crappygameswiki -u "Zppix" -r "Requested - T5153" /home/zppix/crappygameswiki_batchdelete.txt
 * 18:21 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki TTyranny "Tarbiz" --logwiki=metawiki
 * 17:52 paladox: reinstall php on misc2
 * 17:47 paladox: rotate redis.log on misc2
 * 15:02 paladox: restart php7.3-fpm && nutcracker on mw3
 * 15:00 paladox: restart php7.3-fpm && nutcracker on mw2
 * 00:16 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/wiki.udia.ca "', null ) WHERE wiki_dbname = 'udiawiki';
 * 00:15 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/islamkosh.tk "', null ) WHERE wiki_dbname = 'ipediawiki';
 * 00:15 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/glp.ink "', null ) WHERE wiki_dbname = 'cnwiki';
 * 00:14 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/wiki.counterculturelabs.org "', null ) WHERE wiki_dbname = 'cclwiki';

2020-01-24

 * 23:28 paladox: restart php7.3-fpm on mw3 and lizardfs6
 * 23:28 paladox: restart php7.3-fpm on mw2
 * 23:27 paladox: restart php7.3-fpm on mw1
 * 14:12 paladox: MariaDB [mhglobal]> update mw_permissions set perm_removegroupsfromself = '[]' where perm_removegroupsfromself = 'null';
 * 14:10 paladox: Remove abusefilter-hide-log,abusefilter-hidden-log,abusefilter-modify-global,abusefilter-private,abusefilter-private-log,aft-oversighter,autocreateaccount,bigdelete,centralauth-lock,centralauth-oversight,centralauth-rename,centralauth-unmerge,centralauth-usermerge,checkuser,checkuser-log,createwiki,editincidents,editothersprofiles,editothersprofiles-private,flow-suppress,globalblock,globalblock-exempt,globalgroupmembership,globalgrouppermissions,hideuser,interwiki,managewiki-restricted,managewiki-editdefault,moderation-checkuser,mwoauthmanageconsumer,mwoauthmanagemygrants,mwoauthsuppress,mwoauthviewprivate,mwoauthviewsuppressed,oathauth-disable-for-user,oathauth-view-logrenameuser,requestwiki,siteadmin,stopforumspam,suppressionlog,suppressrevision,usermerge,userrights,userrights-interwiki,viewglobalprivatefiles,viewpmlog,viewsuppressed from all wikis.
 * 14:04 paladox: MariaDB [mhglobal]> update mw_permissions set perm_addgroupstoself = '[]' where perm_addgroupstoself = 'null';
 * 14:02 paladox: rm -rf /srv/mediawiki/w/cache/managewiki/** on mw* and lizardfs6
 * 14:01 paladox: MariaDB [mhglobal]> update mw_permissions set perm_removegroupsfromself = '[]' where perm_removegroupsfromself = 'null';
 * 14:01 paladox: MariaDB [mhglobal]> update mw_permissions set perm_removegroups = '[]' where perm_removegroups = 'null';
 * 14:01 paladox: MariaDB [mhglobal]> update mw_permissions set perm_addgroups = '[]' where perm_addgroups = 'null';
 * 13:59 paladox: MariaDB [mhglobal]> update mw_permissions set perm_addgroupstoself = '[]' where perm_addgroupstoself = 'null';
 * 13:38 paladox: remove oathauth-disable-for-user from all groups.

2020-01-23

 * 00:29 paladox_UK_IN_EU: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=spectrobeswiki --search-recursively --skip-dupes ./Spectrobes_Wiki_Images

2020-01-22

 * 23:54 paladox: apt-get upgrade cp3
 * 22:51 paladox: remove oathauth-disable-for-user from all wikis
 * 19:29 paladox_UK_IN_EU: remove oauthauth-disable-for-user from all wikis
 * 18:31 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=spectrobeswiki --report=1 spectrobes_pages_full.xml - per request through email
 * 18:31 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=spectrobeswiki --report=1 spectrobes_pages_full.xml.7z - per request through email
 * 06:52 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki WageUglyDollandMarlaPlaymobil12 "Just Wage" --logwiki=metawiki

2020-01-21

 * 21:29 paladox: restart nutcracker on mw1
 * 14:53 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=wiesepediawiki /home/reception/Wiesepedia-Datadump[0,A-D] on mw1
 * 14:47 Reception123: PURGE BINARY LOGS before '2020-01-20 22:00:00' on db4
 * 14:46 Reception123: manually remove custom domain for hypewiki
 * 14:37 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=gmcwiki /home/reception/graalmilitary_pages_full.xml on mw3
 * 14:33 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=spectrobeswiki /home/reception/spectrobes_pages_full.xml on mw3

2020-01-20

 * 17:05 paladox: ALTER TABLE /*_*/searchindex MODIFY COLUMN si_title TEXT BINARY NOT NULL; - simswiki
 * 16:59 paladox: ran ./2.sh on db4 (basically finding all titles that are in User Blog Comment on wikia and setting ns id to 3007 from 0)
 * 16:54 Zppix: UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/nonsensopedia.org "', null ) WHERE wiki_dbname = 'nonsensopediawiki'; on mw1 using sql.php
 * 01:07 paladox: ALTER TABLE /*_*/searchindex MODIFY COLUMN si_title varchar(300) BINARY NOT NULL default ''; - simswiki (applying
 * 00:49 paladox: MariaDB [simswiki]> update page set page_namespace = 3007 where page_namespace = 3008;
 * 00:42 paladox: remove hack from mw1
 * 00:39 paladox: ALTER TABLE /*_*/searchindex MODIFY COLUMN si_title TEXT BINARY NOT NULL; (applying https://gerrit.wikimedia.org/r/#/c/mediawiki/core/+/565765/8 to simswiki)
 * 00:11 paladox: MariaDB [simswiki]> update page set page_namespace = 3008 where page_namespace = 3007;

2020-01-19

 * 22:30 paladox: hacked rebuildtextindex.php on mw1 to output some debugging info
 * 21:45 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php rebuildall.php --wiki=simswiki
 * 21:44 paladox: ALTER TABLE /*_*/searchindex MODIFY COLUMN si_title varchar(255) BINARY NOT NULL default ''; - simswiki (fixing an issue when rebuilding search index)
 * 21:38 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=hispanowiki batch2 - T5122
 * 21:37 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=ucroniaswiki batch - T5122
 * 21:34 paladox: root@mw1:/srv/mediawiki/w/extensions/ManageWiki/maintenance# sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki=toastycraftwiki --overwrite -T5119
 * 21:34 paladox: root@mw1:/srv/mediawiki/w/extensions/ManageWiki/maintenance# sudo -u www-data php populateGroupPermissionsWithDefaults.php --wiki=aranziaronzowiki --overwrite -T5119
 * 17:45 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=hispanowiki batch2
 * 17:34 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php /srv/mediawiki/w/maintenance/deleteBatch.php --wiki=ucroniaswiki batch
 * 15:05 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=loginwiki --report=1 Wikipedia-20200119150417.xml
 * 14:57 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=loginwiki --report=1 Wikipedia-20200119145611.xml
 * 13:54 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php rebuildall.php --wiki=simswiki
 * 08:51 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php deleteBatch.php --wiki=thefinalrumblewiki /home/reception/deletebatch.txt
 * 01:23 paladox: restart php7.3-fpm on lizardfs6
 * 00:25 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importImages.php --wiki=thefinalrumblewiki --skip-dupes --search-recursively TFR_Wikia_Images/ (per request on discord)

2020-01-18

 * 22:15 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=simswiki --report=1 --no-updates sims_pages_full.xml
 * 22:09 paladox: root@mw1:/srv/mediawiki/w/maintenance# sudo -u www-data php deleteBatch.php --wiki=thefinalrumblewiki batch - per request on discord
 * 20:46 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=simswiki --report=1 sims_pages_full.xml
 * 17:09 paladox: that script is adding "urlshortener-create-url" to * for all wikis
 * 17:08 paladox: root@mw1:/home/paladox# ./foreachwikiindblist /srv/mediawiki/dblist/all.dblist
 * 16:58 paladox: delete from mw_permissions where perm_dbname = '131parkhurstwiki' and perm_group = "all.dblist";
 * 15:44 paladox: delete from mw_permissions where perm_dbname = '131parkhurstwiki' and perm_group = "\\*" LIMIT 1;
 * 15:44 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = "\\*" LIMIT 1;
 * 15:44 paladox: delete from mw_permissions where perm_dbname = '131parkhurstwiki' and perm_group = "\\*" LIMIT 1;
 * 15:44 paladox: delete from mw_permissions where perm_dbname = '131parkhurstwiki' and perm_group = "\\*" LIMIT 1;
 * 15:44 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = "\\*" LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = "\*" LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = '\*' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = '+' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = '+' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '1920swiki' and perm_group = '+' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '1920swiki' and perm_group = '"*"' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '131parkhurstwiki' and perm_group = '"*"' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '131parkhurstwiki' and perm_group = '\*' LIMIT 1;
 * 15:43 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = '\*' LIMIT 1;
 * 15:42 paladox: delete from mw_permissions where perm_dbname = '00sretrolivingwiki' and perm_group = '"*"' LIMIT 1;
 * 14:16 Reception123: manually remove custom domain from teessidehackspacewiki

2020-01-17

 * 19:10 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=thefinalrumblewiki /home/reception/thefinalrumble_pages_full.xml
 * 02:11 paladox: rm -rf /srv/mediawiki/w/cache/manage*/** on mw* and lizardfs6
 * 01:48 paladox: restoring mw_permissions from a newish backup (after running "delete from mw_permissions where perm_group = "\*";" which seemed to updated 4,000 results)
 * 01:43 paladox: update mw_permissions set perm_permissions = '["autocreateaccount","createaccount","edit","createpage","createtalk", "writeapi","viewmywatchlist","editmywatchlist","viewmyprivateinfo", "editmyprivateinfo","editmyoptions","abusefilter-log","abusefilter-log-detail", "abusefilter-view","autocreateaccount","centralauth-merge", "oathauth-enable","urlshortener-create-url"]' where perm_group = '*' and perm_dbname = 'default';
 * 00:12 paladox: apt-get upgrade - misc2

2020-01-16

 * 23:53 paladox: /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/extensions/ManageWiki/maintenance/modifyGroupPermission.php --addperms="urlshortener-create-url,urlshortener-manage-url" sysop
 * 23:47 paladox: update mw_permissions set perm_permissions = '["editsitecss","upload_by_url","editsitejson","editsitejs","skipcaptcha","block","createaccount","delete","deletedhistory","deletedtext","undelete","editinterface","editusercss","edituserjson","edituserjs","import","importupload","move","move-subpages","move-rootuserpages","move-categorypages","patrol","autopatrol","protect","editprotected","rollback","upload","reupload","reupload-shared","unwatchedpages","autoconfirmed","editsemiprotected","ipblock-exempt","blockemail","markbotedits","apihighlimits","browsearchive","noratelimit","movefile","unblockself","suppressredirect","mergehistory","managechangetags","deletechangetags","abusefilter-modify","abusefilter-modify-restricted","abusefilter-revert","deletelogentry","deleterevision","abusefilter-log-detail","override-antispoof","globalblock-whitelist","massmessage","nuke","tboverride","titleblacklistlog", "delete-dump","generate-dump","view-dump","urlshortener-create-url","urlshortener-manage-url"]' where perm_group = 'sysop' and perm_dbname = 'default';
 * 02:34 paladox: reboot misc4
 * 02:15 paladox: downgraded puppetdb on puppet1 - broke ssl connection to postgresql
 * 01:48 paladox: root@db4:/home/paladox# sudo service postgresql restart
 * 01:44 paladox: on puppet1
 * 01:44 paladox: apt-get upgrade puppet-agent puppetdb puppetdb-termini puppetserver salt-common salt-minion
 * 01:43 paladox: upgrade puppet-agent on cp4

2020-01-15

 * 08:26 Reception123: sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki AnimalLovingGuy "ScrewSociety" --logwiki=metawiki

2020-01-14

 * 23:10 paladox: reinstall cp2 as debian buster
 * 23:09 paladox: restart php7.3-fpm on mw2 for config change
 * 23:09 paladox: turn TUN off on cp2
 * 23:06 paladox: restart php7.3-fpm on mw3 for config change
 * 22:59 paladox: rebuild lc on mw* and lizardfs6
 * 17:39 paladox: apt-get upgrade - lizardfs6
 * 01:43 paladox: sudo -u www-data php update.php --wiki=sagan4wiki --quick --force - mw1
 * 01:21 paladox: MariaDB [sagan4wiki]> ALTER TABLE /*_*/cargo_tables ADD field_helper_tables text NOT NULL;

2020-01-13

 * 15:32 paladox: MariaDB [mhglobal]> UPDATE cw_wikis SET wiki_settings = REPLACE(wiki_settings, '"wgServer":"https:\/\/wintermoor.net "', null ) WHERE wiki_dbname = 'wintermoorwiki';

2020-01-12

 * 23:29 paladox: removed redis from all mediawiki servers
 * 15:25 paladox: repooled lizardfs6
 * 15:15 paladox: temporarily depool lizardfs6
 * 08:32 Reception123: reception@mw1:/srv/mediawiki/w/maintenance$ sudo -u www-data php purgeList.php --all --purge --wiki twinkletestwiki + rebuildall.php
 * 08:06 Reception123: ran rebuildall on twinkletestwiki

2020-01-11

 * 23:35 paladox: [23:30:52] <+paladox>	!log switched TUN off on misc2
 * 23:34 paladox: apt-get upgrade on misc2
 * 22:52 paladox: upgrade nodejs on misc3
 * 22:48 paladox: resizing misc3 - https://clientarea.ramnode.com/index.php?rp=/announcements/431/More-SSD-for-Standard-KVM.html
 * 17:04 paladox: apt-get upgrade - bacula1
 * 16:58 paladox: root@bacula1:/bacula/backup# rm DB*
 * 16:57 Reception123: sudo -u www-data php /srv/mediawiki/w/maintenance/rebuildLocalisationCache.php --wiki loginwiki on mw* and lizardfs6
 * 14:59 Reception123: DELETED and DROPPED dbs https://phabricator.miraheze.org/P247
 * 14:52 Reception123: reception@mw1:~$ sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/deleteWikis.php --wiki loginwiki --delete Reception123
 * 11:12 Reception123: reception@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance$ sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki GasMask0217 "KP619" --logwiki=metawiki
 * 10:29 Reception123: run fixStuckGlobalRename.php on all wikis for username above
 * 10:18 Reception123: sudo -u www-data php fixStuckGlobalRename.php --wiki=aawikiwiki PaperClip "FireBarrier101" --logwiki=metawiki
 * 04:25 paladox: repool lizardfs6
 * 04:18 paladox: depool lizardfs6
 * 04:18 paladox: repool mw3
 * 04:12 paladox: depool mw3
 * 04:12 paladox: repool mw2
 * 04:08 paladox: depool mw2
 * 04:07 paladox: repool mw1
 * 04:04 paladox: depool mw1

2020-01-09

 * 22:07 paladox: sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=inforevivalwiki --report=1 Wiki-20200109132453.xml
 * 21:20 paladox: rebuild lc on mw* and lizardfs6
 * 19:34 paladox: ALTER TABLE /*$wgDBprefix*/matomo ADD INDEX matomo_wiki (matomo_wiki); - db4
 * 19:17 paladox: MariaDB [mhglobal]> ALTER TABLE /*$wgDBprefix*/cw_wikis ADD INDEX wiki_dbname (wiki_dbname); -db4
 * 18:59 paladox: MariaDB [mhglobal]> ALTER TABLE /*$wgDBprefix*/mw_namespaces ADD INDEX ns_dbname (ns_dbname); -db4
 * 18:59 paladox: MariaDB [mhglobal]> ALTER TABLE /*$wgDBprefix*/mw_permissions ADD INDEX perm_dbname (perm_dbname); - db4
 * 00:57 paladox: restart php7.3-fpm on mw2

2020-01-08

 * 21:37 paladox: restart mysql on db4
 * 21:33 paladox: restart mysql on db5
 * 19:56 paladox: restart php7.3-fpm on lizardfs6 and mw*
 * 17:16 paladox: SET GLOBAL thread_cache_size = 100; - (db4|db5)
 * 16:52 paladox: MariaDB [(none)]> SET GLOBAL thread_cache_size = 80;
 * 16:33 paladox: MariaDB [(none)]> SET GLOBAL thread_pool_stall_limit = 10;
 * 16:14 paladox: MariaDB [(none)]> SET GLOBAL slow_query_log = 'OFF';
 * 16:10 paladox: MariaDB [(none)]> SET GLOBAL long_query_time = 10;
 * 16:04 paladox: MariaDB [(none)]> SET GLOBAL long_query_time = 15;
 * 16:04 paladox: MariaDB [(none)]> SET GLOBAL slow_query_log = 'ON';
 * 15:30 paladox: restart php7.3-fpm on mw* and lizardfs6
 * 15:21 Reception123: purge binary logs BEFORE '2020-01-08 08:00'

2020-01-07

 * 17:16 paladox: root@mw1:/srv/mediawiki/w/extensions/CentralAuth/maintenance# sudo -u www-data php fixStuckGlobalRename.php --wiki=receptionflamewarswiki --logwiki=metawiki ErtasVideos Hendicted

2020-01-06

 * 20:22 Reception123: reception@mw2:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki testwiki /home/reception/Wikipedia-20200106202002.xml

2020-01-05

 * 19:47 paladox: rebuild lc on mw* and lizardfs6
 * 00:25 paladox: ./bin/auth strip --user Paladox --all-types (misc4)

2020-01-04

 * 10:13 Reception123: reception@mw3:/srv/mediawiki/w/maintenance$ sudo -u www-data php importDump.php --wiki loginwiki /home/reception/TestWiki-20200104100040.xml

2020-01-02

 * 18:48 paladox: depool, repool and restart php7.3-fpm on mw3
 * 18:44 paladox: depool, repool and restart php7.3-fpm on mw2
 * 18:43 paladox: restart php7.3-fpm on mw1
 * 18:31 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/maintenance/importDump.php --wiki=radioscanningtwwiki --report=1 X.xml
 * 03:10 paladox: rm -rf /srv/mediawiki/w/cache/managewiki/** on mw* and lizardfs6
 * 03:01 paladox: root@test1:/srv/mediawiki/w/extensions/StopForumSpam/maintenance# sudo -u www-data php updateBlacklist.php --wiki=sdiywiki
 * 02:44 paladox: upgrade phabricator on misc4
 * 02:42 paladox: MariaDB [(none)]> drop database uncyclopediakrwiki;
 * 02:30 paladox: root@mw1:/home/paladox# sudo -u www-data php /srv/mediawiki/w/extensions/CreateWiki/maintenance/renameWiki.php --wiki=loginwiki --rename uncyclopediakrwiki truthwiki paladox

2020-01-01

 * 17:49 paladox: root@misc1:/home/paladox# deluser labster
 * 15:53 Reception123: sudo -u www-data php fixStuckGlobalRename.php --wiki=200xwiki PaperClip "FireBarrier101" --logwiki=metawiki
 * 14:24 Reception123: MariaDB [phabricator_herald]> UPDATE herald_rule SET isDisabled = '1' WHERE id = '21'; (empty)
 * 04:36 paladox: root@mw1:/srv/mediawiki/w# /usr/local/bin/foreachwikiindblist /srv/mediawiki/dblist/all.dblist /srv/mediawiki/w/maintenance/sql.php /srv/mediawiki/w/extensions/QuizGame/sql/patches/patch-add-default-choice_answer_count.sql
 * 00:37 paladox: restart varnish cp2,cp4