Toolserver Issue Tracker

https://jira.toolserver.org/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+%22%5Bmigrated+to+bugzilla%2C+do+not+use%5D+DrTrigon%27s+tools%22+OR+reporter+%3D+%27drtrigon%27+OR+summary+%7E+drtrigon+OR+description+%7E+drtrigon+OR+comment+%7E+drtrigon+OR+environment+%7E+drtrigon+ORDER+BY+key+DESC

An XML representation of a search request

en-us from 0 to 157 (157) 5.0.6 733 31-05-2012

[TS-1683] Please install "fabric" Created: Sat, 14 Sep 2013 20:28:42 +0000 Updated: Mon, 6 Jan 2014 12:43:03 +0000

Status: Open
Project: Toolserver
Component/s: Software installation,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Task Priority: Major
Reporter: drtrigon Assignee: Unassigned
Resolution: Unresolved Votes (Watches): 0 (2)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified


Description

Could you please install "fabric" [1]? Thanks in advance!

[1] http://docs.fabfile.org/en/1.7/


Comments




[TS-1669] nosetests "Permission denied" Created: Wed, 17 Jul 2013 16:45:38 +0000 Updated: Wed, 18 Dec 2013 22:07:03 +0000 Resolved: Wed, 18 Dec 2013 22:07:03 +0000

Status: Resolved
Project: Toolserver
Component/s: Software installation,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: Tim.Landscheidt
Resolution: Cannot Reproduce Votes (Watches): 0 (2)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to TS-1301 Install 'doxygen' on (all) login server(s) Resolved

Description

When trying to run nosetests, the following problem appears:

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.6/dist-packages/coverage.py", line 473, in save
cache = open(self.cache, 'wb')
IOError: [Errno 13] Permission denied: '.coverage'
Error in sys.exitfunc:
Traceback (most recent call last):
File "/usr/lib/python2.6/atexit.py", line 24, in _run_exitfuncs
func(*targs, **kargs)
File "/usr/lib/python2.6/dist-packages/coverage.py", line 473, in save
cache = open(self.cache, 'wb')
IOError: [Errno 13] Permission denied: '.coverage'

this '.coverage' file should be created in the local directory when exectuting the command.


Comments
Comment by Tim.Landscheidt [ Wed, 18 Dec 2013 22:07:03 +0000 ]

If I run nosetests --with-coverage, .coverage gets created in the current directory. If you mean something different, please reopen this bug.





[TS-1636] Login and account Created: Thu, 11 Apr 2013 07:40:50 +0000 Updated: Thu, 11 Apr 2013 07:53:03 +0000 Resolved: Thu, 11 Apr 2013 07:53:03 +0000

Status: Resolved
Project: Toolserver
Component/s: Accounts, Loginserver: General/Unknown,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: Marlen Caemmerer
Resolution: Fixed Votes (Watches): 0 (2)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://lists.wikimedia.org/pipermail/toolserver-l/2013-April/005879.html

Description

Hello beloved Admins!

Today I recognized that my account has expired (what a pitty) so I
tried as usual to login and renew but just get this:

> $ ssh drtrigon at nightshade.toolserver.org
> Your account has expired; please contact your system administrator
> Connection closed by 185.15.59.201

...so now what to do?

Thanks and Greetings!
DrTrigon


Comments
Comment by nosy [ Thu, 11 Apr 2013 07:53:03 +0000 ]

Done. Set new expiry date to 1st June.





[TS-1635] mayapple config & cron jobs Created: Thu, 11 Apr 2013 07:35:50 +0000 Updated: Mon, 6 May 2013 14:57:07 +0000 Resolved: Mon, 6 May 2013 14:57:07 +0000

Status: Resolved
Project: Toolserver
Component/s: Stable server,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: Tim.Landscheidt
Resolution: Not a bug Votes (Watches): 0 (2)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://lists.wikimedia.org/pipermail/toolserver-l/2013-March/005850.html

Description

Hello everybody!
Hello nosy!

Do you remember the issue below...?

On 05.03.2013 13:03, Marlen Caemmerer wrote:
>> Then another issue - I think (again - sorry but yes! we have
>> some problem with cron job execution. When looking at my bots
>> (even thought this is not COMPLETELY clear beacuse of other issue
>> mentioned before) but also at the lower left of [4] to be precise
>> [5] and compare "Week 6" with "Week 7" till now ("Week 9") then
>> we clearly have some issues. The first part of "Week 7" may be
>> until mid "Week 8" looks resonable, but the rest looks "free
>> floating" to me which may point to some load or other issue...
>> what do you think? Thanks again.
>>
>> [4] http://munin.toolserver.org/Login/hawthorn/cron_jobs_sh.html
>> [5]
>> http://munin.toolserver.org/Login/hawthorn/cron_jobs_sh-month.png
>>
>>
>
>> Thanks for pointing to this. Cron runs again.

I think it crashed another time...

If you compare "Week 10/11" with "Week 12" till now ("Week 13") then
we clearly have some issues.
The first part of "Week 11" may be until mid "Week 12" looks
resonable, but the rest looks "free floating" to me which may point to
some load or other issue... what do you think? Thanks again. )

Greetings and all the best!!
DrTrigon


Comments
Comment by Tim.Landscheidt [ Tue, 30 Apr 2013 18:48:47 +0000 ]

I don't think it's very useful to diagnose problems with cron by looking at some statistics. Do you have jobs not executed? Then please name them clearly so the admins can look into the issue. Otherwise there's no problem to fix.

Comment by drtrigon [ Wed, 1 May 2013 11:51:39 +0000 ]

Ok let's try to be clear.

Regarding THIS concrete issue mentioned here, please have a look at [1] where I noted: "As I noticed it seems as if the cronjobs are executed correct at the moment, so may be TS-1635 can be closed."

Regarding your statement "I don't think it's very useful to diagnose problems with cron by looking at some statistics." I have to disagree in some points. Yes it is not THE tool of choice to diagnose the cause of the issues. BUT it is clearly an accurate sign that something is going wrong when the number of executed jobs starts slowly to decrease and "floats freely" as I mentioned in other mails earlier... Btw. I was just mentioning this in cases were I cleary HAD jobs NOT executed in order to have something to refer to. If you can point me to another better indicator I will enjoy using it.

That was also the deeper reason for the question in my mail [1]: "Can someone give me a brief introduction on what changed and how this
works now?" Thus I would be thankful for any further hints, info, explanation or else givin me more insight...

[1] http://lists.wikimedia.org/pipermail/toolserver-l/2013-April/005980.html

Thanks a lot and Greetings

Comment by drtrigon [ Sat, 4 May 2013 01:22:15 +0000 ]

...and now as I said that and do a look at the latest data (from this week) I have to withdraw and report again a cron job issue!

Comment by Tim.Landscheidt [ Mon, 6 May 2013 14:56:25 +0000 ]

It's not useful to infer anything from the number of cron jobs run (aside from "0") because the number of cron jobs isn't static. It's a function of how many cron jobs users schedule. Perhaps some tool has been disabled, some code has been optimized, who knows? I much prefer if the roots were concentrating their efforts on resolving reproducible bugs instead of wildly stabbing in the dark.

I'm closing this issue now even though I don't know why you didn't do that yourself. BTW, I don't see a new report from you in the Toolserver project.





Install 'lunatic-python' (together with 'lua')(TS-1303)

[TS-1583] Install lua dev(el) package(s) Created: Sat, 15 Dec 2012 11:26:26 +0000 Updated: Mon, 6 Jan 2014 12:44:41 +0000

Status: Open
Project: Toolserver
Component/s: None
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: Unassigned
Resolution: Unresolved Votes (Watches): 0 (2)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to DRTRIGON-129 Check to usefulness and capabilities of Labs (vs. TS) Closed
is related to DRTRIGON-118 Enable lua support to enable even better template programming inclusion (subster) Closed
is related to DRTRIGON-86 Test the re-write branch an decide what parts to migrate Closed

Description

Please install lua dev(el) package(s) in order to enable to compile against the lua libraries. Needed are e.g. lua.h, lauxlib.h, ...

Thanks a lot and greetings


Comments
Comment by drtrigon [ Thu, 27 Dec 2012 22:57:15 +0000 ]

Have to look into the compilation in more detail. May be it is already possible on the TS - else could be done on labs...

Comment by drtrigon [ Thu, 27 Dec 2012 22:58:38 +0000 ]

Since could my be done on labs else.

Comment by drtrigon [ Fri, 28 Dec 2012 20:37:58 +0000 ]

Needed code implemented in r435 (works on labs already).

Comment by Tim.Landscheidt [ Wed, 18 Dec 2013 22:20:23 +0000 ]

Do you still need the packages on Toolserver?





[TS-1557] acctrenew missing on linux hosts Created: Sun, 21 Oct 2012 08:03:44 +0000 Updated: Wed, 18 Dec 2013 23:30:44 +0000 Resolved: Wed, 18 Dec 2013 23:30:44 +0000

Status: Resolved
Project: Toolserver
Component/s: Loginserver: General/Unknown,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Task Priority: Major
Reporter: drtrigon Assignee: Tim.Landscheidt
Resolution: Won't Fix Votes (Watches): 1 (3)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified
Environment:

Nightshade, linux



Description

acctrenew is missing on nightshade, as you can see:

drtrigon@nightshade:~$ acctrenew
-bash: acctrenew: command not found

btw.: also the notice "Your account will expire on ..." is missing on nightshade when loggin in


Comments
Comment by Tim.Landscheidt [ Tue, 23 Oct 2012 23:07:50 +0000 ]

See also https://github.com/scfc/tsutils for an initial conversion of the build system to autotools.

Comment by Tim.Landscheidt [ Wed, 18 Dec 2013 23:30:44 +0000 ]

It probably does no longer make sense to invest time in a Linux port; ssh $SOLARISHOST should be good enough for the remaining time .





[TS-1554] Check if DrTrigonBot (framework) can run on linux Created: Tue, 16 Oct 2012 17:29:13 +0000 Updated: Mon, 6 Jan 2014 12:44:56 +0000

Status: Open
Project: Toolserver
Component/s: None
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Task Priority: Major
Reporter: drtrigon Assignee: Unassigned
Resolution: Unresolved Votes (Watches): 0 (2)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified
Environment:

Nightshade, linux


Issue Links:
Relation
is related to TS-1462 (n/a) (n/a)
is related to TS-1466 Check if pywikipedia can run on linux Resolved
is related to DRTRIGON-86 Test the re-write branch an decide what parts to migrate Closed

Description

DrTrigonBot scripts and helper do have various issues with the current linux environment.

  • needs python 2.7
    • as mentioned in TS-1466; TimedRotatingFileHandler
    • TS-1462; when will python 2.7 become stable (estimate?)
  • needs locale 'en_US'
  • (both systems should use same kind of OS; 32bit OR 64bit...

Comments
Comment by drtrigon [ Tue, 16 Oct 2012 17:39:08 +0000 ]

DaB could you please install locale 'en_US' to linux hosts also? Thanks and greetings!

In order to make this working:

drtrigon@nightshade:~$ python
Python 2.6.6 (r266:84292, Sep 13 2012, 17:02:19)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import locale
>>> locale.setlocale(locale.LC_TIME, 'en_US')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python2.6/locale.py", line 513, in setlocale
return _setlocale(category, locale)
locale.Error: unsupported locale setting

Comment by Tim.Landscheidt [ Tue, 16 Oct 2012 23:27:58 +0000 ]

Why don't you use en_US.UTF8?

Comment by drtrigon [ Wed, 17 Oct 2012 06:44:04 +0000 ]

several reasons;

  • if possible I do not want to change working code (and this is well tested on solaris boxes)
  • I do need it for LC_TIME only; so no need for UTF8
  • (I'm in a hurry
Comment by Tim.Landscheidt [ Wed, 17 Oct 2012 14:18:13 +0000 ]

Interesting:

[tim@passepartout ~]$ ssh yarrow.toolserver.org LANG=en_US.UTF8 date
Wed Oct 17 13:35:48 UTC 2012
[tim@passepartout ~]$ ssh willow.toolserver.org LANG=en_US date
Wednesday, October 17, 2012 01:36:00 PM UTC
[tim@passepartout ~]$

AFAIS, you use en_US only in pywikipedia/subster.py to parse dates in mails. Isn't that what is email.utils.parsedate()'s explicit purpose? It will be much more robust to any changes in system environment.

Comment by drtrigon [ Fri, 19 Oct 2012 17:48:07 +0000 ]

It looks like you found THE HOLY GRAIL! Thanks for this hint - I was not aware of this and implemented a very dubious piece of code...

I changed to 'email.utils' now! We have to wait some days to see if it works in every situation, but my test so far looked good...

Thanks for this!! (what about the others?

Greetings

Comment by Tim.Landscheidt [ Sat, 20 Oct 2012 03:15:53 +0000 ]

No problem. What do you mean by "the others"?

Comment by drtrigon [ Sat, 20 Oct 2012 07:35:22 +0000 ]

(Sorry I was thinking about "the other" issues, like e.g.:

  • when will python 2.7 become stable (estimate?)

This has to happen BEFORE all solaris hosts are gone, e.g. ...

Comment by Tim.Landscheidt [ Sat, 20 Oct 2012 17:38:56 +0000 ]

Well, according to http://packages.debian.org/wheezy/python, Python 2.7.3 will be shipped with Debian Wheezy, and that seems to be in beta testing now (cf. http://wiki.debian.org/DebianWheezy). I don't know if it is possible to mix packages from different distribution releases in Debian.

Comment by drtrigon [ Sat, 20 Oct 2012 18:52:08 +0000 ]

As I understand; mix=backport and there will be no backport of python 2.7 as mentioned by DaB in TS-1462. That's the reason why I am asking for the release date (estimate) or at least want to make clear that solaris hosts are needed until then...

Comment by drtrigon [ Sat, 10 Nov 2012 10:41:13 +0000 ]

Would it be possible to install locale 'de_DE'? That would be cool! Thanks

Comment by drtrigon [ Sun, 11 Nov 2012 11:57:56 +0000 ]

Just noticed (qacct) that e.g. my 'mainbot' SGE job uses twice the memory (~0.7-1GB) on the 64-bit linux hosts compared to (~0.35-0.5GB) on the 32-bit solaris machines!! According to reducing the footprint of python applications (wingolog) this might indeed be possible...

Comment by Tim.Landscheidt [ Thu, 19 Dec 2013 16:56:35 +0000 ]

DrTrigon, is this still an issue for you?





[TS-1534] Activate solaris cron in order to load balance the jobs reliable Created: Sat, 6 Oct 2012 17:19:38 +0000 Updated: Tue, 9 Oct 2012 19:56:04 +0000 Resolved: Tue, 9 Oct 2012 14:22:02 +0000

Status: Resolved
Project: Toolserver
Component/s: None
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: DaB.
Resolution: Won't Fix Votes (Watches): 0 (4)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified


Description

Since weeks DrTrigon bot scripts do notoriously fail and the situation seams to become worser an degenerate with time.

As Merlissimo mentioned several times the usage of solaris cron ("cron") instead of vixie cron ("cronie") would solve the problem or at least make it possible to configure cron in order to solve it.

At the moment solaris cron is not active on submit, could that be changed in order to allow users to migrate an solve the issue with not running cron(ie) jobs.

Thanks a lot and greetings


Comments
Comment by drtrigon [ Sat, 6 Oct 2012 17:21:42 +0000 ]

Sorry for simply assigning that to you - but whom else?

Comment by Tim.Landscheidt [ Sat, 6 Oct 2012 17:58:09 +0000 ]

Could you be a bit more specific of what is failing and how?

Comment by drtrigon [ Sat, 6 Oct 2012 18:22:58 +0000 ]

I discussed this wit DaB and Merlissimo several times. Essentially it looks like the jobs from vixie cron (cronie) do simply not get executed anymore. This started about 6 months ago (by that time sometimes 1 job a day was missed) became better then after adding RAM and started about 1 or 2 months ago to become really worse (constantly missing at least 1 job a day) to a point were now most of the days just 1 (of total 6 jobs) gets executed at all. In a few week my bots do not run anymore it this continues - and I am not able to migrate to Labs currently because of a lack in time... (nor to maintain my bots daily

It looks very like RAM and/or vixie cron do limit the number of executed jobs - which would not be tha case with solaris cron. Furthermore even delaying my jobs (e.g. from 0:00 to 0:07 and 0:11 later) did not help in any way. The really strange things here is it looks like I am the only user that experinces this behaviour. All discussions (with DaB and Merlissimo) lead me to the conclusion the it is simply the overload of the TS.

Merlissimo pointed out that the use of solaris cron should allow to delay jobs exceeding memory or other limits until other jobs are done and then run them later instead of simply forgetting them. This is a need for my bots since it is essential that I can count on the fact the do get executed ever day - the time (e.g. 5 mintues later) it is not of that high importance!

Do you need more discussion links (TS maillist) or some run statistics of my bots or anything other and more specific?

Greetings

Comment by Tim.Landscheidt [ Sat, 6 Oct 2012 19:48:01 +0000 ]

Do you use qcronsub from your crontab or do you directly execute your bot?

Comment by drtrigon [ Sat, 6 Oct 2012 20:38:24 +0000 ]

qcronsub (of course and according to qacct output it does not get executed by vixie cron (I should have mentioned that) and according to Merlissimo it's not an SGE issue either. (as far as I understand)

Comment by Tim.Landscheidt [ Sat, 6 Oct 2012 20:57:40 +0000 ]

AFAIUI, qcronsub doesn't submit a job if a job with the same name is already executing. Could that be an issue here?

To not further stab in the dark, could you provide a non-working example, i. e. which host you use for the cron(ie)tab and the line with the command that is failing?

Comment by drtrigon [ Sat, 6 Oct 2012 21:28:12 +0000 ]

No that is not problem here in fact I do use that as a feature for my infinitely running bot to trigger it once a day in case it crashed (what rarely happens - thus it is the only one really running at the moment).

Here my cronie tab file:

11 2 * * * qcronsub -l h_rt=12:00:00 -l virtual_free=500M -l arch=sol -N mainbot $HOME/pywikipedia/bot_control.py -default -cron
11 0 */14 * * qcronsub -l h_rt=02:00:00 -l virtual_free=100M -l arch=sol -N compbot $HOME/pywikipedia/bot_control.py -compress_history:[] -cron
11 0 * * * qcronsub -l h_rt=INFINITY -l virtual_free=200M -l arch=sol -N ircbot $HOME/pywikipedia/bot_control.py -subster_irc -cron
41 0 * * * qcronsub -l h_rt=06:00:00 -l virtual_free=200M -l arch=sol -N subster_frr $HOME/pywikipedia/bot_control.py -subster -cron -lang:frr
11 1 * * * qcronsub -l h_rt=06:00:00 -l virtual_free=200M -l arch=sol -N subster_en $HOME/pywikipedia/bot_control.py -subster -cron -lang:en
41 1 * * * qcronsub -l h_rt=06:00:00 -l virtual_free=200M -l arch=sol -N subster_nl $HOME/pywikipedia/bot_control.py -subster -cron -lang:nl
11 * * * * qcronsub -l h_rt=02:00:00 -l virtual_free=200M -l arch=sol -N subster_ar $HOME/pywikipedia/bot_control.py -subster -cron -lang:ar
11 0 * * * qcronsub -l h_rt=06:00:00 -l virtual_free=200M -l arch=sol -N subster_meta $HOME/pywikipedia/bot_control.py -subster -cron -family:meta -lang:

11 20 * * * qcronsub -l h_rt=04:00:00 -l virtual_free=500M -l arch=lx -N catimages $HOME/pywikipedia/bot_control.py -catimages -cron

11 0 * * * qcronsub -l h_rt=00:05:00 -l virtual_free=50M -l arch=sol -m as -j y -b y -N maintenance $HOME/warnuserquota.py >/dev/null

but as mentioned I did not change settings before it started to cause issues - afterwards I did move my jobs from :00 and :30 to :07 and then to :11 which did not help at all.

Comment by Tim.Landscheidt [ Sat, 6 Oct 2012 22:27:04 +0000 ]

Thanks. And where is that crontab installed? submit.toolserver.org? Is it installed as a crontab or a cronietab?

Comment by drtrigon [ Sat, 6 Oct 2012 22:40:19 +0000 ]

As mentioned in the initial post on 'submit' and it is 'vixie cron' = 'cronie'.

My wish here is to install 'solaris cron' = 'cron' in order to resolve that problem (according to Merlissimo).

Comment by Tim.Landscheidt [ Sat, 6 Oct 2012 23:34:33 +0000 ]

Usually, vixie-cron/cronie's executable is called "crontab". Toolserver has a special version called "cronietab" (cf. https://wiki.toolserver.org/view/Cronietab#Cronietab). Just to make sure: You installed it via "cronietab"?

submit.toolserver.org is idling most of the time. I'll take a look at 0:11Z how the load develops.

Comment by Tim.Landscheidt [ Sun, 7 Oct 2012 00:57:09 +0000 ]

Around 0:15Z the idling dropped for a short period slightly below 50 %, but otherwise it was always well beyond 90 %. Load averages at 0:15Z were 0.69, 0.25, 0.13 and at 0:20Z 0.03, 0.12, 0.11.

It requires a root to look at the logfiles, but for debugging you could prepend "qcronsub" with "echo" to see if the cron(ie)tab is at least active.

Comment by drtrigon [ Sun, 7 Oct 2012 08:35:03 +0000 ]

Thanks for your effort. Im am quite sure that DaB did look at the log files this at least once some time back...

DaB could you comment on this?

Comment by dab [ Sun, 7 Oct 2012 15:13:36 +0000 ]

We can not use solaris-cron for 2 reasons:
*Its syntax is other than cronie's (and user complained about that in the past) – but this would be no show-stopper.
*It's not cluster-aware and we need that (otherwise no job would run if one of the two submit-hosts fail or each job would run twice).
I will think about another solution but the best thing would be if someone with c(++)-experience would just write a patch for cronie – it can no be that hard to implement some kind of throttle or execution-list.

Comment by drtrigon [ Sun, 7 Oct 2012 16:33:53 +0000 ]

DaB could you may be talk to Merlissimo? I am not an expert here, but as I understod him solaris cron should work well together with SGE...

Anyway the other solution you suggested (patch for cronie) would be fine for me as well! Just - please - solve this issue. If I can help with my basic C/C++ knowledge - let me know!

Thanks a lot for your time and greetings!!

Comment by Tim.Landscheidt [ Sun, 7 Oct 2012 18:44:40 +0000 ]

Well, before we add throttling to cronie (and my own experience with the maintainers isn't very promising with regard to adding new features), we should perhaps first find out if there really is a problem that we need a solution for. So what do the log files actually say?

Comment by nosy [ Sun, 7 Oct 2012 20:46:50 +0000 ]

As far as I see cronie and solaris cron are both running on submit. The solaris cron can be reached via calling crontab, cronietab is edited via command cronie.

root@clematis:~# cronie -l -u drtrigon

  1. DO NOT EDIT THIS FILE - edit the master and reinstall.
  2. (/tmp/crontab.ji5t0i/crontab installed on Wed Dec 23 11:02:53 2009)
  3. (Cron version – $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
  4. m h dom mon dow command

##11 2 * * * qcronsub -l h_rt=12:00:00 -l virtual_free=500M -m as -j y -b y -N mainbot $HOME/pywikipedia/bot_control.py -default -cron
#11 2 * * * qcronsub -l h_rt=12:00:00 -l virtual_free=500M -N mainbot $HOME/pywikipedia/bot_control.py -default -cron
11 2 * * * qcronsub -l h_rt=12:00:00 -l virtual_free=500M -l arch=sol -N mainbot $HOME/pywikipedia/bot_control.py -default -cron
#11 0 */14 * * qcronsub -l h_rt=02:00:00 -l virtual_free=100M -N compbot $HOME/pywikipedia/bot_control.py -compress_history:[] -cron
11 0 */14 * * qcronsub -l h_rt=02:00:00 -l virtual_free=100M -l arch=sol -N compbot $HOME/pywikipedia/bot_control.py -compress_history:[] -cron
#11 0 * * * qcronsub -l h_rt=INFINITY -l virtual_free=200M -l arch=* -N ircbot $HOME/pywikipedia/bot_control.py -subster_irc -cron
#11 0 * * * qcronsub -l h_rt=INFINITY -l virtual_free=200M -N ircbot $HOME/pywikipedia/bot_control.py -subster_irc -cron
11 0 * * * qcronsub -l h_rt=INFINITY -l virtual_free=200M -l arch=sol -N ircbot $HOME/pywikipedia/bot_control.py -subster_irc -cron
#41 0 * * * qcronsub -l h_rt=06:00:00 -l virtual_free=200M -N subster_frr $HOME/pywikipedia/bot_control.py -subster -cron -lang:frr
41 0 * * * qcronsub -l h_rt=06:00:00 -l virtual_free=200M -l arch=sol -N subster_frr $HOME/pywikipedia/bot_control.py -subster -cron -lang:frr
#11 1 * * * qcronsub -l h_rt=06:00:00 -l virtual_free=200M -N subster_en $HOME/pywikipedia/bot_control.py -subster -cron -lang:en
11 1 * * * qcronsub -l h_rt=06:00:00 -l virtual_free=200M -l arch=sol -N subster_en $HOME/pywikipedia/bot_control.py -subster -cron -lang:en
#41 1 * * * qcronsub -l h_rt=06:00:00 -l virtual_free=200M -N subster_nl $HOME/pywikipedia/bot_control.py -subster -cron -lang:nl
41 1 * * * qcronsub -l h_rt=06:00:00 -l virtual_free=200M -l arch=sol -N subster_nl $HOME/pywikipedia/bot_control.py -subster -cron -lang:nl
#11 * * * * qcronsub -l h_rt=02:00:00 -l virtual_free=200M -N subster_ar $HOME/pywikipedia/bot_control.py -subster -cron -lang:ar
11 * * * * qcronsub -l h_rt=02:00:00 -l virtual_free=200M -l arch=sol -N subster_ar $HOME/pywikipedia/bot_control.py -subster -cron -lang:ar
#11 0 * * * qcronsub -l h_rt=06:00:00 -l virtual_free=200M -N subster_meta $HOME/pywikipedia/bot_control.py -subster -cron -family:meta -lang:
11 0 * * * qcronsub -l h_rt=06:00:00 -l virtual_free=200M -l arch=sol -N subster_meta $HOME/pywikipedia/bot_control.py -subster -cron -family:meta -lang:

11 20 * * * qcronsub -l h_rt=04:00:00 -l virtual_free=500M -l arch=lx -N catimages $HOME/pywikipedia/bot_control.py -catimages -cron

#11 0 * * * qcronsub -l h_rt=00:05:00 -l virtual_free=50M -m as -j y -b y -N maintenance $HOME/warnuserquota.py >/dev/null
11 0 * * * qcronsub -l h_rt=00:05:00 -l virtual_free=50M -l arch=sol -m as -j y -b y -N maintenance $HOME/warnuserquota.py >/dev/null

root@clematis:~# crontab -l drtrigon
crontab: can't open your crontab file.

Please try to move your cron jobs. Does this work better?

Comment by dab [ Sun, 7 Oct 2012 21:10:39 +0000 ]

No, please do NOT that. The crontab is NOT cluster-aware the jobs would not be executed in a case of a server-error.

Comment by dab [ Mon, 8 Oct 2012 01:04:47 +0000 ]

I added now a simple throttle (a simple sleep) to our cronie. AFAIS there are no more memory-problems and no failing task – but I have to look a hole day to be sure.

Comment by drtrigon [ Mon, 8 Oct 2012 09:07:51 +0000 ]

DaB IF this truly works you are the preserver of my believe in TS and a worthy hero! May I spend you a pan galactic gargle blaster sometime?!

Back to real bussiness - since you wrote this comment on 01:04:47 it was in the middle of the bunch of my tasks running. So I have to wait another day for a useful insight. May be more than 1 day would be better to watch and since I do have time the next weekend I will report back within 1 week from now - if this ok for you? (at the moment it looks like 5 of 6 run today - lots better than before)

Meanwhile; thanks a lot to all involved here: DaB, Tim, Nosy and of course Merlissimo and giftpflanze (and all those I forgot now! - sorry

Comment by dab [ Tue, 9 Oct 2012 14:22:02 +0000 ]

I close this bug now, because solaris cron can not be used and I patched cronie. If cronie fails again, please open a new bug for that. Thanks for your patience.

Comment by drtrigon [ Tue, 9 Oct 2012 19:56:04 +0000 ]

I have to thank for your work!! I hope everything works all right, if not I will do as you said!





[TS-1466] Check if pywikipedia can run on linux Created: Fri, 27 Jul 2012 13:35:21 +0000 Updated: Tue, 16 Oct 2012 17:37:58 +0000 Resolved: Thu, 13 Sep 2012 16:07:37 +0000

Status: Resolved
Project: Toolserver
Component/s: None
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Task Priority: Major
Reporter: DaB. Assignee: DaB.
Resolution: Fixed Votes (Watches): 2 (5)
Labels: linux, pywikipedia,
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Depends
depends on TS-1452 (n/a) (n/a)
Relation
is related to TS-1449 Install libraries and python bindings needed for catimages bot Open
is related to TS-1554 Check if DrTrigonBot (framework) can run on linux Open

Description

drtrigon reported in TS-1449 that the pywikipedia-framework is still complaining about our python-version. While I'm sure that I patched python correctly (and the patched version IS running), I will speak with a pywikipedia-dev where the problem is.


Comments
Comment by valhallasw [ Fri, 27 Jul 2012 15:03:01 +0000 ]

Python issue item: http://bugs.python.org/issue10254

The bug is still current on nightshade:

valhallasw@nightshade:~$ cat > unicodetest.py
import sys, unicodedata
print repr(sys.version),

teststr = u'\u092e\u093e\u0930\u094d\u0915 \u091c\u093c\u0941\u0915\u0947\u0930\u092c\u0930\u094d\u0917'
if unicodedata.normalize('NFC', teststr) == teststr:
print 'OK'
else:
print 'FAIL'
valhallasw@nightshade:~$ python unicodetest.py
'2.6.6 (r266:84292, Jul 11 2012, 14:46:24) \n[GCC 4.4.5]' FAIL
valhallasw@nightshade:~$ python2.5 unicodetest.py
'2.5.5 (r255:77872, Nov 28 2010, 16:43:48) \n[GCC 4.4.5]' OK

Comment by russell [ Tue, 7 Aug 2012 20:30:54 +0000 ]

If python on Linux could be upgraded to version 2.7, that would solve this bug...

Comment by dab [ Sun, 12 Aug 2012 19:13:58 +0000 ]

Ok, then I have to rebuild python 2.6. We need the build-host first, so I add the dependency.

Comment by dab [ Thu, 13 Sep 2012 16:07:37 +0000 ]

Should be working now.

Comment by drtrigon [ Wed, 19 Sep 2012 17:23:21 +0000 ]

I think this bug should be re-opened since code using:

import re
re.sub(..., flags=...)

does not work after the downgrade to python 2.6.6 anymore because the optional parameter 'flags' was introduced in python 2.7.

Comment by valhallasw [ Sat, 6 Oct 2012 10:53:51 +0000 ]

If this is in pywikipedia code, file a bug @ sf.net, as pywikipedia should be 2.6-compatible.

If it's in your own code, use
re.compile(..., flags=...).sub(...)
instead.

Comment by drtrigon [ Tue, 9 Oct 2012 15:21:59 +0000 ]

Yes I was considering that patch - and applied it now. I did also report the occurences I found in pywikipedia.

Thanks so far and greetings!

Comment by drtrigon [ Thu, 11 Oct 2012 17:36:43 +0000 ]

Could somebody confirm if logging.handlers.TimedRotatingFileHandler has issues in python 2.6? I am not sure and not able to find good info about this on the web.

Edit: Can confirm it myself now; have a look at http://bugs.python.org/issue8117 (which was NOT merged into 2.6!)





[TS-1449] Install libraries and python bindings needed for catimages bot Created: Mon, 16 Jul 2012 18:32:29 +0000 Updated: Mon, 6 Jan 2014 12:41:20 +0000

Status: Open
Project: Toolserver
Component/s: Software installation,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Task Priority: Major
Reporter: drtrigon Assignee: Unassigned
Resolution: Unresolved Votes (Watches): 0 (3)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Attachments: Fileout     
Issue Links:
Depends
depends on TS-1452 (n/a) (n/a)
Relation
is related to DRTRIGON-120 New bot script/part for opencv image recognition and classification Closed
is related to TS-1360 Install OpenCV libraries and python bindings Resolved
is related to TS-1466 Check if pywikipedia can run on linux Resolved

Description

In the course of writing a new bot script in order to categorize commons images by computer vision (face detection, ...) the installation of several libraries (with C++ headers and python bindings) is essential.

Here a list of some of them:

  • "exiftool" the executable (NOT the perl module)
  • and I need the libraries missing here:
cd /home/drtrigon/pywikipedia/jseg/
make segdist_cpp.so

the error output from g++ is way to long, I was not able to get to the
top (more than 4096 lines!) and look at the origin of the errors, but
I assume its related to one of those (from makefile):

-ljpeg -lm -lboost_python
`pkg-config --libs --cflags python`
(the last returns: No package 'python' found)

Thank you very much!
Greetings
DrTrigon


Comments
Comment by dab [ Mon, 16 Jul 2012 19:23:37 +0000 ]

Installed "libimage-exiftool-perl". While it has "perl" in the name, there is /usr/bin/exiftool in it what should be what you need.

Comment by drtrigon [ Mon, 16 Jul 2012 20:13:37 +0000 ]

This works like a charm (tested by accident - thanks!

What about the others? E.g. python-dev(el) may be something with python-boost too...?

Comment by dab [ Mon, 16 Jul 2012 21:09:47 +0000 ]

libboost-python1.42-dev,libboost-python1.42.0 and libpython2.6 are already installed so that is not the problem. Can you redirect the output of the compilation-try to a file?

Comment by drtrigon [ Tue, 17 Jul 2012 16:35:12 +0000 ]

Might there be an issue with pkg-config? When running it, I get this:

drtrigon@nightshade:~/pywikipedia/jseg$ pkg-config --libs --cflags python
Package python was not found in the pkg-config search path.
Perhaps you should add the directory containing `python.pc'
to the PKG_CONFIG_PATH environment variable
No package 'python' found

(this is why I assumed something with a python package to be wrong...

Comment by drtrigon [ Tue, 17 Jul 2012 16:38:51 +0000 ]

Here the result of running:

make segdist_cpp.so &> out
Comment by dab [ Tue, 17 Jul 2012 18:41:33 +0000 ]

Please have a look at http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=632393 .

Comment by drtrigon [ Tue, 17 Jul 2012 21:09:57 +0000 ]

Ok... I looked at the link and hope I got the point...

I did following test; I replaced

`pkg-config --libs --cflags python`

in the makefile by $(CONF) which is defined as

PYTHON = `pyversions -d`
CONF = -I/usr/include/$(PYTHON)  -l$(PYTHON)

which is from your link and nicely resolves to the same like on my machine. So I am glad to report that the compilation works seamless now - we are converging against the end! Thanks for that.

Now I need the python module pydmtx (which is again a bit exotic but I hope in the repos). And zbar together with zbar-dev(el) inorder to get the header zbar.h.

EDIT: this is still suboptimal since pyversions is specific to debian systems only, so use

PYTHON = python`python -V 2>&1 | grep -o '[0-9]*\.[0-9]*' | head -n 1`
Comment by drtrigon [ Wed, 18 Jul 2012 11:53:06 +0000 ]

...and the python module rsvg is needed as well.

Comment by dab [ Wed, 18 Jul 2012 13:43:27 +0000 ]

zbar: libzbar-dev, libzbar0, python-zbar and zbar-tools?
rsvg: python-rsvg?
pydmtx: No packages at the moment, we have to wait for the build-host

Comment by drtrigon [ Wed, 18 Jul 2012 15:53:16 +0000 ]
  • zbar: Yes! All of them, please! (my os fedora does not have python-zbar I built it by myself, that's great! so we could try just installing python-zbar and I try to use it directly - may be no need for zbar.h)
  • rsvg: Yes! (I think it is the right one, on my system it was gnome-python2-rsvg)
  • pydmtx: What a pitty! May I build it myself? (it looks like I really NEED this build-host...
Comment by dab [ Wed, 18 Jul 2012 19:23:25 +0000 ]

Installed them. You can compile pydmtx yourself as a temporary solution, but you are fully responsible for it than.

Comment by drtrigon [ Wed, 18 Jul 2012 20:26:55 +0000 ]

Thanks a lot!!! Looks good!

In order to compile pydmtx I would need libdmtx, libdmtx-dev(el) packages containing the library and header for simplicity. Are they available?

Comment by drtrigon [ Thu, 19 Jul 2012 12:48:05 +0000 ]

Another important library I forgot, since it is installed on willow is pysvn.

Comment by dab [ Thu, 19 Jul 2012 13:10:00 +0000 ]

Installed libdmtx-dev, libdmtx0a and python-svn.

Comment by drtrigon [ Thu, 19 Jul 2012 17:11:55 +0000 ]

Thanks a lot!! (pydmtx compiles now too)

All modules and libraries needed seem to be up and running.

I've tested the bot code now:

  1. it runs,
  2. it seems to run like and yield the same results as on my machine so far and
  3. I was able to get it running through SGE as well!!!

Today is THE DAY!! )

Thanks a lot again for your enduring patience and support during this setup task!! I am deeply indebted to you now! I would like to offer you a Pan-Galactic Gargle Blaster or something more digestible - as you wish! Seriously let me know if there is something I can do!

Comment by drtrigon [ Fri, 20 Jul 2012 21:29:19 +0000 ]

Is in my oppinion this ticked could be considered as resolved or closed.

But since it is not I have another question; what about upgrading python to 2.7.2+? Since the pywikipedia framework nags me all the time with:

================================================================================
WARNING: your python version might trigger issue #3081100***
More information: See https://sourceforge.net/support/tracker.php?aid=3081100
Please update python to 2.7.2+ if you are running on wikimedia sites!***
================================================================================

(or may be use the patch that River? created some time ago for the other machines...

Good night and good luck!

Comment by dab [ Sun, 22 Jul 2012 20:47:20 +0000 ]

We will not move to python 2.7 until Debian Wheezy becomes stable (somewhen next year). Our python 2.6 HAS the patch included that fix the unicode-problem.

Comment by drtrigon [ Mon, 23 Jul 2012 09:19:45 +0000 ]

Ok then we have to wait until next year!

Now that's strange, when looking at the pywikipedia source: wikipedia.py line 8142 and 8242 ("# TEST for bug #3081100")
...you can see that there is a check for the unicode issue (not the python version) so the message should be suppressed if the the unicode-problem is solved sufficiently...!? (and now I am confused...

Greetings and thanks!

Comment by drtrigon [ Tue, 24 Jul 2012 21:35:04 +0000 ]

Do you mind if would ask you to install 'pdftotext' binary executable from poppler library?

Thanks!

Comment by dab [ Thu, 26 Jul 2012 15:05:24 +0000 ]

I will speak with the pywikipedia-guys where the problem is. I have mark poppler-utils for installation, should be done during the next hour.

Comment by drtrigon [ Sun, 29 Jul 2012 16:09:15 +0000 ]

Thanks for both!! I will watch the other discussion/ticket/maillist as well...

Comment by drtrigon [ Sun, 2 Sep 2012 11:37:04 +0000 ]

Hello DaB! (I'm back again

Would it be possible to install 'libmagic' (python magic module) and 'ffprobe' (from ffmpeg) additionally? That would be great too!

Thanks so far! Greetings

Comment by dab [ Mon, 3 Sep 2012 15:06:44 +0000 ]

"python-magic - File type determination library using "magic" numbers (Python bindings)" this one? ffprobe is marked for installation.

Comment by drtrigon [ Mon, 3 Sep 2012 15:52:36 +0000 ]

Yes exactly!! Thanks a lot!!! Greetings

Comment by dab [ Tue, 4 Sep 2012 13:46:21 +0000 ]

Fixed with rev. 16. Should be installed within 1h.

Comment by drtrigon [ Tue, 4 Sep 2012 14:47:32 +0000 ]

Fixed? rev. 16? I'm confused now...

Comment by dab [ Wed, 5 Sep 2012 12:46:55 +0000 ]

It was installed. "Rev. 16" means the puppet-history-entry (I will better post a link next time).

Comment by drtrigon [ Mon, 10 Sep 2012 20:16:58 +0000 ]

Is there a more recent version 'ffprobe' available? One with support of "-print_format json" is needed, e.g. something around 0.10? (the one installed was built on Jan 30 2010)

And 'pdfimages' supporting "-p" would also be very useful.

Comment by drtrigon [ Thu, 13 Sep 2012 22:46:44 +0000 ]

The new 'ffprobe' from 'libav-tools-package' is clearly better (thanks!) - I can works with this for the moment.





[TS-1402] Cronie jobs intermittently failing to run Created: Mon, 11 Jun 2012 14:09:09 +0000 Updated: Mon, 10 Sep 2012 14:18:09 +0000 Resolved: Sat, 14 Jul 2012 22:08:06 +0000

Status: Resolved
Project: Toolserver
Component/s: General/Unknown,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: Russell Blau Assignee: Unassigned
Resolution: Fixed Votes (Watches): 1 (4)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified
Environment:

clematis



Description

See issue TS-1340; the exact same symptoms are occurring again since 3 June. Some cronie jobs run, some don't, and the set that fails is different each day. Following is the list of jobs that were supposed to run each day, and whether they did or did not:

3 June 07:30 UTC: NO
3 June 08:00 UTC: Yes
3 June 09:00 UTC: Yes
3 June 09:30 UTC: Yes
4 June 07:30 UTC: Yes
4 June 08:00 UTC: Yes
4 June 09:00 UTC: Yes
4 June 09:30 UTC: Yes
5 June 06:30 UTC: Yes
5 June 07:30 UTC: Yes
5 June 08:00 UTC: NO
5 June 09:00 UTC: NO
5 June 09:30 UTC: Yes
6 June 07:30 UTC: NO
6 June 08:00 UTC: NO
6 June 09:00 UTC: Yes
6 June 09:30 UTC: Yes
6 June 09:45 UTC: Yes
7 June 07:30 UTC: NO
7 June 08:00 UTC: Yes
7 June 09:00 UTC: NO
7 June 09:30 UTC: Yes
7 June 09:45 UTC: Yes
8 June 07:30 UTC: Yes
8 June 08:00 UTC: NO
8 June 09:00 UTC: NO (see note below)
8 June 09:30 UTC: Yes
8 June 09:45 UTC: Yes
9 June 07:30 UTC: NO
9 June 08:00 UTC: NO
9 June 09:00 UTC: NO
9 June 09:30 UTC: Yes
9 June 09:45 UTC: Yes
10 June 07:30 UTC: NO
10 June 08:00 UTC: Yes
10 June 09:00 UTC: Yes
10 June 09:30 UTC: Yes
11 June 07:30 UTC: NO
11 June 08:00 UTC: NO
11 June 09:00 UTC: Yes
11 June 09:30 UTC: Yes

The only pattern I can see is that the failures all appear to occur later than 06:30 and earlier than 09:30.

The jobs that did not run appear not to have run at all (no output or error files generated), and there were no e-mail error messages, EXCEPT for 8 June 09:00 UTC when I received the email pasted below:

Return-Path: <root@toolserver.org>
From: root@toolserver.org (Cron Daemon)
To: russell@toolserver.org
Subject: Cron <russell@hawthorn> qcronsub -l h_rt=4:00:00 -l virtual_free=500M -N en_cr -o $HOME/en_cr.out -e $HOME/en_cr.err $HOME/pywiki-rewrite/scripts/category_redirect.py
Content-Type: text/plain; charset=646
Auto-Submitted: auto-generated
X-Cron-Env: <SHELL=/bin/sh>
X-Cron-Env: <HOME=/home/russell>
X-Cron-Env: <PATH=/opt/local/bin:/opt/ts/gnu/bin:/opt/ts/bin:/opt/ts/mysql/5.1/bin:/opt/ts/perl/5.10/bin:/opt/ts/python/2.6/bin:/opt/ts/php/5.3/bin:/opt/ts/ruby/1.9/bin:/opt/ts/mono/2.0/bin:/opt/ts/tcl/8.5/bin:/usr/ccs/bin:/sge62/bin/sol-amd64:/usr/bin:/usr/sbin:/usr/sfw/bin:/usr/postgres/8.3/bin:/opt/jobserver/bin>
X-Cron-Env: <LOGNAME=russell>
Message-Id: <20120608090008.5A55634F00@hawthorn.toolserver.org>
Date: Fri, 8 Jun 2012 09:00:08 +0000 (UTC)

Unable to run job: failed sending gdi request.
Exiting.


Comments
Comment by drtrigon [ Wed, 20 Jun 2012 20:43:02 +0000 ]

I do experience similar issues, I will add an exact list of missed jobs later when I have time to investigate this further.

I addressed this issue already on the maillist, please confer [1] and all its follow-up.
[1] http://lists.wikimedia.org/pipermail/toolserver-l/2012-June/005020.html

Comment by drtrigon [ Sun, 24 Jun 2012 22:07:06 +0000 ]

So now I spent a few hours investigating this mess, here my results.

Daily jobs (run once per day):

Jun 05; OK
Jun 06; OK
Jun 07; "mainbot" not executed
Jun 08; OK
Jun 09; OK
Jun 10; OK
Jun 11; OK
Jun 12; OK
Jun 13; "subster_en" not executed
Jun 14; OK
Jun 15; "subster_en" not executed
Jun 16; "subster_en" not executed
Jun 17; "mainbot" not executed
Jun 18; "maintenance", "mainbot" not executed
Jun 19; "subster_nl", "subster_frr" not executed
Jun 20; "subster_meta" not executed
Jun 21; "subster_en" not executed
Jun 22; OK
Jun 23; "mainbot" not executed
Jun 24; "subster_nl", "mainbot" not executed

Hourly job "subster_ar" (runs once per hour):

Jun 05; not executed at 12:00, 22:00
Jun 06; not executed at 13:00
Jun 07; not executed at 12:00, 16:00
Jun 08; not executed at 03:00, 07:00, 09:00, 21:00
Jun 09; not executed at 04:00, 08:00, 23:00
Jun 10; not executed at 00:00, 09:00
Jun 11; not executed at 01:00, 03:00, 08:00, 23:00
Jun 12; not executed at 02:00, 03:00, 04:00, 08:00, 14:00
Jun 13; not executed at 00:00, 10:00, 11:00, 14:00, 18:00
Jun 14; not executed at 00:00, 10:00, 15:00
Jun 15; not executed at 00:00, 02:00, 08:00, 16:00, 22:00
Jun 16; not executed at 00:00, 06:00, 17:00
Jun 17; not executed at 17:00, 21:00
Jun 18; not executed at 21:00, 23:00
Jun 19; not executed at 01:00, 09:00, 18:00, 22:00
Jun 20; OK
Jun 21; not executed at 00:00, 04:00, 10:00, 12:00, 14:00
Jun 22; not executed at 03:00, 06:00, 19:00
Jun 23; not executed at 01:00, 16:00, 18:00, 22:00
Jun 24; not executed at 00:00

Checking for error mails related to them, gave just ONE hit:

Jun 12;

Job 2141994 (mainbot) Aborted
Exit Status = 137
Signal = KILL
User = drtrigon
Queue = medium-sol@wolfsbane.toolserver.org
Host = wolfsbane.toolserver.org
Start Time = 06/11/2012 20:23:13
End Time = 06/11/2012 23:06:30
CPU = 00:38:01
Max vmem = 301.402M
failed assumedly after job because:
job 2141994.1 died through signal KILL (9)

Job 2141994 (mainbot) was killed by drtrigon@willow.toolserver.org

...BUT I got a lot of others mail messages (like the one below) even
thought THAT jobs runned successful...

Jun 07;

Job 2128237 (subster_meta) Aborted
Exit Status = -1
Signal = unknown signal
User = drtrigon
Queue = medium-sol@willow.toolserver.org
Host = willow
Start Time = <unknown>
End Time = <unknown>
CPU = NA
Max vmem = NA
failed before prolog because:
shepherd exited with exit status 7: before prolog

Jun 21;

Job 2177842 (subster_ar) Aborted
Exit Status = -1
Signal = unknown signal
User = drtrigon
Queue = longrun-sol@willow.toolserver.org
Host = willow
Start Time = <unknown>
End Time = <unknown>
CPU = NA
Max vmem = NA
failed assumedly before job because:
fork failed: Not enough space

Jun 22;

Job 2181872 (ircbot) Aborted
Exit Status = -1
Signal = unknown signal
User = drtrigon
Queue = longrun-sol@willow.toolserver.org
Host = willow
Start Time = <unknown>
End Time = <unknown>
CPU = NA
Max vmem = NA
failed before writing exit_status because:
shepherd exited with exit status 19: before writing exit_status

I have NO IDEA what could be the cause of this, but it looks to me that
it is getting worse (from week to week)... any hint would be greatly appreciated!

Thanks a lot and greetings

Comment by merlissimo [ Sun, 24 Jun 2012 22:52:52 +0000 ]
  • 2141994 (mainbot) was deleted by user
  • 2128237 and 2177842 on willow: sge itself failed because of memory problems (not enough space). After sge control was up again no such expected process was found. (in both cases this was the only pid not found, pid of all other jobs were still alive)
    **This should be wolved by RAM upgrade
  • 2181872 was killed by restart of willow and no rerun flag was set.
Comment by drtrigon [ Mon, 25 Jun 2012 09:34:29 +0000 ]

The point is; since about 2 week my bots do not run reliable anymore (and it is random)!!! In fact this means I have to check every single day which bots did run or not, and then start them by hand. I do not have time, motivation or ressources to do this... So in fact; either we are able to solve this or my bot does go down, at least until the issue is solved... In my eyes this is no real choice...

Comment by drtrigon [ Mon, 25 Jun 2012 12:39:58 +0000 ]

You can also confer the following graph:

http://toolserver.org/~drtrigon/cgi-bin/panel.py?action=logstat&format=graph-ecount

"subster" has to be runned 5 times... EVERY day!

Comment by drtrigon [ Mon, 9 Jul 2012 13:06:44 +0000 ]

...so no replies no progress here at all?

From my side I can report a vague sign of rising hope that I may work out "magically". If you take a look a t the link given in the previous comment, you can see all jobs/bots worked now for about 1 week as expected. May be this continues...?

Greetings

Comment by dab [ Sat, 14 Jul 2012 22:08:06 +0000 ]

The memory-update should have fix that. If not, please re-open.

Comment by saper [ Sun, 15 Jul 2012 08:58:17 +0000 ]

I think I see this problem today as well. I programmed by job to start every hour to investigate.

Comment by russell [ Mon, 10 Sep 2012 14:18:09 +0000 ]

Please re-open this issue; cron jobs once again are failing to run, intermittently and unpredictably. This has affected both my personal account (russell) and an MMP I belong to (dpl). The problem is less frequent than before but still occurring.





[TS-1360] Install OpenCV libraries and python bindings Created: Sat, 21 Apr 2012 09:25:41 +0000 Updated: Mon, 16 Jul 2012 18:34:01 +0000 Resolved: Sat, 14 Jul 2012 22:06:22 +0000

Status: Resolved
Project: Toolserver
Component/s: Software installation,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Task Priority: Major
Reporter: drtrigon Assignee: DaB.
Resolution: Fixed Votes (Watches): 0 (2)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to DRTRIGON-120 New bot script/part for opencv image recognition and classification Closed
is related to TS-1449 Install libraries and python bindings needed for catimages bot Open
URL: http://commons.wikimedia.org/wiki/User:Multichill/Using_OpenCV_to_categorize_files

Description

In the course of writing a new bot script in order to categorize commons images by computer vision (face detection, ...) the installation of OpenCV library with C++ headers and python bindings is essential.

Here a list:

1) python bindings in order to support "import cv" and "import cv2"

2) C++ headers for python and OpenCV in order to be able to compile (g++) code with:

$ pkg-config --libs --cflags opencv python
-I/usr/include/opencv -I/usr/include/python2.7 -lopencv_core -lopencv_imgproc -lopencv_highgui -lopencv_ml -lopencv_video -lopencv_features2d -lopencv_calib3d -lopencv_objdetect -lopencv_contrib -lopencv_legacy -lopencv_flann -lpython2.7

Thanks a lot and greetings
DrTrigon


Comments
Comment by drtrigon [ Sat, 21 Apr 2012 09:27:52 +0000 ]

Toolserver software installation (libraries missing) needed prior to the first bot run on ts.

Comment by drtrigon [ Fri, 27 Apr 2012 22:07:45 +0000 ]

Bot request is open: http://commons.wikimedia.org/wiki/Commons:Bots/Requests/DrTrigonBot

Comment by drtrigon [ Fri, 27 Apr 2012 22:08:14 +0000 ]

Hope you can help here - thanks in advance!

Comment by dab [ Fri, 22 Jun 2012 13:25:53 +0000 ]

Installed the followng packages at the linux-boxes for a start:
*python-opencv
*libcv2.1

Comment by drtrigon [ Sun, 24 Jun 2012 23:08:44 +0000 ]

Which server shall I use? Tried 'yarrow' is that ok?

drtrigon@yarrow:~/pywikipedia$ python
Python 2.6.6 (r266:84292, Dec 26 2010, 22:31:48)
[GCC 4.4.5] on linux2
Type "help", "copyright", "credits" or "license" for more information.

(this python version is quite old, at least 2.7.2 should be used with pywikipedia framework)

>>> import cv

...so cv python bindings are ok (thanks!)...

>>> import cv2
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named cv2

...but cv2 bindings are missing...

>>> import Image
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: No module named Image

...as well as PIL (python imaging module) too.

Can you please add 'cv2' and 'Image' bindings as a next step? - Thanks!

Edit: Additionally 'Image' (PIL) and may be 'scipy' please.

Comment by dab [ Sat, 14 Jul 2012 22:06:22 +0000 ]

Installed python-imaging; there is also "python-imaging-tk", reopen if you need that too. python-scipy was already installed. For cv2 there is no debian-package at the moment, so you have to live with version 1 for the moment.





[TS-1317] over-quota warn-mail missing / does not work Created: Mon, 27 Feb 2012 19:26:05 +0000 Updated: Mon, 23 Dec 2013 04:36:36 +0000

Status: Open
Project: Toolserver
Component/s: None
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: Marlen Caemmerer
Resolution: Unresolved Votes (Watches): 0 (2)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Duplicate
is a duplicate of TS-618 (n/a) (n/a)
Relation
is related to TS-1313 More space - double quota for drtrigon Resolved

Description

As mentioned in TS-1313 by DaB. "AFAIK there is a warn-mail, please check your inbox". Since I assume those mails are sent to the same address than all other TS messages (SGE, cron, ...), I have to report that I never got such a message or notification in the past.


Comments
Comment by drtrigon [ Mon, 27 Feb 2012 19:26:56 +0000 ]

As mentioned there "If you find no mail, please open another bugreport."

Comment by drtrigon [ Sat, 3 Mar 2012 10:26:03 +0000 ]

Follow up to TS-1313 as mentioned.

Comment by drtrigon [ Mon, 16 Jul 2012 14:34:27 +0000 ]

http://lists.wikimedia.org/pipermail/toolserver-l/2012-July/005092.html

Comment by Tim.Landscheidt [ Thu, 19 Dec 2013 15:06:00 +0000 ]

Nosy, I assume with a look at TS-618 that this issue is fixed as well?





[TS-1313] More space - double quota for drtrigon Created: Sun, 26 Feb 2012 19:15:30 +0000 Updated: Mon, 27 Feb 2012 19:26:56 +0000 Resolved: Mon, 27 Feb 2012 15:28:06 +0000

Status: Resolved
Project: Toolserver
Component/s: Accounts,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Task Priority: Major
Reporter: drtrigon Assignee: DaB.
Resolution: Fixed Votes (Watches): 0 (0)
Labels: quota, user,
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to TS-1317 over-quota warn-mail missing / does not work Open

Description

According to [1] I would like to request a doubled quota for my account. I do not plan to use the whole ~500MB but more than 250MB from time to time. I would like to prevent the bot and tools from beeing blocked because of me not logging in for more than 7 days...

[1] https://wiki.toolserver.org/view/Disk_quota

Btw.: Is there a posiblity to get informed/warned on the exceeded quota by mail? e.g. by using 'warnquota' [2]?

[2] http://linux.die.net/man/8/warnquota

Thanks a lot and greetings


Comments
Comment by dab [ Mon, 27 Feb 2012 15:28:06 +0000 ]

Increased soft to 500MB, hard to 1GB. AFAIK there is a warn-mail, please check your inbox. If you find no mail, please open another bugreport.





[TS-1303] Install 'lunatic-python' (together with 'lua') Created: Sun, 19 Feb 2012 17:25:37 +0000 Updated: Sun, 23 Dec 2012 22:02:16 +0000 Resolved: Sun, 9 Sep 2012 19:38:08 +0000

Status: Resolved
Project: Toolserver
Component/s: Software installation,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Task Priority: Major
Reporter: drtrigon Assignee: DaB.
Resolution: Fixed Votes (Watches): 0 (2)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Depends
depends on TS-1452 (n/a) (n/a)
Relation
is related to DRTRIGON-118 Enable lua support to enable even better template programming inclusion (subster) Closed
Sub-Tasks:
Key
Summary
Type
Status
Assignee
TS-1583 Install lua dev(el) package(s) Sub-task Open Unassigned
URL: http://labix.org/lunatic-python

Description

Please install lunatic-python on the servers. As mentioned in DRTRIGON-118 the mw software will include lua at some point, thus it would be useful to be able to use it in bots also.

Thanks and Greetings


Comments
Comment by dab [ Wed, 18 Jul 2012 13:48:44 +0000 ]

There is a "python-lua"-package in testing, but not in stable. I will try to backport it, when we get a build-host.

Comment by drtrigon [ Wed, 18 Jul 2012 16:47:17 +0000 ]

Thanks a lot! That's very nice!! I'm not sure; I think it's an other package then the one mentioned (and tested), but forget about that we have to take what we get... And it is worth a try.

Meanwhile we could install lua (with libs) anyway since it a dependency?

Thanks and greetings

Comment by dab [ Wed, 18 Jul 2012 19:15:15 +0000 ]

Installed lua5.1. Also the following libraries:
*liblua5.1-lpeg2
*liblua5.1-socket2
*liblua5.1-sql-mysql-2
*liblua5.1-zip0
There are many more. You can use

apt-cache search lua|grep lua|sort

for a list and

apt-cache show PACKAGENAME

to get details about a package.
Tell me which you need and I will install them too.

Comment by drtrigon [ Wed, 18 Jul 2012 21:34:23 +0000 ]

That's perfect for the moment! Thanks a lot for this!!!

Will you install python-lua automatic once the build-host is available? Or do I have to trigger this?

Thanks again and greetings!

Comment by dab [ Sun, 9 Sep 2012 19:38:08 +0000 ]

Closes with rev-19. Please check if the package is working and re-open this bug if not.





[TS-1302] Install 'nosetests' and 'coverage' (python-coverage) on (all) login server(s) Created: Sun, 19 Feb 2012 17:11:48 +0000 Updated: Wed, 25 Dec 2013 06:56:14 +0000 Resolved: Wed, 25 Dec 2013 06:56:14 +0000

Status: Resolved
Project: Toolserver
Component/s: Software installation,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Task Priority: Major
Reporter: drtrigon Assignee: Tim.Landscheidt
Resolution: Fixed Votes (Watches): 0 (1)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Depends
depends on TS-1301 Install 'doxygen' on (all) login server(s) Resolved

Description

Please install nosetests [1] and coverage (for python) [2] on the login servers, I would like to use them there.

[1] http://readthedocs.org/docs/nose/en/latest/
[2] http://nedbatchelder.com/code/coverage/

Thanks and Greetings


Comments
Comment by Tim.Landscheidt [ Wed, 25 Dec 2013 06:56:14 +0000 ]

Packages python-coverage and python-nose are installed on nightshade and yarrow.





[TS-1301] Install 'doxygen' on (all) login server(s) Created: Sun, 19 Feb 2012 17:06:58 +0000 Updated: Wed, 17 Jul 2013 16:47:36 +0000 Resolved: Sun, 30 Jun 2013 16:35:04 +0000

Status: Resolved
Project: Toolserver
Component/s: Software installation,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Task Priority: Major
Reporter: drtrigon Assignee: DaB.
Resolution: Fixed Votes (Watches): 0 (3)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Depends
is a dependency for TS-1302 Install 'nosetests' and 'coverage' (python-coverage) on (all) login server(s) Resolved
Relation
is related to TS-1669 nosetests "Permission denied" Resolved
URL: http://www.stack.nl/~dimitri/doxygen/

Description

Please install doxygen on the login servers (I think it was installed on nightshade once .

Thanks and Greetings


Comments
Comment by drtrigon [ Sat, 27 Apr 2013 17:05:05 +0000 ]

While doing this could you please also install 'coverage'?? Thanks a lot!!

Comment by platonides [ Sun, 28 Apr 2013 19:30:46 +0000 ]

I don't see a package named coverage in debian. Do you mean python-coverage? Or perhaps jscoverage or libpod-coverage-perl?

Comment by drtrigon [ Sun, 28 Apr 2013 20:12:04 +0000 ]

The one usually used with nosetests... I guess python-nose along with python-coverage, the shell command that should be available afterwards are:
nosetests and coverage

After installing doxygen the command doxygen should be available.

Comment by dab [ Sun, 28 Apr 2013 21:39:54 +0000 ]

I installed doxygen and nosetest with revision 137. I can not find a package for "coverage" too, should I install python-coverage?

Comment by drtrigon [ Mon, 29 Apr 2013 14:32:48 +0000 ]

Thanks!

I think python-coverage is the right one but can confirm this only after the package was installed...

As you installed doxygen and nosetests, why do I get

$ doxygen --help
-bash: doxygen: command not found
$ nosetests --help
-bash: nosetests: command not found

Comment by drtrigon [ Sat, 18 May 2013 20:07:55 +0000 ]

So what's the actual state here - as mentioned they are still not available - are you sure you installed them...?!??!

Greetings

Comment by dab [ Sat, 1 Jun 2013 15:20:15 +0000 ]

Looks like there is a problem with the version of your puppet. And I have also trouble to find the debian-packages at the moment. The hole problem should be solved after the Debian-update next week.

Comment by dab [ Sun, 30 Jun 2013 16:35:04 +0000 ]

Installed python-coverage at yarrow and nightshade by hand.

Comment by drtrigon [ Sun, 14 Jul 2013 09:45:06 +0000 ]

Cool thanks!

Edit: After testing it I'm happy since 'doxygen' and 'nosetests' are available and work, but what about (tested on nightshade)

$ coverage --help
-bash: coverage: command not found

so 'coverage' is missing? Or not in the system path? ...

Comment by drtrigon [ Sun, 14 Jul 2013 10:27:38 +0000 ]

'doxygen' is installed and works for me - but what about 'coverage'??

Comment by dab [ Sun, 14 Jul 2013 20:53:57 +0000 ]

There is no package in Debian with "coverage", only "python-coverage" which I installed. Please check if that is the program you need.

Comment by drtrigon [ Sun, 14 Jul 2013 22:09:39 +0000 ]

The package is the right one (of course but something is missing/strange. I am using a fedora system, so I can not exactly name the correct path, but compared to mine system:

  1. Where is the directory ".../python?.?/site-packages/coverage/"? It should be something like "/usr/lib/python2.6/site-packages/coverage/".
  2. Where is the python script "/usr/bin/coverage"?

if the second one would be in place, then the command "coverage" would work as expected.

In fact currently I am not able to find any signs of an installed "python-coverage" package - on nightshade - at all. Can you tell me where it was supposed to be installed to?

Comment by drtrigon [ Wed, 17 Jul 2013 16:47:36 +0000 ]

I found it; it's located at "/usr/lib/python2.6/dist-packages/coverage.py".

But now another problem appeared with "nosetests", please confer TS-1669.





[TS-1181] Fisheye "Failed to retrieve activity from the JIRA server Toolserver" Created: Sat, 17 Sep 2011 12:53:49 +0000 Updated: Sun, 22 Dec 2013 19:08:06 +0000

Status: Open
Project: Toolserver
Component/s: FishEye,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: Unassigned
Resolution: Unresolved Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified
Environment:

(accessed from firefox)


Issue Links:
Duplicate
is a duplicate of TS-1158 (n/a) (n/a)
URL: https://fisheye.toolserver.org/user/drtrigon

Description

When using fisheye and clicking on the user "drtrigon" (or the symbol) in [1] I come to [2] but there the following message appears in a red box:

Failed to retrieve activity from the JIRA server Toolserver: javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
Ignore errors from Toolserver

When not clicking onto "drtrigon" but moving the mouse pointer over it, the tooltip or hint opening read all the time

Retrieving drtrigon...

but never finishes or gets a result.

I do not know what this means or if this results in further problems. But together with TS-1102 and the fact that my emergency solution (to use websvn [3]) failed this night as DaB reported to me, I feel quite uncomfortable... It would be nice if fisheye could become more reliable since it is a good tool (except that it is slow) and thus important to me.

[1] https://fisheye.toolserver.org/changelog/drtrigon
[2] https://fisheye.toolserver.org/user/drtrigon
[3] http://toolserver.org/~drtrigon/websvn-2.3.1/listing.php?repname=drtrigon


Comments




[TS-1102] FishEye is not reflecting svn commits for project "drtrigon" Created: Sat, 9 Jul 2011 10:09:55 +0000 Updated: Tue, 30 Aug 2011 16:46:05 +0000 Resolved: Tue, 30 Aug 2011 15:47:18 +0000

Status: Resolved
Project: Toolserver
Component/s: Subversion,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: DaB.
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to TS-563 (n/a) (n/a)
is related to TS-522 (n/a) (n/a)
URL: https://fisheye.toolserver.org/changelog/drtrigon

Description

Project "drtrigon" on FishEye [1] is stuck at about revision #120 but the Subversion repository is at about revision #121.
(in the past such updates were delayed for quite some minutes - this is known - but now a few hours passed and nothing happened)

Seams to be the same issue like in TS-522 and TS-563.

Some (may be helpful) further notes:

  • I have not used the toolserver services (login, ...) for some months - coming back and trying to update the working copy of my repo in my home dir I got the error

    svn: Repository moved permanently to 'https://svn.toolserver.org/svnroot/drtrigon/pywikipedia'; please relocate

    from this it seams to me that the SVN URL has changed from 'http' to 'https' and this seams to break things (I have done nothing else - honestly

  • another hint may be the issue that my websvn 'installation' [2] (used in the past, but still there complains about

    Fehler beim Ausführen des Befehls: svn --non-interactive --config-dir /tmp list --xml 'https://svn.toolserver.org/svnroot/drtrigon/@'
    OPTIONS of 'https://svn.toolserver.org/svnroot/drtrigon': Server certificate verification failed: issuer is not trusted (https://svn.toolserver.org)

    which is due to the fact that the user under which the web server runs does not get access to the repo without certificate

[1] https://fisheye.toolserver.org/changelog/drtrigon
[2] http://toolserver.org/~drtrigon/websvn-2.3.1/listing.php?repname=drtrigon


Comments
Comment by drtrigon [ Thu, 14 Jul 2011 12:14:00 +0000 ]

I got the websvn installation running again by following:
http://www.doexpert.net/a/4e086715042f0a27acd69250
and adding

$config->setSvnConfigDir('/home/drtrigon/.subversion');

This way the same SVN config than for me (drtrigon) is used by websvn.

Comment by dab [ Tue, 30 Aug 2011 15:47:18 +0000 ]

Should be fixed now.

Comment by drtrigon [ Tue, 30 Aug 2011 16:46:05 +0000 ]

Works. Thanks a lot!





[TS-936] Install 'pysvn' module for python Created: Sun, 6 Feb 2011 12:09:04 +0000 Updated: Wed, 9 Feb 2011 10:09:34 +0000 Resolved: Wed, 9 Feb 2011 10:09:34 +0000

Status: Resolved
Project: Toolserver
Component/s: Software installation,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Task Priority: Major
Reporter: drtrigon Assignee: River Tarnell
Resolution: Fixed Votes (Watches): 0 (1)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Depends
depends on MNT-960 (n/a) (n/a)
is a dependency for DRTRIGON-68 release new version 0.3 Closed

Description

Hello Admins!

I'd like to use 'pysvn' module to gather some info for my bot. Could you please install it?

Thanks a lot!
Greetings


Comments




[TS-748] JIRA and FishEye Linkers Created: Tue, 7 Sep 2010 12:27:42 +0000 Updated: Wed, 18 Dec 2013 23:05:36 +0000

Status: Open
Project: Toolserver
Component/s: FishEye, JIRA,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: Alexander Mette
Resolution: Unresolved Votes (Watches): 0 (1)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified


Description

According to [1] it should be possible to link JIRA tickets to FishEye commits. I tried this for a few commits as visible in [2], but whitout success. And as I can see in e.g. [3] I'm not alone with this behaviour. So may be this feature was just not switched on yet, I am not aware how to use it or it does not work properly. Can you help me please? (May be this is just a FEATURE REQUEST or a SUPPORT issue.)

Thanks a lot!

[1] http://confluence.atlassian.com/display/FISHEYE016/Linkers
[2] https://fisheye.toolserver.org/changelog/drtrigon
[3] https://fisheye.toolserver.org/changelog/reports


Comments
Comment by drtrigon [ Mon, 12 Sep 2011 22:52:34 +0000 ]

I've taken the liberty of assigning you to this also. Greetings!

Comment by Tim.Landscheidt [ Wed, 18 Dec 2013 23:05:36 +0000 ]

As amette fixed the JIRA issue, assigning to him (though low priority).





[TS-710] Backup of '.../drtrigon/pywikipedia' needed/requested Created: Tue, 3 Aug 2010 13:30:06 +0000 Updated: Tue, 3 Aug 2010 17:24:53 +0000 Resolved: Tue, 3 Aug 2010 17:24:53 +0000

Status: Resolved
Project: Toolserver
Component/s: General/Unknown,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Task Priority: Major
Reporter: drtrigon Assignee: River Tarnell
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Depends
is a dependency for MNT-723 Restore files for drtrigan Resolved

Description

Hello Admins!

I've created a little problem for me, I did a 'svn remove' on the wrong
directory and thus lost my bot code... or (can't decide...) this
happened during try for first commit, so no old data are there.

Can you maybe give me the most recent data from backup of
.../drtrigon/pywikipedia back?!? If not I think I can restore the code
myself, but the data would also be useful...

Thank you very much and Greetings!
DrTrigon


Comments
Comment by drtrigon [ Tue, 3 Aug 2010 17:24:53 +0000 ]

Meanwhile I did some dirty and ugly tricks, but now it should be fine for me: no need for backuped data anymore.

Thanks anyway! Greetings
DrTrigon





[TS-337] FishEye repository for DrTrigon/DrTrigonBot (pywikipedia based) Created: Wed, 2 Sep 2009 09:15:19 +0000 Updated: Fri, 27 Aug 2010 08:55:23 +0000 Resolved: Thu, 26 Aug 2010 18:21:06 +0000

Status: Resolved
Project: Toolserver
Component/s: None
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Task Priority: Major
Reporter: drtrigon Assignee: River Tarnell
Resolution: Fixed Votes (Watches): 0 (1)
Labels: creation, software,
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified


Description

Hello!

I'd like to have the DrTrigonBot source in a repository on FishEye. There was a notice I have to open a ticket here, can you help me please?!

Thanks a lot and greetings!
DrTrigon


Comments
Comment by daniel [ Wed, 2 Sep 2009 09:27:14 +0000 ]

on jira yes, but not in the queue for account applications. i'll move the ticket.

Comment by river [ Sat, 10 Oct 2009 06:48:47 +0000 ]

What is the Toolserver username or SVN URL you want added?

Comment by river [ Wed, 6 Jan 2010 03:35:16 +0000 ]

No response from the user for 3 months. Please re-open this ticket if you still need your repository added.

Comment by drtrigon [ Tue, 3 Aug 2010 11:55:23 +0000 ]

I would like to get an SVN for "drtrigon" and/or "DrTrigonBot" running
from toolserver on german wiki. I'd like to get more or less the same or
similar like "Guandalug" for "BLUbot" [1] or "Guandalug's tools" [2].
This means something like "DrTrigon's tools" with components "DrTrigonBot"
(so [3] and [4] are the URLs of interesst) if this is possible?

Another question: Can this SVN repo [4] also be added to FishEye [5] as
mentioned in the toolserver wiki, please?

Thank you very much! (sorry for the late, and wrong addressed replies)
Greetings
DrTrigon

[1] https://jira.toolserver.org/browse/GUANDALUG/component/10430
[2] https://jira.toolserver.org/browse/GUANDALUG#selectedTab=com.atlassian.jira.plugin.system.project:summary-panel
[3] https://jira.toolserver.org/browse/DRTRIGON
[4] https://svn.toolserver.org/svnroot/drtrigon/
[5] https://fisheye.toolserver.org/

Comment by river [ Thu, 26 Aug 2010 18:21:06 +0000 ]

You already have an SVN repository, described at https://wiki.toolserver.org/view/Subversion. I've added it to FishEye (https://fisheye.toolserver.org/browse/drtrigon).

I created a JIRA project: <https://jira.toolserver.org/browse/DRTRIGON>. You can add components to it yourself as needed.

Comment by drtrigon [ Fri, 27 Aug 2010 08:55:23 +0000 ]

> You already have an SVN repository, described at https://wiki.toolserver.org/view/Subversion.

Yeah! Meanwhile I recognized this too...

Thanks a lot and greetings!





[MNT-723] Restore files for drtrigan Created: Tue, 3 Aug 2010 15:48:54 +0000 Updated: Tue, 3 Aug 2010 17:32:46 +0000 Resolved: Tue, 3 Aug 2010 17:32:46 +0000

Status: Resolved
Project: Maintenance
Component/s: None
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Minor work Priority: Major
Reporter: River Tarnell Assignee: River Tarnell
Resolution: Fixed Votes (Watches): 1 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Depends
depends on TS-710 Backup of '.../drtrigon/pywikipedia' needed/requested Resolved

Description

Restore drtrigon/pywikipedia from /home backup, requested by user.


Comments
Comment by drtrigon [ Tue, 3 Aug 2010 17:28:32 +0000 ]

Was able to cheat and solve this issue somehow... So the restore is not needed anymore. Thanks anyway!





[DRTRIGON-133] support of coming https security upgrade Created: Thu, 22 Aug 2013 20:26:42 +0000 Updated: Fri, 3 Jan 2014 21:36:00 +0000 Due: Wed, 28 Aug 2013 00:00:00 +0000 Resolved: Fri, 3 Jan 2014 21:36:00 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: None
Affects Version/s: None
Fix Version/s: 1.6,
Security Level: Public (all users)

Type: Improvement Priority: Major
Reporter: Herzi Pinki Assignee: drtrigon
Resolution: Answered Votes (Watches): 0 (3)
Labels: privacy, protocol_relative_urls,
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified
Environment:

all


URL: https://toolserver.org/~drtrigon/cgi-bin/sum_cat_disc.py

Description

DrTrigonBot category discussion summary (https://toolserver.org/~drtrigon/cgi-bin/sum_cat_disc.py), although called via https, provides links with plain http protocol.

For the upcoming security update, can your bot please use protocol relative urls?

see https://meta.wikimedia.org/wiki/HTTPS


Comments
Comment by drtrigon [ Sun, 22 Sep 2013 10:18:36 +0000 ]

Solved in https://gerrit.wikimedia.org/r/#/c/85521/.

Please check it and re-open the ticket if there are some issues left.

Comment by drtrigon [ Sun, 22 Sep 2013 18:14:00 +0000 ]

Do this for all other cgi scripts as well!

Comment by bugzilla-exporter [ Fri, 3 Jan 2014 21:36:00 +0000 ]

This bug has been migrated to Bugzilla: https://bugzilla.wikimedia.org/59613





[DRTRIGON-132] Second stage of substitutions for metadata info Created: Sat, 26 Jan 2013 10:15:17 +0000 Updated: Fri, 1 Feb 2013 22:14:19 +0000 Resolved: Fri, 1 Feb 2013 22:14:19 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - subster,
Affects Version/s: 1.6,
Fix Version/s: 1.6,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (1)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to DRTRIGON-115 Configurable edit summary for subster Resolved
URL: http://de.wikipedia.org/wiki/Wikipedia:WikiProjekt_Vorlagen/Werkstatt#letztes_Datum_einer_Seite_.C3.BCbergeben

Description

Include a 2nd stage of substitutions after the regular ones and after the check whether to save or not (because page content has changed due to substitutions) was already performed. This stage will not contain data from the source but metadata such as:

  • actual timestamp (generated by bot)
  • 'date' or 'Last-Modified' (from urllib)
  • more (e.g. source data size, ...)

essentially it should contain info that might change all the time but should not trigger a page update. The data should get inserted as very last step just before saving.


Comments
Comment by drtrigon [ Sat, 26 Jan 2013 13:44:26 +0000 ]

May be enhance concept in order to properly handle other things also:

  • error output
  • summary/edit-comment substitution

( both could/will profit of the new subst method that is needed by metadata subst - see comments in code point "5.)" )

Comment by drtrigon [ Fri, 1 Feb 2013 22:14:19 +0000 ]

All this (and more) done in r11022 upstream.

The summary subster thing has to be further discussed e.g. on http://de.wikipedia.org/wiki/Benutzer_Diskussion:HaeB#SubsterBot.2C_feeds_und_feature_request.





[DRTRIGON-131] Reduce bot_control.py influence, try to unlink features Created: Wed, 2 Jan 2013 21:46:37 +0000 Updated: Sun, 22 Sep 2013 10:32:31 +0000 Resolved: Sun, 22 Sep 2013 10:32:31 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia, DrTrigonBot - subster,
Affects Version/s: 1.5,
Fix Version/s: 1.6,
Security Level: Public (all users)

Type: Improvement Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (1)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to DRTRIGON-86 Test the re-write branch an decide what parts to migrate Closed

Description

Try to simplify functions and to unlink/move them from bot_control.py and put into framework upstream or bot scripts.

  • make the magicwords mechanism independent from bot_control.py by using subster_irc.py (which runs all the time) and connect to it by IRC channel (private communication) or alternatively by XML-RCP
  • error handler (with mail)
    • let the logging module do this, with help of class AdminEmailHandler
  • logger
    • e.g. move logging/logger stuff to wikipedia.py function log(...) (use '-log') and emulate 'info', 'warning', ... by using messages/text like "INFO: ...", "WARNING: ..." (as already done), ...
  • generic info and version output
  • several patches and work-a-rounds for other buggy code (e.g. py2.6)

The first point would also remove the need to run sum_disc.py, subster.py and others in sequence (one of the main reasons for bot_control.py) and allow to run them all in parallel (also because of new log file regime) and thus give more freedom for cron-job setup.


Comments
Comment by drtrigon [ Thu, 10 Jan 2013 16:51:09 +0000 ]

magicwords mechanism replaced with a new (better) one in r459 and r460 (upstream r10878).

Comment by drtrigon [ Thu, 10 Jan 2013 21:55:59 +0000 ]

version output (not generic info) partly introduced in trunk version.py but needs more work.

in rewrite version.py is a script thus a new module version.py (in pywikibot directory) has to be created and the important functions have to be moved from script to module. nothing done here.

edit: trunk and rewrite are both done now!

Comment by drtrigon [ Tue, 15 Jan 2013 18:50:47 +0000 ]

Logging needs a format change in wikipedia.py in order to finally have the same log output format as rewrite. This change was already prepared but is still commented (out), confer r10931 (line 8750-8751).

Attention: this needs also a change in panel.py (that breaks compatibility) therefore archive and remove of ALL logs on server (TS) and then start with the new format.

Comment by drtrigon [ Mon, 21 Jan 2013 17:24:49 +0000 ]

Please have also a look at http://de.wikipedia.org/wiki/Benutzer:DrTrigonBot/Status and the comment:

make the daily subster run independend of sum_disc in order to have good update here!

Comment by drtrigon [ Sun, 22 Sep 2013 10:32:31 +0000 ]

Done.

Consider change from bot_control.py to pwb.py (compatibility with rewrite/core) as well as current crontab.





[DRTRIGON-130] Enable Wikidata support Created: Tue, 11 Sep 2012 21:13:05 +0000 Updated: Sun, 23 Dec 2012 22:07:59 +0000 Resolved: Sun, 23 Dec 2012 21:11:00 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - subster,
Affects Version/s: 1.4,
Fix Version/s: 1.5,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (1)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Duplicate
is a duplicate of DRTRIGON-108 Use of templates across wikis (to simplify and unique Subster edits) Closed
Relation
is related to DRTRIGON-86 Test the re-write branch an decide what parts to migrate Closed
URL: http://meta.wikimedia.org/wiki/Wikidata/Bots

Description

Look at meta:Wikidata/Bots and may be at the watchlist [1] too.

[1] http://meta.wikimedia.org/wiki/Special:Watchlist


Comments
Comment by drtrigon [ Sun, 23 Dec 2012 21:11:00 +0000 ]

Started with r10821 upstream.





[DRTRIGON-129] Check to usefulness and capabilities of Labs (vs. TS) Created: Tue, 11 Sep 2012 21:03:14 +0000 Updated: Fri, 28 Dec 2012 15:25:51 +0000 Resolved: Fri, 28 Dec 2012 15:25:51 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - catimages, DrTrigonBot - pywikipedia, DrTrigonBot - subster, DrTrigonBot - toolserver scripts,
Affects Version/s: 1.4,
Fix Version/s: 1.5,
Security Level: Public (all users)

Type: Task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (1)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to DRTRIGON-86 Test the re-write branch an decide what parts to migrate Closed
is related to TS-1583 Install lua dev(el) package(s) Open
URL: https://labsconsole.wikimedia.org/wiki/Main_Page

Description

Account "DrTrigon" ("drtrigon") created on Labs and all points (1-5) of Help:Access#Initial_log_in_and_password_change done.

Further steps according to those documents and may be [1] too.

[1] http://wikimedia.7.n6.nabble.com/Getting-started-with-Wikimedia-Labs-especially-for-bot-authors-td4980968.html


Comments
Comment by drtrigon [ Thu, 27 Dec 2012 22:54:55 +0000 ]

Switched rewrite from TS to labs (runs on labs now) and is started from crontab. Some cgi script parts like panel (main view) works now too. So we have don some giant leaps towards a working solution. The possibility to install software (on your own) is very useful. Needed changes done in r433 and r434. For current state (actual progress) look at:

https://labsconsole.wikimedia.org/wiki/DrTrigonBot

Open points (among others) are:

  • on bots-apache01 install: python-matplotlib, python-numpy, python-BeautifulSoup, python-lxml (since this a 'non-root' environment use puppet)
  • open questions for chat: mail, backup, crontab/queue (like SGE), git, (bots-apache01 & puppet), ...
Comment by drtrigon [ Thu, 27 Dec 2012 22:55:46 +0000 ]

Since rewrite runs on labs now.

Comment by drtrigon [ Fri, 28 Dec 2012 15:25:51 +0000 ]

Labs starts becoming useful and productive (has not all options TS has, but others - e.g. sudo access). With the migration of rewrite we can start general migration. Depends of course on the future of the TS...





catimages several new features (todo list from commons)(DRTRIGON-124)

[DRTRIGON-128] Future improvements as duplicate detection, classification and code clean-up (TD-018, TD-019, TD-020) Created: Sun, 8 Jul 2012 15:12:43 +0000 Updated: Sat, 18 Jan 2014 19:15:34 +0000 Resolved: Sat, 18 Jan 2014 19:15:34 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - catimages, DrTrigonBot - pywikipedia,
Affects Version/s: 1.3,
Fix Version/s: 1.6,
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Answered Votes (Watches): 0 (1)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to DRTRIGON-86 Test the re-write branch an decide what parts to migrate Closed

Description
ID State What
TD-018 Request Implement a working duplicate detection (as well as other geometric verifications and checks) for all informations including 'Position' data (like e.g. Faces, Eyes, People, ...)
TD-019 Request Clean up the whole bot code. Merge similar code to functions. Clean file handling (download, check type, convert to jpeg may be png if needed, process further, ...). Docu needed (user and developer).
TD-020 Request In current state we have a lot of recognition and detection algorithms. What we need next is a good classification (RTrees, KNearest, Boost, SVM, MLP, NBayes, ...) based on those retrived data.

Comments
Comment by drtrigon [ Sat, 8 Sep 2012 22:19:16 +0000 ]
  • TD-018: DONE (implemented in '_dropRegions')
  • TD-020: DONE (implementation started with mlpy, but could also use opencv, scikit-learn or other)

TD-019 "code cleanup" OPEN (can be combined with change to rewrite branch)

Comment by drtrigon [ Sat, 18 Jan 2014 19:15:34 +0000 ]

Missed by "Bugzilla Bug Exporter (valhallasw)".





catimages several new features (todo list from commons)(DRTRIGON-124)

[DRTRIGON-127] Different techniques (of different bots) to find categories should not exclude each other (TD-013) Created: Sun, 8 Jul 2012 15:10:22 +0000 Updated: Sun, 8 Jul 2012 15:20:51 +0000 Resolved: Sun, 8 Jul 2012 15:20:51 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.3,
Fix Version/s: 1.3,
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (2)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified


Description
ID State What
TD-013 Request If Multichill's bot categorized a file with data from common sense, will your bot run on this file? different techniques to find categories should not exclude each other

Comments
Comment by drtrigon [ Sun, 8 Jul 2012 15:20:51 +0000 ]

Actually they DO NOT exclude each other. It s meerely a question of what pagegenerator gets used. At the current state r355 the bot understands the following command line switches:

-cat[:#]    Use a category as recursive generator
            (if no given 'Category:Media_needing_categories' is used)

-start[:#]  Start already form File:[:#] or if no file given start
            from top (instead of resuming last run).

-single:#   Run for one (any) single page only.

if this has to be extended in order to suit the needs, please RE-OPEN THIS ticket.





catimages several new features (todo list from commons)(DRTRIGON-124)

[DRTRIGON-126] Minor improvements and bug fixes for template and category handling (TD-010, TD-014) Created: Sun, 8 Jul 2012 15:08:31 +0000 Updated: Mon, 16 Jul 2012 12:13:09 +0000 Resolved: Mon, 16 Jul 2012 12:13:09 +0000

Status: Resolved
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - catimages,
Affects Version/s: 1.3,
Fix Version/s: 1.4,
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (1)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified


Description
ID State What
TD-010 not working if the user clicks on ''check categories now'' either on HotCat or the template, the bot-category (that isn't important in this case anymore) is automatically removed
→ TD-008
TD-014 Request add more (template) magic as [[User:Rillke]] suggested

Comments
Comment by drtrigon [ Sun, 8 Jul 2012 16:04:10 +0000 ]

TD-010: http://commons.wikimedia.org/w/index.php?title=User_talk:Multichill&diff=prev&oldid=73979501
TD-014: http://commons.wikimedia.org/w/index.php?title=User_talk:Rillke/Discuss/2012&diff=prev&oldid=73978894

Comment by drtrigon [ Sun, 15 Jul 2012 16:17:19 +0000 ]

TD-014: http://commons.wikimedia.org/w/index.php?title=User_talk:Rillke/Discuss/2012&diff=74309621&oldid=74242086

...some solution using additional javascript found... has to be activated on a per user basis, thus implemented now.

Comment by drtrigon [ Mon, 16 Jul 2012 12:11:48 +0000 ]

TD-010: http://commons.wikimedia.org/w/index.php?title=Commons:Village_pump&oldid=74353024#User:DrTrigonBot_and_other_categorization_stuff

It does not look like there is somebody out there knowing how the template has to be used with the category parameter, since this is rarely done...

At the moment the bot adds e.g.

{{Check categories|year=2012|month=July|day=16|category=[[Category:Categorized by DrTrigonBot]]}}

and when clicking onto "Check them now!" this template/line is removed but all categories present (some of them added by bot before) are kept untouched, [[Category:Categorized by DrTrigonBot]] too.

How to configure HotCat to remove a specific category, when used on a image description page the bot edited, is not obvious to me - may be not possible at all...

To summarize: the state here is still the same "not working" BUT it is an open question whether "not working" is an appropriate description since it might be possible that the template works perfectly well and I just missunderstood its function and effects.

Comment by drtrigon [ Mon, 16 Jul 2012 12:13:09 +0000 ]

TD-010: no changes
TD-014: Ok

Thus this ticket will be resolved now, if there are any objections here PLEASE RE-OPEN THIS TICKET.





catimages several new features (todo list from commons)(DRTRIGON-124)

[DRTRIGON-125] Enable further recognition/detection methods (TD-005, TD-011) Created: Sun, 8 Jul 2012 14:01:13 +0000 Updated: Sat, 8 Sep 2012 22:20:31 +0000 Resolved: Sat, 8 Sep 2012 22:20:31 +0000

Status: Resolved
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - catimages,
Affects Version/s: 1.3,
Fix Version/s: 1.4,
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (1)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified


Description
ID State What
TD-005 Request distinguish photos from diagrams
→ (e.g. BoW see below)
TD-011 Request work through [[:Category:Ships by name]] and sort images into [[:Category:Red ships (bot categorized)]], [[:Category:Green ships (bot categorized)]] etc

Further could include:

  • text recognition
  • other media like sound or video files

Comments
Comment by drtrigon [ Sat, 8 Sep 2012 22:11:26 +0000 ]
  • TD-005: DONE (without BoW; that is still up to be done)
  • TD-011: is an intressting request, but lies far in the future since first a ship detection (haarcascade or BoW has to be implemented)
  • text recognition: extraction of embbeded text layers is done (real recognition e.g. with ocropus/ocropy is still open)
  • other media like sound or video files: DONE (first simple categorization, should still get improved)




[DRTRIGON-124] catimages several new features (todo list from commons) Created: Sun, 8 Jul 2012 12:48:34 +0000 Updated: Mon, 9 Jul 2012 12:23:22 +0000 Resolved: Mon, 9 Jul 2012 12:23:22 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.3,
Fix Version/s: 1.3,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (2)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Sub-Tasks:
Key
Summary
Type
Status
Assignee
DRTRIGON-125 Enable further recognition/detection ... Sub-task Resolved drtrigon
DRTRIGON-126 Minor improvements and bug fixes for ... Sub-task Resolved drtrigon
DRTRIGON-127 Different techniques (of different bo... Sub-task Closed drtrigon
DRTRIGON-128 Future improvements as duplicate dete... Sub-task Closed drtrigon
URL: http://commons.wikimedia.org/w/index.php?title=User:DrTrigonBot/ToDo&oldid=73963461

Description
ID State What
TD-001 Ok bot categories should be one of:
  • [[:Category:Categorized by Image-Recognizing-DrTrigonBot]] or [[:Category:Categorized by DrTrigonBot evaluating the file contents]]
  • [[:Category:Categorized by DrTrigonBot]] and [[:Category:OpenCV]] → implemented by placing the Categories into the according bot templates (may be not suitable for OpenCV vs. Exif)
TD-002 Ok FileContentsByBot (and my be [[User:DrTrigon/Template:BotCatNote]]):
  • use regular wiki-markup for tables instead of template
  • the information about face-positions on file-description pages should be added using an own template/ own templates to allow easily reading the data and custom markup. Perhaps we want to hide them later or a JavaScript will hide them and transforms into image-notes.
  • use template and javascript instead of Image-annotations
  • would be elegant to have 1 template that does annotations and gives a structure that can be parsed
  • ''one'' template with following features: Expose a similar HTML (hidden) like ImageAnnotator with ''opt-in'' JavaScript, Expose machine-readable HTML for the faces/eyes and the color and all other stuff your bot finds, Expose the visible interface of in information-field like requested/suggested by EugeneZelenko, Scalable for future use
TD-003 Ok once we agreed on the format on file description pages, I will go over those the bot edited already and update them to look all the same - by hand
→ (TD-001), TD-002
TD-004 Ok add color-categories
TD-005 Request distinguish photos from diagrams
→ (e.g. BoW see below)
TD-006 Ok on the result-page it would be handy if the faces would be framed somehow, e.g. with ImageAnnotator and/or div
→ result-page: [[User:DrTrigon/Category:Unidentified people (bot tagged)]]
TD-007 Ok list added categories in edit summary
TD-008 Ok if the face takes a big portion of the image, add [[:Category:Portraits]] and if the face fills the image, [[:Category:Faces]]
TD-009 Ok check categories should either contain the date or look like

{{Check categories|year=<year>|month=<month>|day=<day>|category=[[Category:Categorized by OpenCV-bot]]}}

TD-010 not working if the user clicks on ''check categories now'' either on HotCat or the template, the bot-category (that isn't important in this case anymore) is automatically removed
→ TD-008
TD-011 Request work through [[:Category:Ships by name]] and sort images into [[:Category:Red ships (bot categorized)]], [[:Category:Green ships (bot categorized)]] etc
TD-012 Ok could be worth tuning (re-defining) this color table and in a further step enhance the number of color categories (and thus the coverage of the color space) here in commons (but this would need a huge number of new categories!)
→ [[Commons:Categories for discussion]]
TD-013 Request If Multichill's bot categorized a file with data from common sense, will your bot run on this file? different techniques to find categories should not exclude each other
TD-014 Request add more (template) magic as [[User:Rillke]] suggested
TD-015 Ok have to agree on all label names (create some consistent naming) and go over all templates to double check them (this might also affect the javascript!)
→ TD-002
TD-016 Ok improve docs to resemble and explain all label used (in TD-015)
→ TD-015
TD-017 Ok Before running the color categorization, crop 25% of the image in order to give the bot a more human eye. ([[User:Rillke]])
TD-018 Request Implement a working duplicate detection (as well as other geometric verifications and checks) for all informations including 'Position' data (like e.g. Faces, Eyes, People, ...)
TD-019 Request Clean up the whole bot code. Merge similar code to functions. Clean file handling (download, check type, convert to jpeg may be png if needed, process further, ...). Docu needed (user and developer).
TD-020 Request In current state we have a lot of recognition and detection algorithms. What we need next is a good classification (RTrees, KNearest, Boost, SVM, MLP, NBayes, ...) based on those retrived data.

Comments
Comment by drtrigon [ Mon, 9 Jul 2012 12:23:22 +0000 ]

All tasks marked as 'Ok' are implemented now.

All other are moved to sub-tasks DRTRIGON-125, DRTRIGON-126, DRTRIGON-127 and DRTRIGON-128.





[DRTRIGON-123] sum_disc: all related issues Created: Fri, 8 Jun 2012 18:30:15 +0000 Updated: Fri, 3 Jan 2014 16:20:37 +0000 Resolved: Fri, 3 Jan 2014 16:20:37 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: pywikipedia,
Affects Version/s: 1.3,
Fix Version/s: 1.6,
Security Level: Public (all users)

Type: Improvement Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Answered Votes (Watches): 0 (1)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Sub-Tasks:
Key
Summary
Type
Status
Assignee
DRTRIGON-18 F56 (copied from wiki) Sub-task Closed drtrigon
DRTRIGON-24 F47 (copied from wiki, priority 3) Sub-task Closed drtrigon
DRTRIGON-31 18 (copied from wiki) Sub-task Closed drtrigon
DRTRIGON-32 30 (copied from wiki) Sub-task Closed drtrigon
DRTRIGON-90 sequence of sometimes occuring "KeyEr... Sub-task Closed drtrigon
DRTRIGON-91 false-positive-Meldungen - DÜP Sub-task Closed drtrigon
DRTRIGON-98 Parameter um max. Zeitdiff zw. Benutz... Sub-task Closed drtrigon
DRTRIGON-99 Parameter zusätzlich zu 'altsign_list... Sub-task Closed Saibo
URL: http://sourceforge.net/tracker/?group_id=93107

Description

This task gather all open issues, feature requests, tasks and else related to sum_disc.py with moved upstream. Thus new issues should be reported to pywikipedia bug tracker: http://sourceforge.net/tracker/?group_id=93107


Comments
Comment by bugzilla-exporter [ Fri, 3 Jan 2014 16:20:37 +0000 ]

This bug has been migrated to Bugzilla: https://bugzilla.wikimedia.org/59500





[DRTRIGON-122] Additional options to config sum_disc.py dealing with thread header Created: Wed, 25 Apr 2012 19:27:24 +0000 Updated: Fri, 8 Jun 2012 17:29:25 +0000 Resolved: Fri, 8 Jun 2012 17:29:25 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.3,
Fix Version/s: 1.3,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: Xqt Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified


Description

Please enable an additional option 'ignorehead_list' (which is alread available in your code but not published yet) to ignore some talk heads from the bots notifier. And the opposite option 'matchhead_list' would be great. This should only process such threads who match that given regex.


Comments
Comment by drtrigon [ Fri, 27 Apr 2012 22:06:06 +0000 ]

I assume everything works fine until you contact me again. Hope this is ok?
Greetings

Comment by Xqt [ Mon, 30 Apr 2012 10:41:35 +0000 ]

There is a remaining feature request for option 'matchhead_list' described above, maybe a nice_to_have priority I do not know whether it works as assumed. It did nothing because the signature didn't match. btw. I do not understand that edit: http://de.wikipedia.org/w/index.php?title=Benutzer:BLUbot/Antworten&diff=102565469&oldid=102535563 . This thread wasn't edited by DÜP-accounts before. (all will be fine

Comment by drtrigon [ Fri, 8 Jun 2012 17:29:25 +0000 ]

Implemented upstream in r10334.





[DRTRIGON-121] Subster bot 'error' parameter does not accept '\n' (like in default value) Created: Mon, 12 Mar 2012 19:31:31 +0000 Updated: Fri, 16 Mar 2012 16:14:38 +0000 Resolved: Fri, 16 Mar 2012 16:14:38 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.3,
Fix Version/s: 1.3,
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://de.wikipedia.org/w/index.php?title=Benutzer:DrTrigon/Spielwiese&diff=100797566&oldid=100797518

Description

The 'error' parameter does not get processed as python string, thus '\n' like used in default value does not work.


Comments
Comment by drtrigon [ Fri, 16 Mar 2012 16:14:38 +0000 ]

Done in r291.





[DRTRIGON-120] New bot script/part for opencv image recognition and classification Created: Sat, 10 Mar 2012 23:39:37 +0000 Updated: Mon, 16 Jul 2012 18:34:01 +0000 Resolved: Fri, 16 Mar 2012 16:25:49 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.3,
Fix Version/s: 1.3,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Attachments: Fileopencv_classify.tar.gz     
Issue Links:
Relation
is related to TS-1449 Install libraries and python bindings needed for catimages bot Open
is related to TS-1360 Install OpenCV libraries and python bindings Resolved
URL: http://commons.wikimedia.org/wiki/User:Multichill/Using_OpenCV_to_categorize_files

Description

The ide was raised on [1] by Multichill and is a very intressting one! First test done by him and some by myself look very promising.

The face detection is already implemented in the opencv python-wrappers, but the 'bag of words' method and others not. This is not really a problem since we can use the C++ code examples given at several places in the web and make them into a python module by help of boost.python.

[1] http://lists.wikimedia.org/pipermail/pywikipedia-l/2012-February/007324.html


Comments
Comment by drtrigon [ Sat, 10 Mar 2012 23:43:27 +0000 ]

Some first test done on fedora 15 all with original packages (no additional from other external sources) and pywikipedia code. Everything compiles fine.

Comment by drtrigon [ Fri, 16 Mar 2012 16:25:49 +0000 ]

Added in r293.

At this early stage this is an experimental script only.





[DRTRIGON-119] More/better SGE integration and interaction in bot_control Created: Sun, 4 Mar 2012 12:18:39 +0000 Updated: Fri, 22 Jun 2012 13:47:30 +0000 Resolved: Fri, 22 Jun 2012 13:47:30 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.3,
Fix Version/s: 1.3,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to DRTRIGON-87 Bot ERROR mail cannot be sent sometimes Closed
URL: http://lists.wikimedia.org/pipermail/toolserver-l/2012-March/004754.html

Description

See URL and follow-ups in that thread (maillist). Merlissimo adapted the job submission system to be more like SGE itself. Use the new features and options given at [1].

[1] https://wiki.toolserver.org/view/Job_scheduling


Comments
Comment by drtrigon [ Thu, 21 Jun 2012 23:11:56 +0000 ]

E.g. add the commonly used SGE options in cronie (for qcronsub) to bot_control.py

Comment by drtrigon [ Fri, 22 Jun 2012 13:47:30 +0000 ]

Implemented in r343.

(might need further adoptions and fine tuning in future; just RE-OPEN the ticket then...)





[DRTRIGON-118] Enable lua support to enable even better template programming inclusion (subster) Created: Sun, 19 Feb 2012 17:24:11 +0000 Updated: Fri, 3 Jan 2014 16:03:20 +0000 Resolved: Fri, 3 Jan 2014 16:03:20 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia, DrTrigonBot - subster,
Affects Version/s: 1.2,
Fix Version/s: 1.6,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Answered Votes (Watches): 0 (2)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Attachments: Filetest_python-lua.py     
Issue Links:
Relation
is related to TS-1583 Install lua dev(el) package(s) Open
is related to TS-1303 Install 'lunatic-python' (together with 'lua') Resolved

Description

As mentioned in [1] the mediawiki software will include lua in future as template scripting language. In order to improve subster and enable even closer integration and e.g. write postproc funcs in lua also use [2] to combine lua with python with lua with ...

[1] http://www.mediawiki.org/wiki/Lua_scripting
[2] http://labix.org/lunatic-python


Comments
Comment by drtrigon [ Sun, 19 Feb 2012 20:30:34 +0000 ]

'postproc' functions with lua have some unicode issues (tested) and would make not that much sence - introduces another layer of complexibility - since they are written and modified by the bot operator only!

May be enhancing 'simple' (or introduce a new parameter called 'lua-simple' or 'lua', ...) with lua support would make more sence.

One issue may be that the bot's (local) lua is not in same scope like the lua in the wiki (has other libraries in background). This would always be an issue.

(may be write a lua syntax checker bot...)

Comment by drtrigon [ Sun, 19 Feb 2012 20:37:38 +0000 ]

Example for usage of 'lunatic-python'.

Comment by drtrigon [ Tue, 12 Jun 2012 22:12:45 +0000 ]

Extension:Scribunto
http://www.mediawiki.org/wiki/Extension:Scribunto

Comment by drtrigon [ Sun, 23 Dec 2012 22:22:09 +0000 ]

script_wui.py (from rewrite) has lua support included. TS-1583 needed in order to activate it.

Comment by bugzilla-exporter [ Fri, 3 Jan 2014 16:03:20 +0000 ]

This bug has been migrated to Bugzilla: https://bugzilla.wikimedia.org/59403





[DRTRIGON-117] Tool for unshorten of whole pages (and more...) Created: Wed, 15 Feb 2012 12:21:47 +0000 Updated: Wed, 15 Feb 2012 12:24:20 +0000 Resolved: Wed, 15 Feb 2012 12:24:20 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - toolserver scripts,
Affects Version/s: 1.2,
Fix Version/s: 1.2,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified


Description

Create a new tool (e.g. like xsalt) that unshortens all links on a given page (given by URL).

This tool will be usable for further and more complicated filter options in future also (call them "service-functions"). Functions needing to use other web services (like unshorten API) should be implemented here instead of the bot code directly!


Comments
Comment by drtrigon [ Wed, 15 Feb 2012 12:24:20 +0000 ]

Done in r264 till r266. Now further options can also be included...





[DRTRIGON-116] Possiblity to supress error output to pages for subster Created: Wed, 15 Feb 2012 12:18:36 +0000 Updated: Fri, 17 Feb 2012 01:13:11 +0000 Resolved: Fri, 17 Feb 2012 01:13:11 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.2,
Fix Version/s: 1.2,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://de.wikipedia.org/wiki/Benutzer_Diskussion:DrTrigon#Anfrage_betreffs_Bot_f.C3.BCr_Listenaktualisierung

Description

Introduce a new parameter/option in order to be able swicth error output off. May be the usage of <noinclude> tags or any output template could be configured also with the same option (e.g. set template string like '<noinclude>%s</noinclude>' or '' for no output...).


Comments
Comment by drtrigon [ Fri, 17 Feb 2012 01:13:11 +0000 ]

Done in r271.





[DRTRIGON-115] Configurable edit summary for subster Created: Wed, 15 Feb 2012 12:13:22 +0000 Updated: Sun, 22 Sep 2013 08:56:37 +0000 Resolved: Sun, 22 Sep 2013 08:47:16 +0000

Status: Resolved
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - subster,
Affects Version/s: 1.2,
Fix Version/s: 1.6,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Won't Fix Votes (Watches): 0 (1)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to DRTRIGON-132 Second stage of substitutions for metadata info Closed
URL: http://de.wikipedia.org/wiki/Benutzer_Diskussion:DrTrigon#Subster_f.C3.BCr_Identi.ca-.2FTwitter-Feeds

Description

Introduce a new parameter/option in order to be able to set a specific (for this page only) edit summary the bot uses when modifying a page.


Comments
Comment by drtrigon [ Thu, 12 Jul 2012 16:11:49 +0000 ]

Parameter/Option 'comment' that will be threatened as content to enable full substitution power:

|comment=blabla example: <!--SUBSTER-val--><!--SUBSTER-val--> ...

that becomes e.g. with the value 'test' the comment:

blabla example: test ...

Implementation:

  1. threat the value in 'comment' like the usual content and do all substitutions there too
  2. use e.g. 'removeDisabledParts' from textlib to remove all subster tags (but keep the tag content)
  3. use the result instead of usual edit comment
Comment by drtrigon [ Sat, 26 Jan 2013 21:03:28 +0000 ]

May it would be better to use:

|comment=blabla example: <!--SUBSTER-val--> ...

in order to have a clear difference and prevent the bot changing its own template on the page too!

Comment by drtrigon [ Sun, 17 Feb 2013 16:36:24 +0000 ]
  • what if multiple templates present? -> go through all and take the last one defining 'comment' param - so they overwrite each other; last in - only one out
  • "do all substitutions on comment too" or just some specific ones? if all are done too; how to reproduce the original value in page content?
...
msg = self.subTag(msg, ...)[0]
#substed_tags += tags
msg = pywikibot.textlib.removeDisabledParts(msg)
...
Comment by drtrigon [ Sun, 22 Sep 2013 08:47:16 +0000 ]

No answer on [1] so this might be intressting, but there seams to be no need for it at the moment.

[1] https://de.wikipedia.org/wiki/Benutzer_Diskussion:HaeB#SubsterBot.2C_feeds_und_feature_request





[DRTRIGON-114] Support for named groups in regexs Created: Sat, 4 Feb 2012 17:23:32 +0000 Updated: Fri, 22 Jun 2012 12:40:24 +0000 Resolved: Fri, 22 Jun 2012 12:40:24 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.2,
Fix Version/s: 1.3,
Security Level: Public (all users)

Type: New Feature Priority: Critical
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to DRTRIGON-110 Improve postproc functions Closed
is related to DRTRIGON-105 Support for multiple BS template configurations Closed

Description

Enable the usage of named groups '(?P<name>...)' in regex and combine them with '%(name)s' syntax.


Comments
Comment by drtrigon [ Fri, 17 Feb 2012 01:05:07 +0000 ]

Named group can be used for DRTRIGON-110 also.

Named groups can further be used in 'regex', coupled to 'value' (like multiple BS tags on 1 page) and could become a complete revolution. May be this gives a new mode again and remove completely the need of some postproc usages... Has a lot of potential!!!

Comment by drtrigon [ Fri, 17 Feb 2012 01:07:29 +0000 ]

Mainly in 'postproc' regex, but later also in 'regex' (or else)...

Comment by drtrigon [ Wed, 22 Feb 2012 21:11:58 +0000 ]

Look also at 'replacetext' using 'sub' and thus supporting named groups like:

('replacetext', '(?P<url>http://(ur1.ca\x7Cbit.ly)\S*)', '\x3Cnowiki\x3E\g<url>\x3C/nowiki\x3E')

http://de.wikipedia.org/w/index.php?title=Benutzer:DrTrigon/Spielwiese&oldid=100014206#Test:_19:23.2C_22._Feb._2012_.28CET.29

Comment by drtrigon [ Fri, 22 Jun 2012 12:40:24 +0000 ]

Implemented in r341.

For examples confer http://de.wikipedia.org/w/index.php?title=Benutzer:DrTrigon/Spielwiese&oldid=104677569 and following edits/changes.

New postproc are written and will be enabled too.





[DRTRIGON-113] Enable big data substitutions (in MB range) Created: Tue, 31 Jan 2012 11:00:06 +0000 Updated: Fri, 3 Jan 2014 16:04:02 +0000 Resolved: Fri, 3 Jan 2014 16:04:02 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - subster,
Affects Version/s: 1.2,
Fix Version/s: 1.6,
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Answered Votes (Watches): 0 (1)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://en.wikipedia.org/wiki/Wikipedia_talk:WikiProject_Chess#Automatic_updated_FIDE_rating_in_infobox_.28by_bot.29

Description

The request on enwiki to substitute more data on Template:Elo_rating (e.g. all >= 2000 or even more) failed, as can be seen from the respective log file mainbot (most of it is this write trial).

Investigate how such big write request could be handeled as well. One possible solution is to split the data and write in multiple steps (what is the max. size for one write action?).


Comments
Comment by drtrigon [ Mon, 18 Jun 2012 19:13:07 +0000 ]

Might be related to:

https://bugzilla.wikimedia.org/show_bug.cgi?id=37536

Comment by bugzilla-exporter [ Fri, 3 Jan 2014 16:04:02 +0000 ]

This bug has been migrated to Bugzilla: https://bugzilla.wikimedia.org/59408





[DRTRIGON-112] subster_irc bot forgets wiki login when accessing other mediawiki project Created: Tue, 17 Jan 2012 19:06:41 +0000 Updated: Thu, 16 Feb 2012 22:58:18 +0000 Resolved: Thu, 16 Feb 2012 22:58:18 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.2,
Fix Version/s: 1.2,
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://de.wikipedia.org/w/index.php?title=Benutzer:DrTrigon/Spielwiese&diff=98500261&oldid=98348132

Description

When looking at the logfile from 2012-01-17 for following comments and later:

2012-01-17 19:44:36,744 bot_control INFO Password for user DrTrigonBot on wikipedia:de:

the subster bot (may be just when used by subster_irc...?!?) asks for the wiki password.
(look also at difflink in URL - which triggered the error in the logfile)

We have to protect the bot from logging out when accessing foreign wikis!


Comments
Comment by drtrigon [ Fri, 20 Jan 2012 10:09:29 +0000 ]

This does not apply when accessing the same external wikis by API, as can be seen in [1].

[1] http://de.wikipedia.org/w/index.php?title=Benutzer:DrTrigon/Spielwiese&diff=98597085&oldid=98597019&diffonly=0

Comment by drtrigon [ Sat, 28 Jan 2012 14:06:11 +0000 ]

A possible solution or work-a-round for this problem is to use 'site._load(force=True)' to re-login e.g.

external_buffer = self.site.getUrl(param['url'], no_hostname = True)
if self.site.loggedInAs() is None: self.site._load(force=True)
Comment by drtrigon [ Sat, 28 Jan 2012 14:42:47 +0000 ]

This (work-a-round) was implemented in r259.

Comment by drtrigon [ Sat, 28 Jan 2012 15:06:39 +0000 ]

Critical -> Major

The bug is solved (work-a-round) but since subster_irc runs multi-threaded it is still possible that on thread tries to write in between 'getUrl' and '_load' from other thread and thus will raise and error. Thus the whole logout should be prevented and not happen at all!

Comment by drtrigon [ Mon, 30 Jan 2012 10:33:23 +0000 ]

Started the discussion on the list, look at http://lists.wikimedia.org/pipermail/pywikipedia-l/2012-January/007182.html

From that it seems to be hard to find a consensus since in fact I am "abusing" the function here. But a very slight modification to the function would solve all problems... Thus there has to be a useful consensus...

Comment by drtrigon [ Wed, 1 Feb 2012 16:32:36 +0000 ]

Re-merge with pywikipedia framework (increase to v1.2) and solve the problem as worked out on maillist. Create a new generic getUrl function (from old one in site class) and specialize it into the one in site class. Similar to the solution in rewrite.

May be this has to be discussed further with Merlijn (valhallasw) since from rewrite it seams to me that it could be impossible to separate into the whished 2 functions... but let's solve this later...

Comment by drtrigon [ Thu, 2 Feb 2012 10:09:14 +0000 ]
Comment by drtrigon [ Sun, 5 Feb 2012 02:19:57 +0000 ]

Another issue - may be related, may be not (but can be solved in same session, since hast to be done upstream) is the fact that the framework seems to have trouble handling e.g. arwiki. 'live_version' has a bug there (raises an error) and also login seems to be problematic because of:

2012-02-05 02:11:17,512 bot_control          INFO     WARNING: Your account on wikipedia:ar is blocked by False.
2012-02-05 02:11:17,512 bot_control          INFO     Reason: أيبي التول سيرفر
2012-02-05 02:11:17,512 bot_control          INFO     Editing using this account will stop the run.

The first issue was worked-a-round by skipping this request for 'ar' in r261.

Comment by drtrigon [ Thu, 16 Feb 2012 20:49:55 +0000 ]

The work-a-round was removed because of fix of 'live_version' upstream in r8900. Done in r269.

Comment by drtrigon [ Thu, 16 Feb 2012 22:58:18 +0000 ]

updated analogue to rewrite; 'getUrl' moved/splitted to 'pywikibot.comms.http.request'
the generic one does not attempt to (re)login on the target if it is a wiki

upstream r9901.

Ported and done in r270.





[DRTRIGON-111] Commit clean_user_sandbox upstream (make it independent as well) Created: Mon, 16 Jan 2012 15:49:46 +0000 Updated: Thu, 16 Feb 2012 18:39:53 +0000 Resolved: Thu, 16 Feb 2012 18:39:53 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.2,
Fix Version/s: 1.2,
Security Level: Public (all users)

Type: Task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified


Description

Make clean_user_sandbox.py independent from sum_disc.py techniques and code. This means the bot should trigger on the user sandbox template [1] ONLY! Not on the 'Diene_Mir!' list anymore. Further a global template (not just for me) for usage with this bot should be created (may be explaining more on the bot function and usage) and replace my user sandbox template (use [2] & [3] as inspiration for name and design).

[1] http://de.wikipedia.org/wiki/Benutzer:DrTrigonBot/Spielwiese
[2] http://de.wikipedia.org/wiki/Wikipedia:Spielwiese
[3] http://de.wikipedia.org/wiki/Vorlage:Bitte_erst_NACH_dieser_Zeile_schreiben!_%28Begr%C3%BC%C3%9Fungskasten%29

The code has to be adopted and may be this raises some further needs on how to organize this on the wiki (in a more general way). May be it is also possible to include the whole functionality into clean_sandbox.py and a seperate script in not needed anymore.

The command line usage (help docs) of the script has to be well documented.


Comments
Comment by drtrigon [ Thu, 16 Feb 2012 18:39:53 +0000 ]

Templates were not changed and left as it was.

Done; upstream in r9899 and thus in r268.





[DRTRIGON-110] Improve postproc functions Created: Fri, 13 Jan 2012 14:45:27 +0000 Updated: Sat, 18 Feb 2012 22:23:28 +0000 Resolved: Sat, 18 Feb 2012 22:23:28 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.2,
Fix Version/s: 1.2,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to DRTRIGON-114 Support for named groups in regexs Closed
URL: http://de.wikipedia.org/wiki/Benutzer:DrTrigon/DrTrigonBot/config.css

Description
  1. try to merge 'list' with 'formatedlist', or count how many usage of 'list' are in dewiki and others - may be 'list' can be dropped at all (may be others too...?)
  2. add a new function 'chain' or similar to use multiple postproc functions in sequence, may be hold all the data in DATA[1:] to re-use it later on
  3. with 2) and 'replacetext' the 'notag' parameter could become obsolete...

Comments
Comment by drtrigon [ Sat, 18 Feb 2012 15:19:37 +0000 ]

'formatedlist' adopted in order to ba able to replace 'list'. Thus 'list' and 'wikilist' removed since rarely used.

Comment by drtrigon [ Sat, 18 Feb 2012 16:42:52 +0000 ]

'chain' implemented and all data are stored before function execution in DATA[i] (for i=1,2,...) and the most recent always in DATA[0]. All data are accessible this way.

Comment by drtrigon [ Sat, 18 Feb 2012 22:22:21 +0000 ]

Indeed 'notag' becomes obsolete, is rarely (never) used and can be replaced by

|postproc=('replacetext', '<.*?>', '\'')

and - if another postproc is needed - combined with 'chain'.

The needed changes to http://de.wikipedia.org/wiki/Benutzer:DrTrigon/DrTrigonBot/subster-postproc.css and other languages are done. The changes in code are done in r279.

Comment by drtrigon [ Sat, 18 Feb 2012 22:23:28 +0000 ]

All 3 implemented and done.





[DRTRIGON-109] Subster (and subster_irc) configuration for edits flags onto own CSS page (split CSS config) Created: Fri, 13 Jan 2012 14:32:57 +0000 Updated: Sat, 28 Jan 2012 23:57:59 +0000 Resolved: Sat, 28 Jan 2012 23:57:59 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.2,
Fix Version/s: 1.2,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: https://fisheye.toolserver.org/browse/drtrigon/pywikipedia/subster.py?hb=true#to111

Description

The two latest options or parameters 'flagenable' and 'difflink' should be put into a CSS config page on the specific wiki for which they are valid (dewiki here).

Since we have already a CSS config page [1] and it would be most simple to create an additional one, we should split the CSS config in a general and transparent (with comments included pointing to the other CSS pages) way.
[1] http://de.wikipedia.org/wiki/Benutzer:DrTrigon/DrTrigonBot/config.css


Comments
Comment by drtrigon [ Sat, 28 Jan 2012 23:57:59 +0000 ]

Implemented in r260. The config files are summarized in User:DrTrigon/DrTrigonBot, look at e.g. dewiki, and 'postproc' is needed the other is optional.

subster_irc.py does not automatically reload the config page yet, if changed.





[DRTRIGON-108] Use of templates across wikis (to simplify and unique Subster edits) Created: Mon, 9 Jan 2012 10:02:13 +0000 Updated: Sun, 23 Dec 2012 22:08:53 +0000 Resolved: Sun, 23 Dec 2012 22:08:53 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - subster,
Affects Version/s: 1.2,
Fix Version/s: 1.5,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (1)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by DRTRIGON-130 Enable Wikidata support Closed
URL: https://bugzilla.wikimedia.org/show_bug.cgi?id=9890

Description

As you can see at [1] there are efforts to unique the templates used on 1 single wiki (e.g. commons) like for image file data. This would push the bot tremendously since it would have to work on one single wiki only afterwards!! (and it would also solve the whole interwiki mess...

In fact the option was already included into mw code, but not activated on the installations until now, have a look at [2]. So we are waiting for the activation... After activation we would have to request a botflag on commons (or meta) in order to do the work there.

[1] http://meta.wikimedia.org/wiki/Help:Template#Use_of_templates_across_wikis
[2] https://bugzilla.wikimedia.org/show_bug.cgi?id=9890


Comments
Comment by drtrigon [ Sun, 19 Feb 2012 00:51:53 +0000 ]

http://www.mediawiki.org/wiki/Manual:%24wgEnableScaryTranscluding#How_to_use_scary_transclusion

Comment by drtrigon [ Sun, 23 Dec 2012 22:06:48 +0000 ]

Botflag on commons and meta is granted, but on wikidata missing at the moment.

Comment by drtrigon [ Sun, 23 Dec 2012 22:08:53 +0000 ]

Since DRTRIGON-130 was closed, this one is done too.





[DRTRIGON-107] DrTrigonBot without Botflag (on some pages) Created: Mon, 2 Jan 2012 20:35:53 +0000 Updated: Tue, 3 Jan 2012 13:24:06 +0000 Resolved: Tue, 3 Jan 2012 13:24:06 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.2,
Fix Version/s: 1.2,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:DrTrigon&oldid=97748399#DrTrigonBot_ohne_Botflag

Description

Grip99 requests edits on some pages without bot-flag similar to "PRD" [1]. This can be motivated like CopperBot does [2].

[1] http://de.wikipedia.org/wiki/Wikipedia:Projektdiskussion/PRD-subst
[2] http://de.wikipedia.org/wiki/Benutzer:CopperBot#Warum_erscheinen_die_Edits_des_Bots_auf_der_Beobachtungsliste.3F

As it currently seams, the introduction of an additional list or dict like 'flags' (in subster_irc.py) has to be added to subster.py which is not transparent and handy! Thus a better way should be found, e.g. moving and joining both 'flags' params in subster.py since it is imported into subster_irc.py and thus exposes its variables.


Comments
Comment by drtrigon [ Tue, 3 Jan 2012 13:24:06 +0000 ]

Implemented in r255 and follow-up r256.





[DRTRIGON-106] Translations (TW) for subster.py Created: Fri, 9 Dec 2011 23:00:56 +0000 Updated: Fri, 17 Feb 2012 16:12:34 +0000 Resolved: Fri, 17 Feb 2012 16:12:34 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.2,
Fix Version/s: 1.2,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to DRTRIGON-67 Run on all (or several) wikis at once (like interwiki bot) Closed

Description

Since 'subster.py' is already used on frr-wiki and should also be introduced to en-wiki, the time has come to add translations (TW). Learn from sum_disc.py translations and do it better that time.

When translating 'subster.py' consider to translate 'subster_irc.py' also (tw, messages, ...).

May be helpful could be to look at:

pywikibot.translate(self.site.lang, self.move_comment) -> i18n.twtranslate(self.site.lang, self.move_comment)

Comments
Comment by drtrigon [ Fri, 17 Feb 2012 16:12:34 +0000 ]

Changed according to conversion manual http://www.mediawiki.org/wiki/Manual:Pywikipediabot/i18n_conversion

Done in r275.





[DRTRIGON-105] Support for multiple BS template configurations Created: Fri, 9 Dec 2011 22:56:13 +0000 Updated: Fri, 22 Jun 2012 12:39:38 +0000 Resolved: Fri, 22 Jun 2012 12:39:38 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts,
Affects Version/s: 1.2,
Fix Version/s: 1.3,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to DRTRIGON-114 Support for named groups in regexs Closed

Description

Support for multiple BS (beautiful soup) tags belonging to different templates and url's per page. For this the syntax now beeing

<!--SUBSTER-BS:...-->content...<!--SUBSTER-BS:/-->

has to be slightly modified to become:

<!--SUBSTER-val-BS:...-->content...<!--SUBSTER-BS:/-->

e.g. for value 'val', may be with 'val' in the closing/end tag also.

Then using value='' results in:

<!--SUBSTER--BS:...-->content...<!--SUBSTER-BS:/-->

notice the additional

-

between 'SUBSTER' and 'BS'!


Comments
Comment by drtrigon [ Fri, 22 Jun 2012 11:13:18 +0000 ]

The default syntax using value='' is the same as now

<!--SUBSTER-BS:...-->content...<!--SUBSTER-BS:/-->

(no additional '-' between 'SUBSTER' and 'BS'!) whereas using value='val' results in

<!--SUBSTER-val-BS:...-->content...<!--SUBSTER-val-BS:/-->

btw.: substersim.py (subster simulation panel) has to be adopted to reflect the usage of 'value=' with BS now

Comment by drtrigon [ Fri, 22 Jun 2012 12:39:38 +0000 ]

Implemented in r341.

For examples confer http://de.wikipedia.org/w/index.php?title=Benutzer:DrTrigon/Spielwiese&oldid=104677569 and following edits/changes.





[DRTRIGON-104] Page section resolution issues Created: Sat, 15 Oct 2011 10:14:28 +0000 Updated: Fri, 23 Dec 2011 19:10:14 +0000 Resolved: Fri, 23 Dec 2011 19:10:14 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.1,
Fix Version/s: 1.2,
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:DrTrigon/SumDisc&oldid=94789394

Description

E.g. entries like:

09. October 2011

Diskussion verändert: Benutzer Diskussion:Saibo - letzte Bearbeitung (145.360 Bytes) von User:Saibo (00:22, 09. Oct. 2011)
Diskussion verändert: Benutzer Diskussion:Xqt - letzte Bearbeitung (69.701 Bytes) von User:UV (22:44, 08. Oct. 2011)

are not really ok, since both pages have a lot of sections, but the alogrithm to break them up (find them) seams to have problems now... (may be since 1.18?).


Comments
Comment by drtrigon [ Fri, 23 Dec 2011 19:10:14 +0000 ]

Solved by bug fixes in r232, r233 and r234. The code has become very similar to older r175 again.





[DRTRIGON-103] Support 'Benutzerin' needed with 1.18wmf1 for sum_disc (may be others too) Created: Fri, 7 Oct 2011 18:36:02 +0000 Updated: Fri, 9 Dec 2011 22:19:44 +0000 Resolved: Fri, 9 Dec 2011 22:19:44 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.1,
Fix Version/s: 1.1,
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified


Description

With the new mw 1.18wmf1 gender specific namespace names were introduced. E.g. 'Benutzer'/'Benutzerin' and so on.

To enable the support for those first update/merge with newest pywikipedia revision - then check whether it is working... May be other changes will be needed. But this update to newest pywikipedia rev. should be a good thing anyway!


Comments
Comment by drtrigon [ Sat, 3 Dec 2011 19:07:29 +0000 ]

Small changes on code necessary only (e.g. DRTRIGON-59 solved finally).

But also 2 bugs in pywikipedia framework appeared:

Comment by drtrigon [ Fri, 9 Dec 2011 22:19:44 +0000 ]

Bugs in pywikipedia framework solved by xqt - thanks for this!

Done with re-merge in r221 and r222. This is also the trigger for a new release v1.1.





[DRTRIGON-102] Enable update frequency config parameter for subster Created: Thu, 6 Oct 2011 17:02:15 +0000 Updated: Sat, 3 Dec 2011 11:36:05 +0000 Resolved: Sat, 3 Dec 2011 11:36:05 +0000

Status: Resolved
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.1,
Fix Version/s: 1.1,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified


Description

Introduce a parameter to adjust, configure or apply a test to be able to better adjust the update frequency or the days on which data are updated.

This is e.g. needed for:


Comments
Comment by drtrigon [ Sat, 3 Dec 2011 11:36:05 +0000 ]

Feature added in r219 to r220.

The same syntax like CRON is used but minute and hour columns are dropped. Here is the supported syntax.





[DRTRIGON-101] Enable mail receiving for subster as data source Created: Fri, 16 Sep 2011 10:09:51 +0000 Updated: Sat, 3 Dec 2011 11:26:34 +0000 Resolved: Sat, 3 Dec 2011 11:26:34 +0000

Status: Resolved
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.1,
Fix Version/s: 1.1,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: https://wiki.toolserver.org/view/Mail

Description

Some external data sources and services support E-Mail data exchange. E.g [1] forbids explicitly to parse data, but offers to send them by mail freely (I think [2] has something similar).

[1] http://www.xe.com/ict/?basecur=EUR&historical=false&month=9&day=16&year=2011&sort_by=code&image.x=43&image.y=11
[2] http://www.bundesbank.de/zahlungsverkehr/zahlungsverkehr_bankleitzahlen_download.php

According to [3] a mail forward for "drtrigon+subster@toolserver.org" was created, now all mail to this address get stored into $HOME/data/subster/mail_inbox and can be parsed by the subster bot (e.g. on daily basis) in order to get the data.

[3] https://wiki.toolserver.org/view/Mail

This mechanism can be used e.g. for [4], [5] (gives just a notice that new data are available) and may be (hopefully) others too.

[4] http://de.wikipedia.org/wiki/Vorlage:Wechselkursdaten
[5] http://de.wikipedia.org/wiki/Vorlage:Infobox_Kreditinstitut/DatenDE

As mentioned on [3] one could also consider to use something similar (e.g. mail forward to "drtrigon+bot_control@toolserver.org") to remote control the bots via mail... ))


Comments
Comment by drtrigon [ Sat, 19 Nov 2011 12:10:56 +0000 ]

Enabled additional feature for XLSX2CSV (openpyxl) conversion in r209.

Comment by drtrigon [ Sat, 19 Nov 2011 12:28:57 +0000 ]

Enable additional feature for PDF2TXT with [1] (PDFMiner), [2] (gfx) or others. Could be problematic thus should be done if needed only!

[1] http://www.unixuser.org/~euske/python/pdfminer/index.html
[2] http://www.swftools.org/gfx_tutorial.html

Comment by drtrigon [ Sat, 3 Dec 2011 11:26:34 +0000 ]

First implementations of feature included in r209 to r218.

Used for [1] ([4]) and others (like ITTF) but not yet for [2] ([5]) since there another approach with 'simple' syntax is tested.

Additional tool DrTrigonBot subster mail queue created.





[DRTRIGON-100] Move templates to correct (better) locations Created: Tue, 13 Sep 2011 21:32:27 +0000 Updated: Tue, 27 Dec 2011 22:11:34 +0000 Resolved: Tue, 27 Dec 2011 22:11:34 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.1,
Fix Version/s: 1.2,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://de.wikipedia.org/wiki/Benutzer:DrTrigonBot

Description

All bot templates that have come to some popularity and are used quite often like:

  • Benutzer:DrTrigon/Entwurf/Vorlage:Subster
  • Benutzer:DrTrigon/Entwurf/Vorlage:SumDisc
  • Benutzer:DrTrigon/Entwurf/Vorlage:Spielwiese

have to be moved/renamed to correct (better) locations (may be including the history - if possible?). The question will be; what are the best locations? Clearly the part 'Entwurf' has to be removed. But is the best place directly in Template/Vorlagen namespace (where they also become "Gesichtet")? ...or should they be left in User namespace? One point to consider is the fact the e.g. Benutzer:DrTrigon/Entwurf/Vorlage:Subster is used in Vorlage:Infobox_Kreditinstitut/DatenAT, .../DatenCH und .../DatenDE where it is important to be "Gesichtet"!


Comments
Comment by drtrigon [ Fri, 16 Sep 2011 10:14:16 +0000 ]

May be move them to the User namespace of the bot itself, like MerlBot [1].

[1] http://de.wikipedia.org/wiki/Benutzer:MerlBot/InAction

Comment by drtrigon [ Mon, 26 Dec 2011 20:25:42 +0000 ]

Essentially there are 2 possible new namespaces either 'User' or 'Template' - as it seams in dewiki the 'Template' ns is rarely used for bot templates. Thus I will use 'User' ns of the bot itself. Like MerlBot.

Try to use language independent names.

Comment by drtrigon [ Mon, 26 Dec 2011 20:26:36 +0000 ]
Benutzer:DrTrigon/Entwurf/Vorlage:Spielwiese

renamed/moved to

Benutzer:DrTrigonBot/Spielwiese

(NOT language independent)

Comment by drtrigon [ Mon, 26 Dec 2011 20:30:47 +0000 ]
Benutzer:DrTrigon/Entwurf/Vorlage:SumDisc

renamed/moved to

User:DrTrigonBot/SumDisc
Comment by drtrigon [ Mon, 26 Dec 2011 21:11:49 +0000 ]
Benutzer:DrTrigon/Entwurf/Vorlage:Subster

renamed/moved to

User:DrTrigonBot/Subster

Additionally we change:

Benutzer:DrTrigon/Benutzer:DrTrigonBot/config.css

to

User:DrTrigon/DrTrigonBot/config.css

to be language independent.

Comment by drtrigon [ Tue, 27 Dec 2011 21:37:40 +0000 ]

Subster template renamed/moved on frrwiki also and thus can now be created on enwiki.

Comment by drtrigon [ Tue, 27 Dec 2011 22:11:34 +0000 ]

The are all moved now - but do not become 'Gesichtet' since the are not in template ('Vorlage') namespace.





sum_disc: all related issues(DRTRIGON-123)

[DRTRIGON-99] Parameter zusätzlich zu 'altsign_list' (z.B. 'primary_sign_...') mit primärer Signatur Created: Wed, 31 Aug 2011 21:06:16 +0000 Updated: Fri, 8 Jun 2012 18:42:10 +0000 Resolved: Fri, 2 Sep 2011 16:58:52 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.1,
Fix Version/s: 1.1,
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: Saibo
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Depends
is a dependency for DRTRIGON-91 false-positive-Meldungen - DÜP Closed

Description

Z.B. in [[Benutzer Diskussion:Jlinde]] das Problem, dass alle Abschnitte mit einer Unterschrift in 'altsign_list' erkannt werden.

Zusätzlich zu der 'altsign_list' so etwas wie 'primary_sign_...' einführen mit primärer Signatur die immer (auch noch) im Abschnitt sein muss.

(Kann auch durch edits NACH BLubot ausgelösst werden, sofern die Seite noch innerhalb der 2000 letzten edits steht)


Comments
Comment by drtrigon [ Fri, 2 Sep 2011 16:58:52 +0000 ]

New parameter 'mainsignneeded_switch' introduced in r168. This parameter can be used in combination with 'altsign_list' and forces the existence of the main users (e.g. BLUbot) signature in each page section in order to be relevant.

If a problem with this parameter or this bug appears again, please RE-OPEN this ticket.





sum_disc: all related issues(DRTRIGON-123)

[DRTRIGON-98] Parameter um max. Zeitdiff zw. Benutzer-Signatur und aktuellen Datum zu beschränken Created: Wed, 31 Aug 2011 21:02:05 +0000 Updated: Fri, 3 Jan 2014 16:23:54 +0000 Resolved: Fri, 3 Jan 2014 16:23:54 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: pywikipedia,
Affects Version/s: 1.1,
Fix Version/s: 1.6,
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Answered Votes (Watches): 0 (1)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Depends
is a dependency for DRTRIGON-91 false-positive-Meldungen - DÜP Closed

Description

Z.B. [[Benutzer Diskussion:KHKauffmann]] war trotz zahlreicher Einträge von BLUbot auf der Seite noch nicht in der History, darum waren keine Informationen über den Zustand der vorherigen Abschnitte bekannt. Der edit von BLUbot am 27. Jul. hat die Seite in den Fokus gebracht. Als Resultat davon wurden alle Abschnitte mit Antworten gemeldet (Suhadi ist nicht in altsign_list).

Darum einen Parameter einzuführen um die max. Zeitdiff zw. der Benutzer-Signatur und dem aktuellen Datum beschränken zu können (nach Wunsch).

(Es betrifft hauptsächlich Seiten und Benutzer die kein Archiv mit Archivierung verwenden, da dort teilweise noch sehr alte Einträge auf der Seite stehen)


Comments
Comment by drtrigon [ Fri, 2 Sep 2011 14:05:53 +0000 ]

Alternative Lösungen wären:

  • unendliche History (nie etwas daraus löschen)
  • oder alle die gemeldeten Abschnitte durch Personen von DüP (altsign_list) abschliessen, mit letztem Eintrag (auch wenn die Diskussion sehr alt ist)
Comment by drtrigon [ Fri, 2 Sep 2011 17:42:52 +0000 ]

Ich favorisiere die letzte Lösung, aber das muss weiter diskutiert werden unter http://de.wikipedia.org/wiki/Benutzer_Diskussion:Saibo#max._Zeitdiff_zw._Benutzer-Signatur_und_aktuellen_Datum.

Comment by drtrigon [ Sat, 3 Sep 2011 08:22:47 +0000 ]

max. Zeitdiff zw. Benutzer-Signatur und aktuellen Datum

Hallo Saibo

Wie Du in {{Benutzer:DrTrigon/Vorlage:JIRA|DRTRIGON-91}} richtig bemerkt hast ist die Lösung zu {{Benutzer:DrTrigon/Vorlage:JIRA|DRTRIGON-98}} nicht ganz simpel, bzw. unproblematisch. Der einfache Vorschlag die Zeitdifferenz beschränken zu können ist nicht ideal. Mir ssind mit dieser Variante bisher 3 Möglichkeiten eingefallen... Und leider ist keine davon gut. Um ehrlich zu sein wenn man z.B. folgenden Eintrag mal betrachtet:
*Neue Diskussion: [[Benutzer Diskussion:KHKauffmann]] → [[Benutzer Diskussion:KHKauffmann#Problem_mit_deiner_Datei|Problem mit deiner Datei]], [[Benutzer Diskussion:KHKauffmann#Probleme_mit_deinen_Dateien|Probleme mit deinen Dateien]], [[Benutzer Diskussion:KHKauffmann#Probleme_mit_deiner_Datei|Probleme mit deiner Datei]], [[Benutzer Diskussion:KHKauffmann#Problem_mit_deinen_Dateien|Problem mit deinen Dateien]] - letzte Bearbeitung von [[User:Suhadi Sadono]]/[[User:BLUbot]] (18:00, 27. Jul. 2011)
der wurde aus falschen Gründen gemeldet (darüber müssen wir nicht diskutieren), '''aber''' gemeldet werden ja nur Einträge bei denen niemand der 'altsign_list' die Diskussion abgeschlossen hat. Also ich bin sehr geneigt mich auf den Standpunkt zu stellen, es wäre doch sinnvoll, wenn einfach jemand da einen Eintrag machen würde, z.B. mit dem aktuellen Status, oder was dann Eure letzte Aktion war nachdem keine Antwort gekommen ist oder so. Kurz gesagt; mein Bot (und ich eigentlich auch) sind der Meinung, dass diese betreffenden Diskussion einfach noch nicht abgeschlossen sind. Ich weiss dass das nach einer "billigen" Lösung oder Antwort tönt... Darum: was meinst Du dazu? Grüsse --[[Benutzer:DrTrigon|Dr]][[Benutzer_Diskussion:DrTrigon|Trigon]] 19:40, 2. Sep. 2011 (CEST)

:Du willst jetzt nicht wirklich auch noch hier eine Diskussion anfangen? Nu lass uns doch bitte zu der Sache im JIIIiiira bleiben. Viele Grüße --[[Benutzer:Saibo|Saibo]] ([[BD:Saibo|<small>Δ</small>]]) 03:38, 3. Sep. 2011 (CEST)

:Nein ein Abschluss der Diskussion durch DÜPler wäre unverhältnismäßiger Aufwand. In den meisten Fällen ist das Abschließen nämlich nicht nötig und bedingte einen weiteren Edit.

Comment by drtrigon [ Sat, 3 Sep 2011 08:47:15 +0000 ]

(dachte diese Diskussion wäre in der wiki besser aufgehoben)

Also was würdest Du vorschlagen?

Aus meiner Sicht:

  1. Lösung: "max. Zeitdiff zw. der Benutzer-Signatur und dem aktuellen Datum" führt wie Du bemerkt hast zu false-negatives
  2. Lösung: "unendliche History" würde das Problem nicht wirklich lösen; es würde immernoch min. 1 mal für jede Seite auftreten die noch nicht in der History ist - und sind doch ein paar
  3. Lösung: "Abschluss der Diskussion durch DÜPler" ist aus meiner Sicht die einzige zuverlässige Methode - die leider Zusatz-Aufwand durch Euch bedeutet (aber Du kannst ja z.B. mit einem Edit alle offenen Disk. auf einer Seite beenden). Ich denke da haupsächlich an einen Kommentar der Art, dass diese/s Diskussion/Problem nicht gelösst ist aber von Euch auch nicht mehr weiter beachtet wird (ohne weitere Interaktion des betreffenden Users)
  4. Lösung: Ist mir gerade neu eingefallen, ev. würde es Euch helfen, wenn der Bot zusätzlich zu den bisherigen Infos noch das Datum der letzten Bearbeitung eines Abschnittes und ev. die Aenderung in der Grösse melden würde? Aber das sind einfach zusätzliche Infos und ändern nichts am Melde-Verhalten...

Du siehst, von den 4 Lösungen die mir bisher eingefallen sind, sind 1. und 2. eigentlich Schrott. Darum bin ich hier auch ein bisschen ratlos. Was wäre Dein Lösungs-Vorschlag?

Comment by saibo [ Mon, 5 Sep 2011 15:49:18 +0000 ]

3. Der Zusatzaufwand ist größer, als die Ersparnis. Abgelehnt.
4. Die Aktualität steckt bei den BLUbotabschnitten ja mittlerweile in der Abschnittsüberschrift, sodass es auffällt wenn etwas altes gemeldet wird (wo aber halt evtl. wirklich jmd. geantwortet hat). Zusätzliche Infos: eher nicht, denn es bläst die Liste wohl unnötig auf und angucken würde man wohl doch jeden Eintrag.

Es gibt eine Methode: Dein Bot soll nur die Abschnitte melden, die sich wirklich aktuell geändert haben. Dazu müsste er für jeden Abschnitt von dem er denkt er sei geändert mal die die History der Seite gucken, und schauen, wie der Abschnitt beim vorherigen Botlauf aussah. Wenn er gleich ist, wurde wohl offensichtlich nichts an dem Abschnitt geändert. Nicht einfach - aber das wäre eine Lösung.

Ansonsten eben die Nulllösung: kein Fix für den Bug. (btw: das ist ein Vorteil von Jiiira - man kann Beiträge editieren - das geht bei BZ nicht)

Comment by drtrigon [ Mon, 5 Sep 2011 20:40:00 +0000 ]

http://etherpad.wikimedia.org/DrTrigonbotfalsepositivesfilter

Wie in DUP chat besprochen.

Comment by bugzilla-exporter [ Fri, 3 Jan 2014 16:23:54 +0000 ]

This bug has been migrated to Bugzilla: https://bugzilla.wikimedia.org/59519





[DRTRIGON-97] More options for sum_cat_disc Created: Sat, 27 Aug 2011 19:10:30 +0000 Updated: Fri, 3 Jan 2014 16:06:17 +0000 Resolved: Fri, 3 Jan 2014 16:06:17 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - toolserver scripts,
Affects Version/s: 1.1,
Fix Version/s: 1.6,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Answered Votes (Watches): 0 (1)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Duplicate
is a duplicate of DRTRIGON-85 Integrate sum_cat_disc as bot Closed
URL: http://toolserver.org/~drtrigon/cgi-bin/sum_cat_disc.py

Description

Flominator requested to get the functionality 'sum_cat_disc' provides built into a bot this is done by subster as described in DRTRIGON-85. Some suggestions on what features should be available are given here:
http://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:DrTrigon&oldid=85471743#Diskussionsbeitr.C3.A4ge_nach_Kategorie

Optionen, die man sich vorstellen könnte:

  • Kategorientiefe / Kategorienausschluss (Möglichkeit Kategorienzweige ausschließen zu können)
  • Dauer bis zum Abräumen
  • Ignorieren von Änderungen durch Bots oder an eingebundenen Vorlagen
  • Ausgabe der Abschnittsüberschrift oder des letzten ändernden Benutzers
  • Mindestanzahl an Änderungen (bevor Seite überhaupt angezeigt wird)

Comments
Comment by flominator [ Sun, 28 Aug 2011 10:21:20 +0000 ]

Since some regional categories might contain strange subcategories (e.g. lots of artist categories under ECHO-Gewinner in Berlin) an option to exclude certain category trees would be good, as well.

Comment by drtrigon [ Sun, 28 Aug 2011 11:08:03 +0000 ]

https://jira.toolserver.org/browse/DRTRIGON-85?focusedCommentId=17600&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-17600

Comment by drtrigon [ Mon, 29 Aug 2011 12:45:20 +0000 ]

Respect https://wiki.toolserver.org/view/Database_access#Slow_queries_and_the_query_killer to 'sum_cat_disc' where needed (Query-Killer is back in action: http://lists.wikimedia.org/pipermail/toolserver-l/2011-August/004305.html).

And consider http://lists.wikimedia.org/pipermail/toolserver-l/2011-August/004322.html also, which means some LIMIT should (may) be added.

Comment by drtrigon [ Tue, 13 Sep 2011 17:30:48 +0000 ]

Verbesserungsoptionen (nach [1] und [2]):

  • deutsches Datum
  • Kategorie für alle Diskussionssammelseiten

[1] http://de.wikipedia.org/wiki/Benutzer_Diskussion:Flominator/Archiv17#sum_cat_disc_usw.
[2] http://de.wikipedia.org/wiki/Benutzer_Diskussion:Flominator#sum_cat_disc_usw.

Comment by drtrigon [ Tue, 13 Sep 2011 20:20:38 +0000 ]

r193: deutsches Datum; Datum in jeglicher durch 'locale' unterstützten Sprache, fall-back ist 'en'

r194: Ausgabe (der Abschnittsüberschrift oder) des letzten ändernden Benutzers; letzter Benutzer und Kommentar hinzugefügt (ev. bekommt der SubsterBot Probleme mit dem regex)

Comment by drtrigon [ Mon, 18 Jun 2012 13:33:21 +0000 ]

== Diskussionszusammenfassung ==

Hallo Doc, kannst du im Skript bitte noch einen Parameter ergänzen, mit dem z.B. die [[:Kategorie:Karlspreisträger]] aus [[Wikipedia:WikiProjekt Aachen/Diskussionen]] entfernen könnte? Gruß, --[[Benutzer:Flominator|Flominator]] 08:12, 13. Jun. 2012 (CEST)

Comment by bugzilla-exporter [ Fri, 3 Jan 2014 16:06:17 +0000 ]

This bug has been migrated to Bugzilla: https://bugzilla.wikimedia.org/59423





F53 (copied from wiki)(DRTRIGON-15)

[DRTRIGON-96] 'subster.py': Kann eigentlich öffentlich werden, da mit der [2] jeder den Bot jetzt auch konfigurieren kann Created: Wed, 24 Aug 2011 20:35:09 +0000 Updated: Thu, 25 Aug 2011 11:02:34 +0000 Resolved: Thu, 25 Aug 2011 11:02:34 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts,
Affects Version/s: 1.1,
Fix Version/s: 1.1,
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified


Comments
Comment by drtrigon [ Wed, 24 Aug 2011 20:38:10 +0000 ]

subster.py can also be used to update the used bot revision number in wiki.

Comment by drtrigon [ Wed, 24 Aug 2011 20:38:38 +0000 ]

Another idea for a bot is given in: http://de.wikipedia.org/w/index.php?title=Wikipedia:Bots/Anfragen&oldid=77370441#Vorlage:FIFA-Rang_aktualisieren

This could be done by something very similar to subster.py and thus would be quite interesting.

Comment by drtrigon [ Wed, 24 Aug 2011 20:38:53 +0000 ]

Another idea for option/parameter handling:

http://de.wikipedia.org/wiki/Benutzer:CopperBot/config.css

Comment by drtrigon [ Wed, 24 Aug 2011 20:39:10 +0000 ]

Support http://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:Grip99&oldid=92230444#Subster.

Comment by drtrigon [ Wed, 24 Aug 2011 20:41:31 +0000 ]

According to the ideas mentioned in DRTRIGON-93 (and http://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:Grip99&oldid=92123510#Subster) following concept:

-bot_control.py should have an additional operating mode (e.g. command line param "-ircrc") that runs all the time (has to be restarted by SGE in case of failures) and watches the IRC RecentChanges channel irc://irc.wikimedia.org#de.wikipedia. In case of a page changes the bot_control.py runs subster.py (as usual) in order to update the page. This would make the subster tags pseudo-dynamic (more dynamic than now). May be to simplify, only pages listed/enrolled on a special page are checked this way.-

This would also avoid to increase the execution frequency of subster.py by use of SGE/cron(ie) since the bot would run all the time - this could open additional techniques (e.g. for panel.py or else) - but also give new issues...

Modified 'articlenos.py' code that can be inserted into 'bot_control.py' as ircbot mode for 'subster.py'. Should then be runned as long-running job and use SGE to re-start in case of errors.

Consider the fact that this is equivalent to substersim.py panel but more wiki-like since no external link or tool is needed.

Comment by drtrigon [ Thu, 25 Aug 2011 11:02:34 +0000 ]

all implemented or tests started in r150.





F53 (copied from wiki)(DRTRIGON-15)

[DRTRIGON-95] 'sum_disc.py': Umstellung auf [1] (ev. 'getbacklinks_switch', 'globwikinotify_switch' u.a. weiter betreiben) Created: Wed, 24 Aug 2011 20:33:12 +0000 Updated: Thu, 25 Aug 2011 14:16:32 +0000 Resolved: Thu, 25 Aug 2011 14:16:32 +0000

Status: Resolved
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.1,
Fix Version/s: 1.1,
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified


Description

[1] http://de.wikipedia.org/wiki/Wikipedia:WikiProjekt_LiquidThreads


Comments
Comment by drtrigon [ Wed, 24 Aug 2011 21:05:29 +0000 ]

könnte auch sum_disc.py dazu bringen, dass er noch unsigniertes nachsigniert (wobei sign-bot scheint schwierig ohne live irc und dann aufwändig, bzw. mach viele edits und ist gefährlich falls er amok läuft) - Zusammenarbeit mit [4]?!

[4] http://de.wikipedia.org/wiki/Benutzer:CopperBot

Comment by drtrigon [ Thu, 25 Aug 2011 14:16:32 +0000 ]

Further action like fully CLOSE-ing the ticket depend strongly on e.g. the plans to use LiquidThreads.





F53 (copied from wiki)(DRTRIGON-15)

[DRTRIGON-94] 'replace_tmpl.py' (für ''Individuelle Vorlagenersetz- und Bearbeitungsaufträge''): weiterentwickeln Created: Wed, 24 Aug 2011 20:13:52 +0000 Updated: Thu, 25 Aug 2011 13:42:19 +0000 Resolved: Thu, 25 Aug 2011 13:42:19 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.1,
Fix Version/s: 1.1,
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified


Description
  • Befehle von max. 20 Ersetzungen können direkt ausgeführt werden, alles andere ''nur nach'' Simulation
  • Simulationen werden ausgeführt und das Resultat mit eindeutiger Id auf Ausgabeseite geschrieben, damit kann der Befehl (durch Angabe der Sim-Id) ausgeführt werden
  • In der Simulation werden alle relevanten Informationen gespeichert und bei Ausführung wieder eingelesen, so können problematische Einträge in der Simulationsausgabe noch modifiziert werden und der Bot nutzt einfach diese Angaben
  • Bei jeder Botbearbeitung wird der Name des Auftraggebers vermerkt
  • Simulationen sollten jeder Stunde, die Ersetzungen 1 mal pro Tag laufen
  • Resultate werden mit Verweisen zu ev. Fehlern oder Problemen auf Ausgabeseite zusammengefasst, zur Nachbearbeitung, falls es ein kleiner Auftrag war, oder die Anpassungen der Sim-Daten nicht ausgereicht haben

Comments
Comment by drtrigon [ Wed, 24 Aug 2011 20:22:50 +0000 ]

Instead of further developing 'replace_tmpl.py' make a wiki interface for 'replace.py'. This interface and 'replace.py' are a more generalized version of 'replace_tmpl.py'. Then implement special features e.g. for template replacement and others!

  • subclass replace.py (fixes.py) bot
  • subclass template.py bot
  • make wiki UI for the functions they provide!!
  • create specialized functions like in replace_tmpl.py!!
Comment by drtrigon [ Wed, 24 Aug 2011 20:23:11 +0000 ]

script_wui.py added (instead of replace_tmpl.py) in r133. Is experimental and beta.

Supports replace.py, template.py and templatecount.py so far.

Comment by drtrigon [ Thu, 25 Aug 2011 13:42:19 +0000 ]

'close_wui.py' is implementation of this concept. Test started (finally) in r151.





[DRTRIGON-93] Automatic 'button pressing' for Wikipedia:PRD Created: Thu, 4 Aug 2011 08:33:17 +0000 Updated: Sat, 6 Aug 2011 10:21:16 +0000 Resolved: Sat, 6 Aug 2011 10:05:56 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.0, 1.1,
Fix Version/s: 1.0, 1.1,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to DRTRIGON-15 F53 (copied from wiki) Closed
URL: http://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:Grip99&oldid=92044326#Subster

Description

Enable to open and store a specific page according to wishes given on
http://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:Grip99&oldid=92044326#Subster
in order to handle the following pages:

Think about on how to integrate this into subster OR create a new (sub)bot...?!?


Comments
Comment by drtrigon [ Thu, 4 Aug 2011 21:24:37 +0000 ]

New feature; 'expandtemplates' added in r134 - may be this is "all you need"...

Comment by drtrigon [ Sat, 6 Aug 2011 10:05:56 +0000 ]

Result in http://de.wikipedia.org/w/index.php?title=Benutzer:Grip99/PRD-subst&oldid=92083499#5._August_2011 looks good. However at least one issue remains the Specialpage substitution does not yet work as expected but is not needed either. If this becomes important please RE-OPEN this or OPEN a new ticket.





[DRTRIGON-92] Botbeiträge sollten bei SumDisc alle mit Bot-Flag Created: Sat, 23 Jul 2011 09:36:20 +0000 Updated: Sat, 23 Jul 2011 09:45:54 +0000 Resolved: Sat, 23 Jul 2011 09:45:54 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: 1.0, 1.1,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Cannot Reproduce Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://de.wikipedia.org/w/index.php?title=Benutzer:DrTrigonBot/ToDo-Liste&oldid=91566420#FEATURE_REQUESTS

Description

Die Botbeiträge sollten bei SumDisc alle mit Bot-Flag erfolgen.

Reported by http://de.wikipedia.org/wiki/Benutzer:IWorld.


Comments
Comment by drtrigon [ Sat, 23 Jul 2011 09:45:54 +0000 ]

Alle Bot-Bearbeitungen die ich unter

http://de.wikipedia.org/w/index.php?namespace=2&tagfilter=&limit=1000&hidebots=&title=Spezial%3ALetzte+%C3%84nderungen

finden kann HABEN ein BOTFLAG.

Falls ein entsprechendes Beispiel auftaucht, bitte ein RE-OPEN in diesem Ticket ZUSAMMEN mit dem Beispiel machen.





sum_disc: all related issues(DRTRIGON-123)

[DRTRIGON-91] false-positive-Meldungen - DÜP Created: Sat, 9 Jul 2011 12:02:38 +0000 Updated: Fri, 3 Jan 2014 16:25:15 +0000 Resolved: Fri, 3 Jan 2014 16:25:15 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: pywikipedia,
Affects Version/s: 1.1,
Fix Version/s: 1.6,
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Answered Votes (Watches): 1 (2)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Depends
depends on DRTRIGON-98 Parameter um max. Zeitdiff zw. Benutzer-Signatur und aktuellen Datum zu beschränken Closed
depends on DRTRIGON-99 Parameter zusätzlich zu 'altsign_list' (z.B. 'primary_sign_...') mit primärer Signatur Closed
URL: http://de.wikipedia.org/w/index.php?title=Benutzer:DrTrigonBot/ToDo-Liste&oldid=87748306#BUGS

Description

false-positive-Meldungen - DÜP
Danke und Viele Grüße --Saibo (Δ) 19:47, 17. Mär. 2011 (CET)


  • bei neuem BLUbot-Abschnitt / Grund:
    • [[Benutzer Diskussion:Vucks]] das selbe Problem wie unten. Interessant (bzw. gut) hierbei ist, dass der BLUbot Edit am folgenden Tag (17.) korrekt in einem neuen Abschnitt erkannt wurde und nicht als Antwort auf den vorherigen (vom 16.).
    • [[Benutzer Diskussion:Jlinde]] das selbe Problem wie unten. Aber noch ein zusätzliches, neues Problem; erkannt werden alle Abschnitte mit einer Unterschrift in 'altsign_list'.
    • [[Benutzer Diskussion:Docteur Ralph]] das selbe Problem wie unten.

Zusätzlich zu der 'altsign_list' so etwas wie 'primary_sign_...' einführen mit primärer Signatur die immer (auch noch) im Abschnitt sein muss. - drtrigon

  • kein Grund erkennbar
    • [[Benutzer Diskussion:Wolfgang Deppert]] Die Seite kam am 1. Sep. 2010 in die History und wurde seit dann überwacht. Am 15. Mar. 2011 wurde sie deshalb neu überprüft, leider waren die Infos in der History nicht verwertbar, weil die Abschnitte falsch zugeordnet waren (hiess "* und † - Symbole"). Darum eine Neu-Meldung. Ich denke ausgelösst wurde das Ganze durch die bug fixes und improvements in r108 (um 13. Feb 2011). Kurz gesagt es ist ein Artefakt der code-Aenderung in r108 (etwa) und 'getSections', der Fehler lag in der alten Version. Kann bei anderen Seiten auch passieren, aber darf nur 1 Mal pro Seite geschehen!

Resolution: Fixed - drtrigon

  • kein Grund erkennbar 2
    • [[Benutzer Diskussion:Nienetwiler]] Problem NICHT KLAR; ev. ähnlich wie bei "comment - 10 July 2011 12:17:19" weiter unten, dass der bot Alg. bezüglich Erkennung (Abschnitt-Einteilung) geändert hat und dann alte und "neue" History nicht gepasst haben... ev. aber auch nicht - kann das nicht (mehr) nachvollziehen.

Resolution: (no idea - may be fixed already and was a bug before) - drtrigon


Comments
Comment by drtrigon [ Sat, 9 Jul 2011 23:42:05 +0000 ]

Ich habe mal den ersten Eintrag dieser Liste geprüft, dort ist zwar 'Suhadi' als letzte Signatur und die wurde von 'altsign_list' entfernt, aber der Eintrag hat sich auch nicht geändert. Sollte also nicht gemeldet werden.

Was mir aber sonst so auffällt; kann es sein, dass sich solcher Fehler jeweils monatlich um den 15. häufen? (und ev. auch um den 28.)
An den Tagen wird die 'History' komprimiert, siehe, z.B.:

http://toolserver.org/~drtrigon/DrTrigonBot/20110315.log
http://toolserver.org/~drtrigon/DrTrigonBot/20110415.log

[...] "RUN BOT: compressing discussion summary"

Was denkt Ihr dazu?

Comment by drtrigon [ Sun, 10 Jul 2011 12:17:19 +0000 ]

Weitere false-positives von Benutzer Diskussion:DrTrigon/SumDisc:

  • Diskussion verändert: Diskussion:Jordansche Normalform → Lesenswerter Artikel - letzte Bearbeitung von User:Tolentino (18:44, 06. May. 2011)
  • Diskussion verändert: Wikipedia Diskussion:WikiProjekt LiquidThreads → Das Aussehen und die Bedienung der Liquid Threads - letzte Bearbeitung von User:Umherirrender (20:29, 19. Jun. 2011)

Grund:

  • [[Diskussion:Jordansche Normalform]] Problem NICHT wirklich KLAR... Kann sein, dass die code Aenderungen um den 13. Feb. (oder davor) einen Einfluss auf die Erkennung (z.B. unterteilen in Absschnitte) hatte. Die Aenderungen auf der Seite am 6. Mai haben sie dann in den Fokus gebracht, obschon die Aenderung am Betreffenden Abschnitt wohl viel früher erfolgt waren...
  • [[Wikipedia Diskussion:WikiProjekt LiquidThreads]] Problem NICHT KLAR... Könnte was Aehnliches gewesen sein, da auch hier zuvor ca. ein halbes Jahr keine Edits waren - dazwischen hat die Bot Erknnungs-Qualität wohl geändert...

Resolution: (may be fixed since was a bug before, but unclear) - drtrigon

(could be a section recognition issue in 'getSections', with possible fix in r178)

Comment by saibo [ Tue, 2 Aug 2011 21:06:00 +0000 ]

Grund: [[Benutzer Diskussion:Ralbisser]] das selbe Problem wie unten.

Es betrifft hauptsächlich Seiten und Benutzer die kein Archiv (mit Archivierung) verwenden, da dort teilweise noch sehr alte Einträge auf der Seite stehen.

Resolution: Not a bug - drtrigon

Comment by saibo [ Tue, 2 Aug 2011 21:44:49 +0000 ]

kein DÜP-Abschnitt:

  • Diskussion verändert: [[Benutzer Diskussion:Hæggis]] → [[Benutzer Diskussion:Hæggis#.C3.9Cberschriftensimulation|Überschriftensimulation]] - letzte Bearbeitung von [[User:Hæggis]] (19:37, 08. Jul. 2011)
    Zusammenfassung erstellt von und um: [[Benutzer:DrTrigonBot|DrTrigonBot]] 04:33, 9. Jul. 2011 (CEST)

Grund: [[Benutzer Diskussion:Hæggis]] Problem; erkannt werden alle Abschnitte mit einer Unterschrift in 'altsign_list'. Ausgelösst von BLubots edit am 1. Jul. 2011 war zur Zeit von Saibos edit am 5. und 8. Jul. 2011 wohl noch innerhalb der 2000 letzten edits.

drtrigon

Comment by saibo [ Wed, 3 Aug 2011 20:21:18 +0000 ]
  • Neue Diskussion: [[Benutzer Diskussion:JakobVoss]] → [[Benutzer Diskussion:JakobVoss#Urheberrechtliches_Problem_mit_deinen_Dateien|Urheberrechtliches Problem mit deinen Dateien]] - letzte Bearbeitung von [[User:Liliana-60]]/[[User:BLUbot]] (06:06, 09. Jul. 2011)
    Das ist ein BLUbot-Abschnitt von 2007.

Grund: [[Benutzer Diskussion:JakobVoss]] das selbe Problem wie unten. Auslöser war der Edit von BLUbot vom 09. Jul. 2011. Und gemeldet hat er einen Eintrag von 2007 als BLUbot noch eine Vorlage verwendet hat - diese sieht so aus als ob jemand geantwortet hätte, weil noch Text nach der Signatur steht.

Resolution: Not a bug - drtrigon

Comment by saibo [ Thu, 11 Aug 2011 00:16:01 +0000 ]

Zusammenfassung erstellt von und um: [[Benutzer:DrTrigonBot|DrTrigonBot]] 04:32, 15. Jul. 2011 (CEST)

  • Neue Diskussion: [[Benutzer Diskussion:Thiel1929]] → [[Benutzer Diskussion:Thiel1929#Problem_mit_deinen_Dateien|Problem mit deinen Dateien]] - letzte Bearbeitung von [[User:Kriddl]]/[[User:BLUbot]] (18:03, 14. Jul. 2011)

FALSCH - wohl wegen 2011-07-14T18:03:44 BLUbot auf der Seite

  • Neue Diskussion: [[Benutzer Diskussion:Aquarius70]] → [[Benutzer Diskussion:Aquarius70#Problem_mit_deiner_Datei|Problem mit deiner Datei]] - letzte Bearbeitung von [[User:NordNordWest]]/[[User:BLUbot]] (18:04, 14. Jul. 2011)

FALSCH da war nur BLUbot aktiv und schon gar nicht in dem Abschnitt.


Grund:

  • [[Benutzer Diskussion:Thiel1929]] das selbe Problem wie unten.
  • [[Benutzer Diskussion:Aquarius70]] das selbe Problem wie unten. Was ist gemeint mit "da war nur BLUbot aktiv und schon gar nicht in dem Abschnitt"? Im gemeldeten Abschnitt waren "Aquarius70" und "Chaddy" aktiv. Passt eigentlich.

Es wäre ev. ein Gedanke wert einen Parameter einzuführen um die max. Zeitdiff zw. der Benutzer-Signatur und dem aktuellen Datum beschränken zu können (nach Wunsch).

Resolution: Not a bug - drtrigon

-----------

  • Benutzer Diskussion:Aquarius70: in dem gemeldeten Abschnitt war Aquarius und Chaddy - ja - aber das sind uralte Antworten, die aktuell gemeldet wurden. Das ist falsch. In dem Abschnitt wurde seit 2009 nichts geändert. --Saibo
Comment by saibo [ Tue, 23 Aug 2011 20:36:58 +0000 ]
  • Neue Diskussion: [[Benutzer Diskussion:KHKauffmann]] → [[Benutzer Diskussion:KHKauffmann#Problem_mit_deiner_Datei|Problem mit deiner Datei]], [[Benutzer Diskussion:KHKauffmann#Probleme_mit_deinen_Dateien|Probleme mit deinen Dateien]], [[Benutzer Diskussion:KHKauffmann#Probleme_mit_deiner_Datei|Probleme mit deiner Datei]], [[Benutzer Diskussion:KHKauffmann#Problem_mit_deinen_Dateien|Problem mit deinen Dateien]] - letzte Bearbeitung von [[User:Suhadi Sadono]]/[[User:BLUbot]] (18:00, 27. Jul. 2011)

FALSCH - wohl wegen edit von BLUbot 18:00, 27. Jul. 2011 (CEST) auf der BD


Grund: [[Benutzer Diskussion:KHKauffmann]] war trotz zahlreicher Einträge von BLUbot auf der Seite noch nicht in der History, darum waren keine Informationen über den Zustand der vorherigen Abschnitte bekannt. Der edit von BLUbot am 27. Jul. hat die Seite in den Fokus gebracht. Als Resultat davon wurden alle Abschnitte mit Antworten gemeldet (Suhadi ist nicht in altsign_list).

Resolution: Not a bug - drtrigon

:Was ist das hier denn für deine unübersichtliche Antwortstruktur? Brrr
:Also wenn du alle Fehler als notabug ansiehst, dann lass ich eben das reporten, sorry - kann ich nicht nachvollziehen. Der Bot hat Käse gemeldet. Punkt. --Saibo

Comment by drtrigon [ Wed, 31 Aug 2011 07:38:54 +0000 ]
  1. Die Antwortstruktur ist gerade nicht ideal oder schön - liegt aber unter anderem auch daran, dass Du jeden Bug als comment angehängt hast anstelle davon einen neuen zu eröffnen. Wollte ich dann machen wenn ich mir mal einen Ueberblick verschafft habe, weil ich dann sagen kann welche Fehler zusammengehören und welche nicht (aber ich bin gar noch nicht fertig...).
  2. Wenn DU das so siehst, dann höre ich auch auf zu arbeiten. Wenn Du Dir aber die Zeit nehmen würdest, könntest Du hier auch schon Lösungsvorschläge entdecken (wie z.B. ein Parameter zum Begrenzen der max. Zeitdiff). Aber ich kann nichts daran ändern, dass das aus meiner Sicht der Bot-Funktionsweise KEIN BUG ist. Der Bot soll false-positves (aber eben KEINE false-negatives) melden.
  3. Ich war bisher froh über Deine Rückmeldungen, aber wenn Du es als solche Belastung empfindest, dann sollten wir uns eine andere Möglichkeit überlegen...?
Comment by saibo [ Wed, 31 Aug 2011 10:14:22 +0000 ]
  • Ich habe alles als Kommentar angehängt, da alles false-positives sind - so wie der Bugtitel heißt.
  • Wie kann man hier auf einen Kommentar antworten? In Bugzilla weiß ich wie das geht. In Wikipedia auch - hier nicht.
  • Funktioniert bei dir das hier-eingeloggt-bleiben? Ich muss mich jedes Mal neu anmelden.
  • Das aufführen der falsepositives ist nicht so sehr aufwendig. Aber wenn du sie als "kein bug" ansiehst, dann wirst du sie auch nicht fixen und daher ist das aufführen hier unnötig. Ich sehe sie als Bug an, da sie nicht meiner Erwartung an den Bot entsprechen - da hast du wohl eine andere. Meine Erwartung: Der Bot soll nur die Abschnitte melden, wo eine Antwort auf einen Kommentar von "mir" hinterlassen wurde.
  • Eine max-Zeitdiff könnte man machen. Sind aber eher ein Workaround statt eine Lösung. Und manchmal kommen Antworten auf uralt-BLUbot-Meldungen doch vor (Newbies). Würde also auch zu false-negatives führen.
Comment by drtrigon [ Wed, 31 Aug 2011 15:14:02 +0000 ]
  • Antworten auf ein Kommentar ist wohl ein neuer Kommentar - bin auch kein Experte
  • Was meinst Du mit "jedes mal" neu anmelden - ich muss etwa 1 mal pro Tag (aber mein E-Mail Passwort gebe ich häufiger ein...
  • Nein entschuldige das siehst Du völlig falsch:
    • Nur weil es als 'not a bug' markiert ist, heiss das noch lange nicht dass es nicht z.B. ein 'feature request' oder 'task' sein kann, der immer noch einer Bearbeitung im Sinne eines 'fix' oder ähnliches benötigt; kurz 'not a bug' heisst nicht dass ich es als erledigt ansehe, der status ist ja noch nicht 'resolved' oder 'closed'. Wenn ich diese Fehler einfach hätte ignorieren oder übergehen wollen, hätte ich den bug direkt einfach schliessen können. So ist es aber nicht, keine Sorge!
    • Ja das mit Erwartungen ist so ne Sache... Du musst unterscheiden zwischen dem (mehr oder weniger gut) definiertem Verhalten das der Bot haben soll und dem realen Verhalten auf Grund von äusseren Umständen. Ich denke beim definierten Verhalten sind wir uns einig sonst müsste ein 'feature request' her zu neuen Funktionen. Beim realen Verhalten (um das geht es hier) musst Du Abstriche machen, weil wir keinen Einfluss auf die äusseren Umstände haben. Ein Beispiel; der Bot muss davon ausgehen, dass alle Texte die er verarbeitet korrekten Wiki-Syntax haben, aber das ist leider nicht immer der Fall und dann kann z.B. fast alles passieren. (aber das übelste in dem Bereich sollte abgefangen werden) Somit landen wir da wo wir sind, es wird immer Fehler geben - die Frage ist wieviele es davon gibt und wie gut sie als solche erkennbar sind (u.a.). Die Frage nach der Anzahl bzw. Häufigkeit der Fehler möchte ich übrigens direkt an Dich richten?
    • Solltest Du hingegen Möglichkeiten kennen Bots im Allgemeinen (oder nur diesen hier) zu 100% fehlerfrei hinzubekommen, wäre ich sehr interessiert!?
  • Deine Einwände zu der max. Zeitdiff sind korrekt - es ist immer ein Trade-off. Ob Du das als Work-a-round sehen wills... ist nicht falsch, diese Sichtweise. Ich nutze work-a-round meist im Bezug direkt auf code und für wirklich hässliche Sachen - aber auch das ist ja Geschmacksache. Du solltest Dir hier im klaren sein, dass schon einige solche Sachen in Nutzung sind, angefangen bei 'checkedit_count', denn auch dort ist es theoretisch möglich, dass BLUbot mehr als 2000 Edits pro Tag macht. Schlussendlich sind besonders in diesem Bot viele Approximationen gemacht worden - die aber alles in allem nicht soo übel funktionieren (hoffe ich . Der Grund ist unter Anderen, dass gewisse essentielle Funktionen (z.B. das Zerlegen von Seiten in Abschnitte) nicht effizient mit dem aktuellen mw api Interface gemacht werden können. Mein Funktion ist gut (sag ich mal) - KANN aber nicht perfekt sein, weil Informationen fehlen.
Comment by saibo [ Wed, 31 Aug 2011 18:06:11 +0000 ]
  • Ich habe "Remember my login on this computer" beim Anmelden angeklickt, aber die drei Cookies, die mir Jira gibt, haben nur eine Gültigkeit bis "Ende der Sitzung". Unter "remember me" stelle ich mir eher Tage, Wochen oder gar Monate vor - nicht aber nur bis die Browsersitzung zuende ist. Gut - unrelated zu dem Bug natürlich - aber das zu meiner Bugtracker-für-Wikipedia-Abneigung.
  • Gut, dann ignoriere ich deinen Kommentar "Not a bug" einfach - denn jene Wirkung hat er: gar keine.
  • Ich versuch(t)e alle Fehler hier einzutragen. Wenn du wirklich die Rate wissen willst, dann müsste man mal in die Versionsgeschichte der Botausgaugabeseite gehen und die gesamten Abschnitte des Zeitraums hier zählen. Es kann aber auch sein, dass ich nicht immer alle Fehler hier hingeschrieben hatte (u.a. weil ich keine Lust hatte wieder das Passwort herauszusuchen). Die Fehler hier sind also eine Untergrenze. Wenn ich hier so drüberscrolle scheint mir, dass da einige Fehlen.
  • Ich weiß nicht, welche Hürden dir die Mediawikisoftware so vorlegt. Ja, generell arbeitet der Bot super. Ich möchte nicht manuell alle Edits von BLUbot nachgucken.
Comment by drtrigon [ Wed, 31 Aug 2011 21:24:30 +0000 ]
  • "Remember my login on this computer" mache ich nie - aber das hilft Dir auch nicht weiter - sorry, bin echt überfragt - was sagen die Admins dazu?
  • Ja die Rate muss nicht präzise sein... ...aber z.B. so was wie 1%, 10%, 100% falsch... Ne Grössenordnung... Alles andere müsste man systematisch bestimmen.
  • Die Mediawiki-Software ist halt einfach nicht für so revolutionär neue Ideen gewappnet... ) Im ernst es kommen immer wieder neue praktische Funktionen dazu, aber der Bot ist ja eigentlich darum entstanden, weil ich fand dass die Mediawiki Beobachtungsliste/watchlist nicht umfangreich genug ist - sprich weil ich grundsätzlich andere Vorstellungen hatte in dem Bereich als die Developper upstream... )
  • Hört man immer wieder gerne (das der Bot i.O. arbeitet) - eben es ist ein Trade-off zw. was ist halt so wie's ist (also teilweise fehlerhaft) und was kann doch verbessert werden und bedarf eines Trittes in den Allerwertesten des Entwicklers...
  • Ich denke hab jetzt alle Meldungen zuordnen können, ausser folgender (3 Stück):
    • 17. Mär. 2011 / kein Grund erkennbar 2 (1 Stück)
    • 10 July 2011 (2 Stück)
Comment by saibo [ Wed, 31 Aug 2011 21:32:53 +0000 ]

*JIIRA: Was die Admins sagen? Keine Ahung - ist mir auch recht egal - ich würde den Käse hier ja einfach nicht benutzen, wenn es nicht gewisse Leute gäbe, die es unbedingt wollen. Achja.. smileys kann man wohl auch nicht abschalten. Ich habe jetzt hier übrigens eine nichtexistente Mailadresse in jira eingegeben. Die E-Mails die ich bekomme sind völligst nutzlos, weil man nicht sieht, was nun überhaupt geändert wurde . Statt dessen spammt es mein Postfach voll. ich habe ja noch Updatescanner....

  • ja, ich schaue demnächst mal, dass ich ne Rate finde. Geschätzt 20%.
Comment by drtrigon [ Thu, 1 Sep 2011 07:51:48 +0000 ]
  • Wenn Du eine echte Alternative zu JIRA kennts nur zu; z.B. bugzilla?
  • Ne Rate von 20% ist etwas viel - weniger wär schon wünschens-wert (mal schauen)
  • Wir können Die Diskussion gerne in die Wiki verlagern (JIRA ist nützlich zum verwalten der Fehler - das ist alles)
Comment by saibo [ Thu, 1 Sep 2011 22:28:09 +0000 ]

bugzilla hat auch seine Probleme (wie auf meiner Talkseite besprochen). Aber afaik bleibt man da wenigstens länger eingeloggt. Wenn auch nicht so lang wie bei Wikipedia.

Ne, jetzt bleiben wir hier. Ich würde halt z.B. ein Bugtracking im Wiki aufsetzen. So ähnlich wie WP:PRD - mit Unterseiten für jeden Bug. Und die aktiven Bugs werden auf eine Übersichtsseite in eine sortierbare Tabelle eingebunden. In einem onlyinclude-Bereich der Unterseiten ist dann jeweils der Inhalt einer Tabellenzeile: Kurzbeschreibung/Name, Status, Reporter, Bearbeiter. Auf der Unterseite kann man das aktualisieren und Kommentieren (so wie hier). Dürfte nicht allzu schwer gehen - vor allem, wenn die Benutzer alle keine DAUs sind. Die erledigten Bugs werden entweder auf der Übersichtsseite entfernt und auf eine Archivübersichtsseite eingebunden oder einfach dringelassen.

Comment by drtrigon [ Fri, 2 Sep 2011 12:34:49 +0000 ]

Ich hatte selbst was versucht zu basteln, war fürher in Betrieb, solltest Du ja ev. gesehen haben unter [1] (am Ende der Seite) bzw. verschoben auf [2]... Wenn Du meine direkte, offene Meinung willst: BEIDES IST MUELL! Der Aufwand für Verwaltung und die Uebersichtlichkeit ist immer mangelhaft. Man muss Stunden darauf verwenden das Zeugs zu verwalten, ordnen, querverlinken, erstellen, abhaken usw... Aber hier ist schon so ein System (gleich schlecht) aber ich muss nicht noch Vorlagen und Konzepte ausarbeiten, wie man es wohl am besten schlecht anstellt... ) (mag mich gar nicht damit befassen) und das hier ist in jedem Fall besser als was unter [1] und [2] war/ist, weil dort nicht mal die Problem-ID's sauber und eindeutig waren.

Das System hier ist sowieso in Betrieb für den TS mit dem ich arbeite, von da her ist es für mich egal - ich brauche es sowieso.

Also kurz und gut; ich bin offen für jede Neuerung und Verbesserung, ABER habe keine Lust auch nur 5 Minuten (ok 5 ev. schon dafür zu verschwenden - also wenn Du mir helfen möchtest oder Hand bieten würdest??!? Von mir aus kann man z.B. ja auch ein System extra für Euch (also DüP) erstellen, dann kann ich jeweils die einzelnen Bugs hierhin übernehmen, bei Euch verlinken und abhaken oder so... Wenn es sich bewährt, dann von mir aus für den ganzen Bot und alle Probleme, Features usw.

Was auch noch ist; FishEye sollte theoretisch mit JIRA verlinkbar sein, d.h. wenn Du z.B. mal unter [3] schaust, dort sollten Kommentare wie "DRTRIGON-xx" direkt hierher verlinken und hier "rXXX" wiederum zurück nach FishEye. So könnte man die Tickets hier direkt den code Aenderungen zuordnen, was auch praktisch wäre...

[1] http://de.wikipedia.org/wiki/Benutzer:DrTrigonBot/ToDo-Liste
[2] http://de.wikipedia.org/wiki/Benutzer:DrTrigonBot/ToDo-Liste/DONE
[3] https://fisheye.toolserver.org/changelog/drtrigon

Comment by drtrigon [ Fri, 2 Sep 2011 12:36:20 +0000 ]
  • 3 Issues/Problems open and unclear:
    • 17. Mär. 2011 / kein Grund erkennbar 2 (1 Stück)
    • 10 July 2011 (2 Stück)

(may be all have the same reason, but this is just a speculation)

Comment by saibo [ Sat, 3 Sep 2011 01:45:01 +0000 ]

Fisheye interessiert mich schlicht nicht.

Nun - eine ProblemID könnte man ganz leicht bekommen, wenn man den aktuellen Timestamp nähme. Ich würde mithelfen eine Vorlagen/Unterseitenbasierte Lösung zu basteln - allerdings hast du dann natürlich keine fisheyeintegration und für solche Bugs, wo nur du einen kurzen Kommentar schreibst, ists Overkil mit dennoch zu viel Verwaltungsaufwand Einbindung der Unterseite in die Übersichtsliste. Für länge Diskussionen , wie diese, wären Unterseiten prkatisch - sonst nicht. Insofern (Fisheyeintegration und kleiner Overhead bei Minibugs) treten wir eben die Idee in die Tonne und bleiben bei dem Stand, dass Programmierer, die sich wie Programmierer fühlen wollen nicht richtig mit WP zusammenarbeiten können.

Comment by saibo [ Sat, 3 Sep 2011 01:47:20 +0000 ]

Siehe DRTRIGON-98.

Comment by drtrigon [ Sat, 3 Sep 2011 08:34:38 +0000 ]

Der Punkt ist folgender, warum vergelichst du die Wikipedia mit JIRA? Das ein soll das andere nicht ersetzen... Das eine ergänzt das andere!
Und mir geht es darum die Bug usw. einfach und übersichlich verwalten zu können...

Aber wie gesagt, Du kannst die reports ja alternativ hier melden (wie Du es ursprünglich getan hast) und falls es Dir hilft JIRA einfach komplett ignorieren. Weil in der Wiki die Einträge ja auch als erledigt (durchgestrichen) markiert werden. Das ist zwar für mich etwas umständlicher, aber werde es schon überleben...

Comment by drtrigon [ Fri, 11 May 2012 15:30:21 +0000 ]

http://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:Xqt&oldid=103108994#DrTrigonBot_und_.22BLUbot.22

Wegen der False-Positives, schau Dir doch mal Deine Regex an, vielleicht läuft da was schief. Beispiel: [Benutzer Diskussion:Dieter rogge #Problem mit deiner Datei (01.05.2012)] wird ja zu [Benutzer_Diskussion:Dieter_rogge#Problem_mit_deiner_Datei_.2801.05.2012.29] encodiert. Da ist es zu [Benutzer_Diskussion:Dieter_rogge#Problem_mit_deiner_Datei_2] wie, http://de.wikipedia.org/w/index.php?title=Benutzer%3ABLUbot%2FAntworten&diff=102726250&oldid=102642160 hier aufgeführt, nicht weit. Weil das vorwiegend beim BLUbot auftauchte, habe ich als Übeltäter die Klammern im Verdacht.

Comment by bugzilla-exporter [ Fri, 3 Jan 2014 16:25:15 +0000 ]

This bug has been migrated to Bugzilla: https://bugzilla.wikimedia.org/59524





sum_disc: all related issues(DRTRIGON-123)

[DRTRIGON-90] sequence of sometimes occuring "KeyError: u'parse'" Created: Sat, 9 Jul 2011 11:14:57 +0000 Updated: Fri, 3 Jan 2014 16:25:27 +0000 Resolved: Fri, 3 Jan 2014 16:25:27 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: pywikipedia,
Affects Version/s: 1.1,
Fix Version/s: 1.6,
Security Level: Public (all users)

Type: Sub-task Priority: Minor
Reporter: drtrigon Assignee: drtrigon
Resolution: Answered Votes (Watches): 0 (1)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://de.wikipedia.org/w/index.php?title=Benutzer:DrTrigonBot/Status&direction=prev&oldid=90413407

Description

The error:

Traceback (most recent call last):
File "/home/drtrigon/pywikipedia/bot_control.py", line 225, in run
self.bot.main()
File "/home/drtrigon/pywikipedia/sum_disc.py", line 1307, in main
bot.run()
File "/home/drtrigon/pywikipedia/sum_disc.py", line 369, in run
self.getLatestRelevantNews()
File "/home/drtrigon/pywikipedia/sum_disc.py", line 747, in getLatestRelevantNews
entries = PageSections(page, self._param)
File "/home/drtrigon/pywikipedia/sum_disc.py", line 1138, in _init_
sections = page.getSections(minLevel=1)
File "/home/drtrigon/pywikipedia/dtbext/dtbext_wikipedia.py", line 50, in <lambda>
obj._dict_[\'getSections\'] = lambda *args, **kwds: Page._dict_[\'getSections\'](obj, *args, **kwds)
File "/home/drtrigon/pywikipedia/dtbext/dtbext_wikipedia.py", line 160, in getSections
r = result[u\'parse\'][u\'sections\']
KeyError: u'parse'

occurs from time to time - look at [1] and older changes. As temporary work-a-round the error raised is converted to

pywikibot.Error('Problem occured during data retrieval for sections in %s!' % self.title(asLink=True))

and the 'result' dict gets printed into the log file to be able to track the origin of this down.

In future there should be a final solution to this bug:

  • either keep the current code (without the print) which converts the error such that later the whole page gets processed in one part (fall back)
  • or add a loop with e.g. maximal 3 re-tries (like for low level url retrieval) and then use e.g. the same fall back like before

[1] http://de.wikipedia.org/w/index.php?title=Benutzer:DrTrigonBot/Status&direction=prev&oldid=90413407


Comments
Comment by drtrigon [ Sat, 6 Aug 2011 13:01:29 +0000 ]

With r135 the bot should notify SGE in case of errors and be re-runned by SGE. This could also solve that problem.

Comment by drtrigon [ Thu, 25 Aug 2011 14:48:42 +0000 ]

Seams to occur rarely. Since creation of this ticket and introduction of work-a-round in code at '09 July' until today '25 Aug.' no such event was logged. You should find a debug output prior to a "Problem resolving section data" message, as you can see by looking into 'dtbext.wikipedia.getSections'.

Comment by drtrigon [ Fri, 23 Dec 2011 19:37:28 +0000 ]

The code introduced in r158 to track down this issue is somehow useless since it triggers now a "Problem resolving section data, processing the whole page at once..." which is the most often triggered problem.

Thus the print in front of the raise was enhanced with the message "Query result:" - this string should be simpler to find in the log files in order to reproduce this bug.

Comment by bugzilla-exporter [ Fri, 3 Jan 2014 16:25:27 +0000 ]

This bug has been migrated to Bugzilla: https://bugzilla.wikimedia.org/59525





[DRTRIGON-89] Bot misses changes Created: Fri, 4 Mar 2011 22:03:25 +0000 Updated: Wed, 13 Jul 2011 22:13:56 +0000 Resolved: Wed, 13 Jul 2011 22:13:56 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.0, 1.1,
Fix Version/s: 1.0, 1.1,
Security Level: Public (all users)

Type: Bug Priority: Critical
Reporter: drtrigon Assignee: merl
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://de.wikipedia.org/w/index.php?title=Wikipedia_Diskussion:Meinungsbilder/Tempor%C3%A4re_Aussetzung_der_L%C3%B6schdiskussion&oldid=86053077#tempor.C3.A4r_aussetzen

Description

As reported in chat:

(10:48:10) Merlissimo: http://de.wikipedia.org/wiki/Wikipedia_Diskussion:Meinungsbilder/Tempor%C3%A4re_Aussetzung_der_L%C3%B6schdiskussion#tempor.C3.A4r_aussetzen hätte heute sicher auftauchen müssen

the bot misses a lot of changes; Merlissimo got nothing the last few days... May be the restricted backlinks list is an issue? Or he does too many changes a day (> 500). Since the bot log http://toolserver.org/~drtrigon/DrTrigonBot/20110304.log shows no error...


Comments
Comment by drtrigon [ Fri, 4 Mar 2011 22:07:06 +0000 ]

Is it possible that you have more than 500 changes a day? If not I will switch off backlink restriction to test if this solves the issue.

Comment by drtrigon [ Wed, 13 Jul 2011 22:01:04 +0000 ]

merl (Merlissimo) in chat today:

(18:12:37) Merlissimo: http://de.wikipedia.org/wiki/Wikipedia_Diskussion:Sprachen#Incubator-Status_.E2.80.9Elocked.E2.80.9C_und_.E2.80.9Eapproved.E2.80.9C_scheinbar_.C3.BCberfl.C3.BCssig hätte gemeldet werden müssen

Comment by drtrigon [ Wed, 13 Jul 2011 22:10:45 +0000 ]

The same issue for IWorld:

http://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:DrTrigon&oldid=91203968#Frage

Comment by drtrigon [ Wed, 13 Jul 2011 22:11:16 +0000 ]

accidentially closed, it's too late

Comment by drtrigon [ Wed, 13 Jul 2011 22:13:56 +0000 ]

Looking at all I have to recognize the Signature of Merlissimo is as follows:

[[Benutzer:Merlissimo/Sig|Merl]][[Benutzer Diskussion:Merlissimo/Sig|issimo]] 00:20, 4. Jul. 2011 (CEST)

this does not match any of the regex given in 'checksign_list' mainly because of the missing '--' in front:

--[[Benutzer:Merlissimo/

so the parameter 'checksign_list' has to be modified for 'Merlissimo' as well as for others e.g. 'IWorld'.

Further 'checksign_list' has to be published and documented. As well as all users have to be informed about this possible issue.

(Re-Open this ticket if the issue persists; checks for Merlissimo as well as for IWorld looked ok)





[DRTRIGON-88] Enable Beautiful Soup power for Subster Created: Wed, 23 Feb 2011 20:13:19 +0000 Updated: Tue, 19 Jul 2011 12:21:19 +0000 Resolved: Tue, 19 Jul 2011 12:21:19 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.0, 1.1,
Fix Version/s: 1.0, 1.1,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://www.crummy.com/software/BeautifulSoup/

Description

Introduce a template parameter like e.g. 'beautifulsoup=True' to switch to mode where all tag become:

<!--SUBSTER-BS:html.head.title.contents--><!--SUBSTER-BS:html.head.title.contents-->

and their content get replaced with the data retrieved from:

eval("BeautifulSoup().html.head.title.contents")

which means the text after 'BS:' gets executed as python code (more or less). Attention for violent code intrusion...


Comments
Comment by drtrigon [ Tue, 19 Jul 2011 12:15:53 +0000 ]

Simple code intrusion with ';' is not possible as it seams (which is good):

Traceback (most recent call last):
File "test.py", line 774, in <module>
print eval('BeautifulSoup.BeautifulSoup(a).%s' % item[1])
File "<string>", line 1
BeautifulSoup.BeautifulSoup(a).body; print 'hallo'
^
SyntaxError: invalid syntax

But there are some points to consider, though: http://lybniz2.sourceforge.net/safeeval.html

Comment by drtrigon [ Tue, 19 Jul 2011 12:18:09 +0000 ]

Code done in r129.

Some first tests in http://de.wikipedia.org/w/index.php?title=Benutzer:DrTrigon/Spielwiese&diff=91416840&oldid=91416714

Comment by drtrigon [ Tue, 19 Jul 2011 12:21:19 +0000 ]

As mentioned: code done in r129.

The feature is not documented yet because in testing stage only. If it works well, it has to be documented and included into 'substersim.py'.





[DRTRIGON-87] Bot ERROR mail cannot be sent sometimes Created: Sun, 20 Feb 2011 11:44:05 +0000 Updated: Sun, 4 Mar 2012 12:19:23 +0000 Resolved: Thu, 25 Aug 2011 10:54:52 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts,
Affects Version/s: 1.0, 1.1,
Fix Version/s: 1.1,
Security Level: Public (all users)

Type: Bug Priority: Critical
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to DRTRIGON-119 More/better SGE integration and interaction in bot_control Closed
URL: http://toolserver.org/~drtrigon/DrTrigonBot/20110220.log

Description

2011/02/20 12:06:49:: Sending mail "Bot ERROR" to "DrTrigon" as notification!
2011/02/20 12:06:49:: !!! WARNING: mail could not be sent!

Even though it was tested explicit it seams that it is very frequent for bot ERROR mails not to be sent to me. This is a critical issue since it may happen that I am not informed about such a crash. Another point is this error reporting has to be reliable ! This might be an issue with SGE too, thus look also at http://lists.wikimedia.org/pipermail/toolserver-l/2011-February/003901.html and its follow-ups.


Comments
Comment by drtrigon [ Sun, 10 Jul 2011 21:17:06 +0000 ]

First attempt to solve this issue is to add max. 3 retries for the mail sending. Additionally output of the error occurred was added to get more info.

Committed revision 124.

If this does not work try to add:

  • delay between attempts
  • re-create user object or do something else to retrieve new token
Comment by drtrigon [ Tue, 26 Jul 2011 08:28:53 +0000 ]

Look at http://lists.wikimedia.org/pipermail/pywikipedia-l/2011-July/006885.html and follow-ups for a discussion about this issue.

Comment by drtrigon [ Thu, 25 Aug 2011 10:54:52 +0000 ]

Solved by introducing a work-a-round in r149.
Bug report https://sourceforge.net/tracker/?func=detail&aid=3397944&group_id=93107&atid=603138 filed in order to solve this issue upstream. Then in fact the work-a-round would not be needed anymore, but does no harm also.





[DRTRIGON-86] Test the re-write branch an decide what parts to migrate Created: Sun, 20 Feb 2011 00:09:08 +0000 Updated: Fri, 3 Jan 2014 16:26:11 +0000 Resolved: Fri, 3 Jan 2014 16:26:11 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.3,
Fix Version/s: 1.6,
Security Level: Public (all users)

Type: Task Priority: Critical
Reporter: drtrigon Assignee: drtrigon
Resolution: Answered Votes (Watches): 0 (2)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to TS-1583 Install lua dev(el) package(s) Open
is related to TS-1554 Check if DrTrigonBot (framework) can run on linux Open
is related to DRTRIGON-131 Reduce bot_control.py influence, try to unlink features Closed
is related to DRTRIGON-128 Future improvements as duplicate detection, classification and code clean-up (TD-018, TD-019, TD-020) Closed
is related to DRTRIGON-130 Enable Wikidata support Closed
is related to DRTRIGON-129 Check to usefulness and capabilities of Labs (vs. TS) Closed
URL: http://lists.wikimedia.org/pipermail/pywikipedia-l/2011-January/006627.html

Description

...or (may be) to migrate no parts at all. But test the re-write and look what it is able to do. May be some parts could be done better, easier, faster with the re-write branch.

Look at (Pywikipedia-l) Running the rewrite branch as developer/power-user and (Pywikipedia-l) Running the rewrite branch as developer/power-user (windows).


Comments
Comment by drtrigon [ Mon, 12 Sep 2011 22:15:36 +0000 ]

Another important strategic decision to take (however may be not in the same context) is to drop the files from framework out of the repository and replace by svn:externals from correct revision, look at e.g. [1] and use

svn propedit svn:externals .

in the editor enter

pywikipedia -r8990 http://svn.wikimedia.org/svnroot/pywikipedia/trunk/pywikipedia

to get e.g. revision 8990 from framework repository. Before doing this those files have to be removed from repo. A thing to check is what happens with additional files in directories got by externals...

[1] http://svnbook.red-bean.com/en/1.5/svn.advanced.externals.html

Comment by drtrigon [ Mon, 9 Jul 2012 12:57:28 +0000 ]
  • subster.py with subster_irc.py will be added to trunk again (may be convert it to rewrite then - don't forget to add wikidata support)
  • script_wui.py is in a very early stage and needs a framework with proper interfaces and will thus be converted to rewrite as next
  • catimages.py has to be cleaned up (DRTRIGON-128) and might be easy to convert to rewrite during this clean up

After those steps I should have become familiar with rewrite and will thus be able to decide whether it is useful to convert the trunk bot scripts to rewrite also or will it be fine to keep them in trunk under occasional maintenance.

Comment by drtrigon [ Fri, 12 Oct 2012 08:19:33 +0000 ]

As python 2 finishes with 2.7 [1] do the migration to python 3 [2] in the same rewriting step.

[1] http://www.python.org/dev/peps/pep-0404/
[2] http://docs.python.org/library/2to3.html

Comment by drtrigon [ Sun, 2 Dec 2012 19:30:53 +0000 ]

subster.py with subster_irc.py added to trunk in r10761 and r10762. Conversion to rewrite and if possible to python 3 can and has to start now!

Comment by drtrigon [ Wed, 12 Dec 2012 16:56:00 +0000 ]

script_wui.py moved from trunk to rewrite in r427, r428 and r429. Activated in r430 and is also used as replacement for bot_control.py in order to run clean_sandbox.py.

Comment by drtrigon [ Fri, 28 Dec 2012 15:23:15 +0000 ]

rewrite further migrated to labs, see DRTRIGON-129.

Comment by drtrigon [ Wed, 20 Feb 2013 19:39:16 +0000 ]
  • catimages.py added to trunk in r11097 ...
  • script_wui.py added to rewrite in r11098 ...
    ...as they are. They both still have to be improved as mentioned before!
Comment by drtrigon [ Sat, 27 Apr 2013 21:00:34 +0000 ]

Everything in rewrite was merged upstream, the external(s) libraries too.

Considering trunk everything EXCEPT the external(s) libraries is upstream. This includes unittesting, docs and bot_control.py that was renamed to pwb.py.

Open things left are the migration from trunk to rewrite along with needed framework adoptions and following points:

  • subster.py: jira ticket for [[Vorlage:Subster]] MOVE (NOT redirect) Template here because of "Sichten" and change on all pages with script_wui.py/replace.py
  • substersim.py: test all examples!! (cross site scripting example val9)
  • panel.py: more rrdtool (if possible), may be use bokeh
Comment by drtrigon [ Sun, 19 May 2013 09:55:26 +0000 ]

In trunk everything is now upstream! (Except some minor testing/debuging code and changes as in any usual development process)

Comment by bugzilla-exporter [ Fri, 3 Jan 2014 16:26:11 +0000 ]

This bug has been migrated to Bugzilla: https://bugzilla.wikimedia.org/59528





[DRTRIGON-85] Integrate sum_cat_disc as bot Created: Sat, 19 Feb 2011 08:58:50 +0000 Updated: Sun, 28 Aug 2011 11:02:44 +0000 Resolved: Sat, 27 Aug 2011 19:26:07 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts,
Affects Version/s: 1.1,
Fix Version/s: 1.1,
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Duplicate
is duplicated by DRTRIGON-97 More options for sum_cat_disc Closed
Relation
is related to DRTRIGON-78 List all changed talk pages of articles under one category Closed
URL: http://toolserver.org/~drtrigon/cgi-bin/sum_cat_disc.py

Description

Flominator requested to get the functionality 'sum_cat_disc' provides built into a bot. Some suggestions on how to do this and what features should be available are given here:
http://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:DrTrigon&oldid=85471743#Diskussionsbeitr.C3.A4ge_nach_Kategorie

e.g.:

  • Liste wie bei DrTrigonBot wäre super, die Lösung per Vorlage jedoch perfekt
  • Optionen, die man sich vorstellen könnte:
    • Kategorientiefe
    • Dauer bis zum Abräumen
    • Ignorieren von Änderungen durch Bots oder an eingebundenen Vorlagen
    • Ausgabe der Abschnittsüberschrift oder des Ändernden
    • Mindestanzahl an Änderungen (bevor Seite überhaupt angezeigt wird)

Mir würde allerdings auch eine initiale Version helfen, die "einfach" nur die entsprechenden Links auf irgendeiner Seite postet.


Comments
Comment by drtrigon [ Sat, 19 Feb 2011 09:35:18 +0000 ]

Mir würde allerdings auch eine initiale Version helfen, die "einfach" nur die entsprechenden Links auf irgendeiner Seite postet.

What about 'subster'.... ) test to try it was setup in: http://de.wikipedia.org/w/index.php?title=Benutzer:DrTrigon/Spielwiese&oldid=85478950#Test%201
It's more or less ok, with exception of the missing link syntax and all dates are lost... but is close

Comment by flominator [ Sat, 19 Feb 2011 09:57:16 +0000 ]

Looks great, especially since it contains one talk page I edited

Comment by drtrigon [ Sun, 20 Feb 2011 16:14:18 +0000 ]

Added 'wikilinkedlist' as 'postproc' option in r118; it enables to output linked (within the wiki) lists.

This should enable beta testing of the first "sum_cat_disc bot"; Flo if you like you can start using
it like on my sandbox, the timestamps are missing but the list is ordered by timestamp.

Comment by flominator [ Sun, 20 Feb 2011 21:24:08 +0000 ]

Like this? http://de.wikipedia.org/w/index.php?title=Benutzer%3AFlominator%2FFreiburg&action=historysubmit&diff=85550789&oldid=85544094

Comment by flominator [ Mon, 21 Feb 2011 07:13:19 +0000 ]

It currently throws an exception at http://de.wikipedia.org/w/index.php?title=Benutzer:Flominator/Freiburg&oldid=85558651

Comment by drtrigon [ Mon, 21 Feb 2011 15:17:25 +0000 ]

That is because you have chosen a wrong (not matching) regex. Your choice was

regex=<br />\n<table>(.*?)</table>\n<br />\nTime to process

which produced wrong output.

Correct is:

regex=<br>\n<table>(.*?)</table>\n<br>\nTime to process

and produces correct output.

You may want to have a look at following links:

Comment by flominator [ Mon, 28 Feb 2011 07:05:57 +0000 ]

Thanks. Works great. Two more things:

1. What about including the subster-Bot-Template for sum_cat_disc in a separate template, that can be handled like this {{User:DrTrigon/Talk pages|Category 1;Category 2;...}

2. The bot seems to append exceptions instead of replacing them: http://de.wikipedia.org/w/index.php?title=Benutzer%3AFlominator%2FBreisgau-Hochschwarzwald&action=historysubmit&diff=85860993&oldid=85821723

Comment by drtrigon [ Mon, 28 Feb 2011 10:46:22 +0000 ]

Great!

  1. a first draft was created in http://de.wikipedia.org/wiki/Benutzer:DrTrigon/Entwurf/Vorlage:Talk_pages usage, e.g.
    {{Benutzer:DrTrigon/Entwurf/Vorlage:Talk_pages
    |cat=Baden-W%C3%BCrttemberg
    |value=valSumCatDiscTest
    }}
    

    this template work exact like the other but hides the params 'regex' and 'postproc' from you since especially 'regex' causes you some problems. To have all needed options; the new template supports a parameter 'period' as well. You may use one category per template only. If something is left unclear, please ask!

  2. I have to say, the bot works correctly and as it should, the issues your are experiencing are all related to wrong 'regex' specified; you use
    <br />
    

    in several places instead of

    <br>
    

    which causes the bot not to work! Be carful!! Anyway the new template mentioned hides the 'regex' option from you and should help solving this issue.

Comment by flominator [ Sun, 6 Mar 2011 10:09:00 +0000 ]

That is much easier, thank you.

I do know the difference between the br-tags, but some script in my monobook used to automatically replace br by br/. I deactivated that now.

The template is great, except the naming of the parameters. It works for me, but should be understandable for others as well, don't you think? Maybe we should continue the discussion at the talk page of the template?

What about the parameter names "Kategorie" and "Stunden" (maybe Tage)? Does the consumer really need to specify the "value"?

Comment by flominator [ Sat, 19 Mar 2011 14:08:14 +0000 ]

To be able to scan the results far easier, it would be helpful to have the name of the last user who editied the page.

Comment by drtrigon [ Sat, 9 Jul 2011 19:52:03 +0000 ]

The correct usage for the template (in order to work) is according to: http://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:DrTrigon&oldid=91043669#Subster-Bot

{{subst:Benutzer:DrTrigon/Entwurf/Vorlage:Talk_pages
|cat=Baden-W%C3%BCrttemberg
|value=valSumCatDiscTest
}}

DO NOT FORGET THE

subst:

Confer http://de.wikipedia.org/wiki/Benutzer:DrTrigon/Entwurf/Vorlage:Talk_pages also.

Comment by drtrigon [ Sat, 9 Jul 2011 21:54:23 +0000 ]

Example - DrTrigonBot category discussion summary

added to http://de.wikipedia.org/wiki/Benutzer:DrTrigon/Entwurf/Vorlage:Subster

Comment by drtrigon [ Sun, 10 Jul 2011 21:43:40 +0000 ]

Replace (and remove) helper template in Vorlage:Talk_pages with the concept described at Benutzer Diskussion:DrTrigon#Subster-Bot:

Introduce new parameter 'simple' with following syntax (example):

{{Benutzer:DrTrigon/Entwurf/Vorlage:Subster
|simple={'<type>': 'sum_cat_disc', 'cat': 'Freiburg im Breisgau', 'period': '336'}
|value=valSumCatDiscTest
}}

now use e.g. page Benutzer:DrTrigonBot/Subster to store the full parameter set belonging to a 'simple type' (here 'sum_cat_disc') like:

{|
|sum_cat_disc
|url=http://toolserver.org/~drtrigon/cgi-bin/sum_cat_disc.py?wiki=de&cat=%(cat)s&period=%(period)s
|regex=<br>\n<table>(.*?)</table>\n<br>\nTime to process
|postproc=('wikilinkedlist', '"_blank">(.*?)<')
|-
|...
|}
Comment by drtrigon [ Thu, 14 Jul 2011 22:08:15 +0000 ]

http://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:DrTrigon&oldid=91241546#Subster-Bot

Comment by drtrigon [ Mon, 1 Aug 2011 08:18:02 +0000 ]

http://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:Flominator&oldid=91946130#Subster-Bot_.28Erg.C3.A4nzung.29

Comment by drtrigon [ Fri, 12 Aug 2011 14:54:10 +0000 ]

Respect https://wiki.toolserver.org/view/Database_access#Query_time_limits and add limits to 'sum_cat_disc' where needed.

Comment by drtrigon [ Sun, 14 Aug 2011 20:30:13 +0000 ]

Query-Killer is back in action: http://lists.wikimedia.org/pipermail/toolserver-l/2011-August/004305.html

Comment by drtrigon [ Fri, 26 Aug 2011 17:35:39 +0000 ]

Enabled parse of time-stamps through new postproc option 'formatedlist' [1], an example is given in [2]. The format used '* [[%s]] - %s' can be freely chosen.

[1] http://de.wikipedia.org/w/index.php?title=Benutzer:DrTrigon/Benutzer:DrTrigonBot/config.css&diff=92915758&oldid=92859420
[2] http://de.wikipedia.org/w/index.php?title=Benutzer%3ADrTrigon%2FSpielwiese&action=historysubmit&diff=92915813&oldid=92914625

Comment by drtrigon [ Fri, 26 Aug 2011 22:13:48 +0000 ]

New parameter 'simple' introduced in r158. An example of usage can be found at [1] together with [2]. The syntax is as follows:

Instead of using e.g.

{{Benutzer:DrTrigon/Entwurf/Vorlage:Subster
|url=http://toolserver.org/~drtrigon/cgi-bin/sum_cat_disc.py?wiki=de&cat=Freiburg%20im%20Breisgau&period=336
|regex=<br>\n<table>(.*?)</table>\n<br>\nTime to process
|postproc=('wikilinkedlist', '"_blank">(.*?)<')
|value=valSumCatDiscTest
}}

use now

{{Benutzer:DrTrigon/Entwurf/Vorlage:Subster
|simple={{Benutzer:DrTrigon/Entwurf/Vorlage:Subster/Simple:sum_cat_disc|cat=Freiburg im Breisgau|period=336}}
|value=valSumCatDiscTest
}}

in fact all parameters (including 'value') can be hidden or simplified this way. The Template used look like this

{{((}}Benutzer:DrTrigon/Entwurf/Vorlage:Subster/Simple:sum_cat_disc

|url=http://toolserver.org/~drtrigon/cgi-bin/sum_cat_disc.py?wiki=de&cat={{{cat}}}&period={{{period}}}
|regex=<br>\n<table>(.*?)</table>\n<br>\nTime to process
|postproc=('wikilinkedlist', '"_blank">(.*?)<')
{{))}}

and is called Benutzer:DrTrigon/Entwurf/Vorlage:Subster/Simple:sum_cat_disc [2]. This name can obviously also be chosen quite shorter and then used more comfortable.

This is implemented by passing the text given in 'simple' to the expandtemplates mediawiki api call. This should be as fast as reading a page with options stored but could be improved by caching the used templates for eventual re-use during (same) bot run.

[1] http://de.wikipedia.org/w/index.php?title=Benutzer:DrTrigon/Spielwiese&diff=92924020&oldid=92915813
[2] http://de.wikipedia.org/w/index.php?title=Benutzer:DrTrigon/Entwurf/Vorlage:Subster/Simple:sum_cat_disc&oldid=92923712

Comment by drtrigon [ Sat, 27 Aug 2011 18:50:45 +0000 ]

This 'simple' syntax can also be (ab)used to solve the problem of changing URLs (e.g. FIFA), look at [1].

[1] http://de.wikipedia.org/wiki/Benutzer:DrTrigon/Spielwiese#Test_5.2

Comment by drtrigon [ Sat, 27 Aug 2011 19:26:07 +0000 ]

Integrating sum_cat_disc.py as bot by subster was done finally in r158.

Enhancing the options of sum_cat_disc itself was copied/moved to DRTRIGON-97 because its another issue.

Comment by flominator [ Sun, 28 Aug 2011 10:30:25 +0000 ]

Great changes. Cam we move http://de.wikipedia.org/wiki/Benutzer:DrTrigon/Entwurf/Vorlage:Subster/Simple:sum_cat_disc and localize the names of the parameters used by this template of do you need them for parsing?

Comment by drtrigon [ Sun, 28 Aug 2011 11:02:44 +0000 ]

You can do with the template [1] whatever you want (rename/move, use another, ...), you can also change all the parameter freely. The only thing you have to preserve and follow is the data (and format) of the text returned by the template when processed through [2]. The bot simply expands the template as very first action to get further info, that way it supports full wiki syntax in the 'simple' option. You can even use further functions like done in e.g. [3] to have full dynamic parameters...

[1] http://de.wikipedia.org/wiki/Benutzer:DrTrigon/Entwurf/Vorlage:Subster/Simple:sum_cat_disc
[2] http://de.wikipedia.org/wiki/Spezial:Vorlagen_expandieren
[3] http://de.wikipedia.org/wiki/Benutzer:DrTrigon/Entwurf/Vorlage:FIFA-Weltranglistendaten





[DRTRIGON-84] 'botlist' and 'wikipedia' upstream need some bug fixes Created: Sat, 19 Feb 2011 08:37:33 +0000 Updated: Sat, 19 Feb 2011 13:38:30 +0000 Resolved: Sat, 19 Feb 2011 13:36:57 +0000

Status: Resolved
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.0, 1.1,
Fix Version/s: None
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Attachments: Filebotlist.patch      Filefollowup_patch.patch      Filewikipedia.patch     
Issue Links:
Relation
is related to DRTRIGON-38 pywikipedia framework commits (or patches) Closed

Description
  • 'botlist' crashes after mediawiki update of german wiki to 1.17wmf1 a patch was provided (but should be version dependent also)
  • 'wikipedia' has several methods in Page object to retrieve the mediawiki software number (e.g. 'version', 'live_version', ...) but the 'live_version' in the current revision has always to fall back to 'version' and the internally hard coded version string, which is not that useful

both patches were sent upstream to Xqt, for verification and commit.


Comments
Comment by drtrigon [ Sat, 19 Feb 2011 08:40:05 +0000 ]

patches (without version check)

Comment by drtrigon [ Sat, 19 Feb 2011 08:44:54 +0000 ]

A strange or interesting fact is the member function 'headerDone' (from class '_GetAll') is able to retrieve the 'live_version' also. It uses the other way; instead of using 'Special:Version' it processes the tags given in all (newer?) mediawiki page headers, like e.g.:

<meta name="generator" content="MediaWiki 1.17wmf1" />

to get the actual version. May be we should just use one single method in one single place to solve this...?!?

Comment by drtrigon [ Sat, 19 Feb 2011 08:46:54 +0000 ]

(add one item)

Comment by drtrigon [ Sat, 19 Feb 2011 12:53:12 +0000 ]
  • 'wikipedia' done in r8984
  • 'botlist' done in r8985

I have a follow-up for both of them, but it is not important for the bot to work.

Comment by drtrigon [ Sat, 19 Feb 2011 13:36:13 +0000 ]

Follow-up patches proposed to Xqt. Not done yet, but not that important anyway.

Comment by drtrigon [ Sat, 19 Feb 2011 13:36:57 +0000 ]

Followups not done yet, since not that important...

Comment by drtrigon [ Sat, 19 Feb 2011 13:38:30 +0000 ]

http://svn.wikimedia.org/svnroot/pywikipedia/branches/rewrite/pywikibot/site.py

('live_version' in re-write uses generator meta tag already)





[DRTRIGON-83] Pages with User as last editor should be suppressed Created: Mon, 14 Feb 2011 01:21:14 +0000 Updated: Sun, 10 Jul 2011 21:23:29 +0000 Resolved: Sun, 10 Jul 2011 21:20:51 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: 1.0, 1.1,
Fix Version/s: 1.0, 1.1,
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://de.wikipedia.org/w/index.php?title=Benutzer:Flominator/DrTrigonBot&diff=85220128&oldid=85191408

Description

As visible on the link pages with the User (for which the summary is created) as last editor appear on the list, which is somehow useless. This behaviour is for pages without sections only (really without sections or because of recognition problem) because of the suppressed 'lasteditor' check.

A possible solution could be to check the last editor in very early stage already (when checking the timestamp, e.g.) and thus help to reduce the page recognition work, since this won't be needed anymore then.


Comments
Comment by drtrigon [ Sun, 10 Jul 2011 20:54:49 +0000 ]

The demands are somehow inconsistent, since to decide what 'lasteditor' check to use the page has to be processed and thus the page processing cannot be skipped this way.

Comment by drtrigon [ Sun, 10 Jul 2011 21:20:51 +0000 ]

For pages without sections the 'lasteditor' flag gets checked by comparing the page history with the actual user. This way we have exactly the requested behaviour and the relevancy check becomes more consistent since the 'lasteditor' flag is now present in all situations/cases again.

Comment by drtrigon [ Sun, 10 Jul 2011 21:23:29 +0000 ]

Committed revision 124.





[DRTRIGON-82] The section AFTER the signed one is reported Created: Wed, 9 Feb 2011 09:21:37 +0000 Updated: Tue, 30 Aug 2011 16:39:21 +0000 Resolved: Tue, 30 Aug 2011 16:39:21 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to DRTRIGON-68 release new version 0.3 Closed
URL: http://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:DrTrigon/SumDisc&oldid=85051910

Description

On several pages the section after the signed one is reported, without any reason for it... (look at the URL for some examples)


Comments
Comment by drtrigon [ Wed, 9 Feb 2011 13:17:13 +0000 ]

It seams that this issue is related to a wrong 'byteoffset' returned by 'action=parse&prop=sections' and thus by 'getSections'. This was solved easily.

A new issues appeared now (or is visible now), a lot of sections signed by the user and thus not relevant become listed anyway...

Comment by drtrigon [ Wed, 9 Feb 2011 14:49:47 +0000 ]

Bug solved in r97: http://fisheye.toolserver.org/changelog/drtrigon/?cs=97

Additionally:

  • some PEP 8 garbage
  • pysvn enabled, thanks to TS-936
Comment by drtrigon [ Tue, 30 Aug 2011 15:55:33 +0000 ]

fixed version missing





[DRTRIGON-81] Germanization 'disk' instead of 'disc' was used in several places... Created: Tue, 8 Feb 2011 12:44:54 +0000 Updated: Tue, 30 Aug 2011 16:39:01 +0000 Resolved: Tue, 30 Aug 2011 16:39:01 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified


Description

This has to changed!!


Comments
Comment by drtrigon [ Tue, 8 Feb 2011 21:16:14 +0000 ]

This was mainly important due a miss-spelling/-naming of

http://toolserver.org/~drtrigon/cgi-bin/sum_cat_disc.py

which was named ('disk' instead of 'disc') which is really wrong. 'disk' means 'harddisk' (or 'Diskussion' in German) and 'disc' means 'discussion' which is actually already wrong for 'talk' but at least means the right thing.

Comment by drtrigon [ Tue, 8 Feb 2011 21:17:08 +0000 ]

Done in r90: http://fisheye.toolserver.org/changelog/drtrigon/?cs=90

Comment by drtrigon [ Tue, 8 Feb 2011 21:17:33 +0000 ]

(e.g. look also at http://de.wikipedia.org/wiki/Vorlage:Catscan_Portal)

Comment by drtrigon [ Tue, 30 Aug 2011 15:56:31 +0000 ]

fixed version missing





[DRTRIGON-80] Stability of subster.py Created: Mon, 7 Feb 2011 23:43:25 +0000 Updated: Tue, 30 Aug 2011 16:38:40 +0000 Resolved: Tue, 30 Aug 2011 16:38:40 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://de.wikipedia.org/w/index.php?title=Benutzer:DrTrigon/Spielwiese&oldid=85001399#Test_5

Description

It's somehow fragile. A small missconfiguration causes the (sub-)bot to crash. This should be avoided, since it can at least complete the other templates and pages.


Comments
Comment by drtrigon [ Tue, 8 Feb 2011 22:36:08 +0000 ]

Done in r95: http://fisheye.toolserver.org/changelog/drtrigon/?cs=95

Comment by drtrigon [ Tue, 30 Aug 2011 15:57:05 +0000 ]

fixed version missing





[DRTRIGON-79] Timestamp issue in sum_cat_disk.py Created: Mon, 7 Feb 2011 21:05:06 +0000 Updated: Tue, 30 Aug 2011 16:38:19 +0000 Resolved: Tue, 30 Aug 2011 16:38:19 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - toolserver scripts,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://toolserver.org/~drtrigon/cgi-bin/sum_cat_disk.py?wiki=de&cat=Asien&start=20110207200259&period=24

Description

Under some circumstances (for some pages) the SQL queries fail, thus the script is not able to get the Timestamp. To avoid crashed and get an output in this cases also, an cheap bug fix was applied. The problematic pages are marked with a '' now, look at the link.


Comments
Comment by drtrigon [ Fri, 11 Feb 2011 18:20:11 +0000 ]

r102: http://fisheye.toolserver.org/changelog/drtrigon/?cs=102

a new SQL call scheme was introduced and should also solve this problem (beside it's about 10 times faster) - thanks Merlissimo!

Comment by drtrigon [ Tue, 30 Aug 2011 15:57:40 +0000 ]

fixed version missing





[DRTRIGON-78] List all changed talk pages of articles under one category Created: Sun, 6 Feb 2011 11:22:23 +0000 Updated: Sun, 20 Feb 2011 11:48:39 +0000 Resolved: Mon, 7 Feb 2011 14:06:25 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia, DrTrigonBot - toolserver scripts,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: Florian Straub Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to DRTRIGON-85 Integrate sum_cat_disc as bot Closed
URL: http://de.wikipedia.org/wiki/Benutzer_Diskussion:DrTrigon#Diskussionsbeitr.C3.A4ge_nach_Kategorie

Description

For portals and wikiprojects it would be great to see which talk pages of articles around their topic are currently talked about.


Comments
Comment by drtrigon [ Mon, 7 Feb 2011 14:05:17 +0000 ]

Discussion with Merlissimo in #wikipedia-de-bots (IRC) and Flo showed that it would be best to make a small (web) tool for the TS.

Comment by drtrigon [ Mon, 7 Feb 2011 14:06:25 +0000 ]

Simple first version of the tool done in r82: https://fisheye.toolserver.org/changelog/drtrigon/?cs=82

Comment by drtrigon [ Mon, 7 Feb 2011 14:06:44 +0000 ]

http://toolserver.org/~drtrigon/cgi-bin/sum_cat_disk.py

Comment by flominator [ Sun, 20 Feb 2011 11:48:39 +0000 ]

http://toolserver.org/~drtrigon/cgi-bin/sum_cat_disc.py





[DRTRIGON-77] Statistics in panel.py Created: Thu, 3 Feb 2011 21:25:06 +0000 Updated: Tue, 30 Aug 2011 16:34:03 +0000 Resolved: Tue, 30 Aug 2011 16:34:03 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - toolserver scripts,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Depends
depends on DRTRIGON-68 release new version 0.3 Closed
Relation
is related to DRTRIGON-74 Improve (may be rewrite?) panel.py Closed

Description

The statistical outputs of panel.py will stop to work with the new bot, since it does not report warnings anymore (at least this should be the case). The bot raises an exception where needed, but should still finish somehow!

Since this verbose statistics are not needed at the moment (anymore) the link should may be removed from panel.py.


Comments
Comment by drtrigon [ Thu, 3 Feb 2011 21:42:48 +0000 ]

Solved in r74: http://fisheye.toolserver.org/changelog/drtrigon/?cs=74

The statistics are still there; a minimal version of it. Graphic/Plot was removed, but the code is still present in panel.py an can be re-used at any time!

Comment by drtrigon [ Tue, 30 Aug 2011 15:58:15 +0000 ]

fixed version missing





release new version 0.3(DRTRIGON-68)

[DRTRIGON-76] Clean up all codes Created: Wed, 2 Feb 2011 18:29:02 +0000 Updated: Tue, 30 Aug 2011 16:33:42 +0000 Resolved: Tue, 30 Aug 2011 16:33:42 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified


Description
  • do speed checks
  • remove debug print
  • remove debug/notice comments

Comments
Comment by drtrigon [ Thu, 3 Feb 2011 21:06:31 +0000 ]

Compress History was not working.

Comment by drtrigon [ Thu, 3 Feb 2011 21:09:33 +0000 ]

Check /logs directory, it seams like there are files missing (should be ~50 files, NOT ~25)!

Comment by drtrigon [ Sat, 5 Feb 2011 22:55:17 +0000 ]

All done. History files and compression should be ok.

Comment by drtrigon [ Tue, 30 Aug 2011 15:58:45 +0000 ]

fixed version missing





[DRTRIGON-75] Include sitenotice (if still used ...) Created: Mon, 24 Jan 2011 10:51:50 +0000 Updated: Tue, 30 Aug 2011 16:33:21 +0000 Resolved: Tue, 30 Aug 2011 16:33:21 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - toolserver scripts,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Depends
is a dependency for DRTRIGON-74 Improve (may be rewrite?) panel.py Closed
URL: http://lists.wikimedia.org/pipermail/toolserver-l/2011-January/003849.html

Description

During other improvements of e.g. panel.py script, include also the display of the given sitenotice. Just in case of any future server problems, there is additional info available. May be this should be completed with the status page, thus adding:


Comments
Comment by drtrigon [ Wed, 2 Feb 2011 16:57:45 +0000 ]

Done in r70: https://fisheye.toolserver.org/changelog/drtrigon/?cs=70

Comment by drtrigon [ Tue, 30 Aug 2011 15:59:18 +0000 ]

fixed version missing





[DRTRIGON-74] Improve (may be rewrite?) panel.py Created: Mon, 24 Jan 2011 10:42:33 +0000 Updated: Tue, 30 Aug 2011 16:32:57 +0000 Resolved: Tue, 30 Aug 2011 16:32:57 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - toolserver scripts,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Depends
depends on DRTRIGON-75 Include sitenotice (if still used ...) Closed
Relation
is related to DRTRIGON-77 Statistics in panel.py Closed
URL: http://toolserver.org/~drtrigon/cgi-bin/panel.py

Description

Improve (may be by re-writing) panel.py the main bot status info script.

Add more professional tools (and look) by

  • using RRDtool (look at rrdtool_test.py)
  • rewriting the HTML templates (may be use Cheetah)
  • cleaning up the code
  • ...

Comments
Comment by drtrigon [ Wed, 2 Feb 2011 17:27:29 +0000 ]

The tries to use rrdtool through python result in this:

drtrigon@nightshade:~/devel/experimental$ python rrdtool_test.py

(process:26600): Pango-CRITICAL **: No modules found:
No builtin or dynamically loaded modules were found.
PangoFc will not work correctly.
This probably means there was an error in the creation of:
'/etc/opt/ts/pango/pango.modules'
You should create this file by running:
pango-querymodules > '/etc/opt/ts/pango/pango.modules'

(process:26600): Pango-WARNING **: failed to choose a font, expect ugly output. engine-type='PangoRenderFc', script='common'

(process:26600): Pango-WARNING **: failed to choose a font, expect ugly output. engine-type='PangoRenderFc', script='latin'

The other two points were done in r70: https://fisheye.toolserver.org/changelog/drtrigon/?cs=70

Comment by drtrigon [ Thu, 3 Feb 2011 21:10:35 +0000 ]

Still no answer to: http://lists.wikimedia.org/pipermail/toolserver-l/2011-February/003859.html

Comment by drtrigon [ Thu, 3 Feb 2011 21:45:40 +0000 ]

Either matplotlib or rrdtool for nice graphs/plots.

Comment by drtrigon [ Thu, 3 Feb 2011 21:48:15 +0000 ]

Because of DRTRIGON-77 there is no graph/plot present at the moment, thus the rrdtool is not needed at this point in time... may be later - please re-open this ticket in this case!

Comment by drtrigon [ Tue, 30 Aug 2011 16:00:03 +0000 ]

fixed version missing





[DRTRIGON-73] Subster bot edit comment Created: Sat, 22 Jan 2011 10:10:30 +0000 Updated: Tue, 30 Aug 2011 16:32:33 +0000 Resolved: Tue, 30 Aug 2011 16:32:33 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: New Feature Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://de.wikipedia.org/w/index.php?title=Benutzer:DrTrigonBot/Status&action=history

Description

Subster bot should produce useful and pregnant edit comment! Best would be to include a list with changed tag, e.g. instead of

u'Bot: substituting changed tags.'

we could use something like

u'Bot: substituting A, B, C, ... tag(s).'


Comments
Comment by drtrigon [ Wed, 2 Feb 2011 18:12:36 +0000 ]

Done in r71: https://fisheye.toolserver.org/changelog/drtrigon/?cs=71

Comment by drtrigon [ Tue, 30 Aug 2011 16:01:20 +0000 ]

fixed version missing





[DRTRIGON-72] Improve configuration error handling Created: Sun, 16 Jan 2011 15:05:54 +0000 Updated: Tue, 30 Aug 2011 16:31:59 +0000 Resolved: Tue, 30 Aug 2011 16:31:59 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Bug Priority: Minor
Reporter: drtrigon Assignee: drtrigon
Resolution: Not a bug Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Depends
is a dependency for DRTRIGON-68 release new version 0.3 Closed

Description

The case when a user introduces configuration errors like [1] should be handled appropriate. The current bot 0.2.0000 handles them well, but the new one seams to have problems. This HAS TO BE SOLVED BEFORE release of 0.3...! The bot may trigger an error but the other users HAVE TO BE PROCESSED ANYWAY!!!

[1] http://de.wikipedia.org/w/index.php?title=Benutzer:DrTrigonBot/Diene_Mir!&diff=83913189&oldid=83865096
[2] http://toolserver.org/~drtrigon/DrTrigonBot/20110115.log


Comments
Comment by drtrigon [ Wed, 2 Feb 2011 20:07:37 +0000 ]

By looking at http://toolserver.org/~drtrigon/DrTrigonBot/20110115.log it becomes clear that the bot was already working as expected. Did not throw any error at all (this could be a drawback as well), just 2011/01/15 03:36:04:: Unknown Error. API Error code:invalidtitle.

This is ok (more or less).

Comment by drtrigon [ Tue, 30 Aug 2011 16:02:18 +0000 ]

fixed version missing





pywikipedia framework commits (or patches)(DRTRIGON-38)

[DRTRIGON-71] Merge DrTrigonBot clean_user_sandbox with clean_sandbox Created: Sat, 13 Nov 2010 13:49:10 +0000 Updated: Tue, 30 Aug 2011 16:31:31 +0000 Resolved: Tue, 30 Aug 2011 16:31:31 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Attachments: Fileclean_sandbox.patch      Fileclean_sandbox_v2.patch     
URL: https://sourceforge.net/tracker/?func=detail&aid=3108403&group_id=93107&atid=603141

Description

By looking at the source of clean_user_sandbox in [1] it can be seen that most of the code differences are within 'run()' of the bot class. Thus I would like to propose some slight adaptions in clean_sandbox in such a way, that e.g. the get and put operations can be omitted. May be the two codes can be completely merged/joned together.

If this is accepted I am willing to write the needed code.


Comments
Comment by drtrigon [ Wed, 2 Feb 2011 20:41:51 +0000 ]

http://de.wikipedia.org/wiki/Benutzer_Diskussion:Xqt#Pywikibot

Comment by drtrigon [ Thu, 3 Feb 2011 22:50:00 +0000 ]

If adapted and then published upstream it may be possible that 'dtbext.basic.BasicBot' would also be needed, thus this has also to be adapted and sent upstream...

Comment by drtrigon [ Fri, 4 Feb 2011 14:50:57 +0000 ]

http://lists.wikimedia.org/pipermail/pywikipedia-l/2011-February/006660.html

Comment by drtrigon [ Fri, 4 Feb 2011 14:51:16 +0000 ]

wrong click!!!

Comment by drtrigon [ Fri, 4 Feb 2011 15:12:33 +0000 ]

Done in r75: http://fisheye.toolserver.org/changelog/drtrigon/?cs=75

Waiting for upstream commit...

Comment by drtrigon [ Fri, 4 Feb 2011 22:55:11 +0000 ]

In chat with xqt he came up with a few good ideas. Basically the user sandbox mode should be stand-alone.

Comment by drtrigon [ Fri, 4 Feb 2011 22:59:30 +0000 ]

Done in r77: http://fisheye.toolserver.org/changelog/drtrigon/?cs=77

Again waiting for upstream...

Comment by drtrigon [ Mon, 7 Feb 2011 18:59:26 +0000 ]

If this was done, may be add possibility for debug options (switch off writing to wiki). This could be implemented by using 'self.save()' from 'basic.BasicBot' and sub-classing/modifying it.

Comment by drtrigon [ Fri, 11 Feb 2011 10:56:54 +0000 ]

Another idea to implement in future is to use backlinks to given template instead of user-list.

Comment by drtrigon [ Fri, 11 Feb 2011 10:57:18 +0000 ]

Done in r101: http://fisheye.toolserver.org/changelog/drtrigon/?cs=101

Thanks to xqt!

Comment by drtrigon [ Tue, 30 Aug 2011 16:09:47 +0000 ]

fixed version missing





pywikipedia framework commits (or patches)(DRTRIGON-38)

[DRTRIGON-70] KeyError in userlib.User.contributions Created: Thu, 21 Oct 2010 19:41:28 +0000 Updated: Tue, 30 Aug 2011 16:30:59 +0000 Resolved: Tue, 30 Aug 2011 16:30:59 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Attachments: Fileuserlib_commenthidden.patch     
URL: https://sourceforge.net/tracker/?func=detail&aid=3092329&group_id=93107&atid=603138

Description

Summary: "As you can see the 'comment' item is missing, but a 'commenthidden' is present. I wrote a patch to take this into account. I decided to return an u'' instead of contrib['comment'] in such a case, since in this place an unicode string is expected and thus a None or drop this item is not a good idea."


Comments
Comment by drtrigon [ Thu, 21 Oct 2010 19:42:20 +0000 ]

My proposal to solve this.

Comment by drtrigon [ Sat, 13 Nov 2010 13:21:49 +0000 ]

KeyError fixed in r8701 by xqt as recommended by valhallasw.

May be this introduces new issues because of 'None' but only withing 'Page'.

Comment by drtrigon [ Tue, 30 Aug 2011 16:10:59 +0000 ]

fixed version missing





17 (copied from wiki)(DRTRIGON-29)

[DRTRIGON-69] enhance page write options Created: Sat, 2 Oct 2010 16:38:24 +0000 Updated: Tue, 30 Aug 2011 16:30:28 +0000 Resolved: Tue, 30 Aug 2011 16:30:28 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified


Description

append (for page) and write to specific section number in page should be integrated for several bots


Comments
Comment by drtrigon [ Fri, 15 Oct 2010 22:16:45 +0000 ]

moved to dtbext.basic as 'append' and dtbext.wikipedia as 'append'

but these 'append'(s) are not completely worked out (not fully mature) yet, before sending upstream there has to be done quite some stuff - thus/but will not be sent upstream soon anyway...

Comment by drtrigon [ Tue, 30 Aug 2011 16:11:45 +0000 ]

fixed version missing





[DRTRIGON-68] release new version 0.3 Created: Sat, 2 Oct 2010 16:22:21 +0000 Updated: Tue, 30 Aug 2011 16:30:04 +0000 Resolved: Tue, 30 Aug 2011 16:30:04 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Software installation - no user impact Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Depends
depends on DRTRIGON-44 Update to actual pywikipedia revision Closed
depends on DRTRIGON-8 B55 (copied from wiki) Resolved
depends on DRTRIGON-53 Adapt backup script to new config file scheme before updating bot code Closed
depends on DRTRIGON-65 Some problematic cases Closed
depends on DRTRIGON-72 Improve configuration error handling Closed
depends on DRTRIGON-14 F52 (copied from wiki) Closed
depends on DRTRIGON-15 F53 (copied from wiki) Closed
depends on DRTRIGON-17 F55 (copied from wiki) Closed
depends on TS-936 Install 'pysvn' module for python Resolved
is a dependency for DRTRIGON-77 Statistics in panel.py Closed
Relation
is related to DRTRIGON-82 The section AFTER the signed one is reported Closed
Sub-Tasks:
Key
Summary
Type
Status
Assignee
DRTRIGON-76 Clean up all codes Sub-task Closed drtrigon

Description

feature close; bugs, code bereinigen, testen, usw. dann als 0.3.???? veröffentlichen.


Comments
Comment by drtrigon [ Sun, 16 Jan 2011 17:31:32 +0000 ]

Or version 1.0.???? since it should be stable now because of improved bug handling (e.g. DRTRIGON-72)...

Comment by drtrigon [ Fri, 4 Feb 2011 12:16:21 +0000 ]

'backlinks_list' HAS TO BE PUBLISHED TO Benutzer:DrTrigonBot ON BOT RELEASE !!!

Comment by drtrigon [ Fri, 4 Feb 2011 17:53:04 +0000 ]
  • some issues with the old histories; 'notify tag error' had to be introduced
  • on the other hand, after compression any 'old history' HAVE TO vanish, or something is bad
Comment by drtrigon [ Sat, 5 Feb 2011 23:58:58 +0000 ]

By looking at my last results from the actual old bot, it seams that DRTRIGON-32 could be really needed... May be the possible solution has to be checked and used.

By the way, this is the moment for a FEATURE CLOSE.

Comment by drtrigon [ Sun, 6 Feb 2011 10:59:12 +0000 ]

May be 'checksign_list' has also to be published; depends on its usage.

Comment by drtrigon [ Sun, 6 Feb 2011 15:57:58 +0000 ]

Test bot mails in error case - just to be sure.

Think about new log file scheme, including the job output files.

Comment by drtrigon [ Sun, 6 Feb 2011 16:01:35 +0000 ]

Create and publish Doxygen documentation.

Comment by drtrigon [ Mon, 7 Feb 2011 18:01:28 +0000 ]

http://de.wikipedia.org/wiki/Benutzer:DrTrigonBot has to be updated accordingly.

Comment by drtrigon [ Mon, 7 Feb 2011 18:35:43 +0000 ]

ATTENTION: all debug flag and switches have to be removed!

Comment by drtrigon [ Tue, 8 Feb 2011 15:42:09 +0000 ]

switch debug mode of and go to writing (operational) mode, with new style only changes to bot_control.py are needed

Comment by drtrigon [ Tue, 8 Feb 2011 15:44:58 +0000 ]

New history data file path needed due to SVN issues with non UTF-8 filenames.

Because we want to be able to use svn update in order to get new versions of bot code to the TS (working copy) those files had to be moved away from SVN controlled paths. Is good becuase it results in a better file-tree scheme.

Comment by drtrigon [ Tue, 8 Feb 2011 15:45:55 +0000 ]

Error handling and output (e.g. mailing) checked and adpated, especially bot_control.py and panel.py have to be adapted to each other.

Comment by drtrigon [ Tue, 8 Feb 2011 15:51:53 +0000 ]

chmod 744 bot_control.py

and

svn propset svn:executable ON bot_control.py

needed

Comment by drtrigon [ Tue, 8 Feb 2011 16:07:18 +0000 ]

https://wiki.toolserver.org/view/Job_scheduling#Receiving_mail_when_the_job_starts_or_finishes

what about mail just in case the job crashes??

http://lists.wikimedia.org/pipermail/toolserver-l/2011-February/003901.html

Comment by drtrigon [ Tue, 8 Feb 2011 16:13:00 +0000 ]

Change cron(ie)-tab on submit.toolserver.org with

cronie -e (to list use: cronie -l)

to this:

0 2 * * * cronsub -s mainbot $HOME/pywikipedia/bot_control.py -default -cron
0 0 */14 * * cronsub -s compbot $HOME/pywikipedia/bot_control.py -compress_history:[] -cron
#0 6 * * * cronsub -s substerbot $HOME/pywikipedia/subster_beta.py 2>> $HOME/public_html/DrTrigonBot/subster.html

Comment by drtrigon [ Tue, 8 Feb 2011 17:43:29 +0000 ]

#$ -m as

as one of the first lines in script together with sys.exit(100) can be used to have SGE send mail in case of script errors or else - thanks Merlissimo!!

Comment by drtrigon [ Tue, 8 Feb 2011 22:42:48 +0000 ]

According to http://de.wikipedia.org/w/index.php?title=Benutzer:DrTrigonBot&oldid=85040463

Aktuelle Version: 1.0.95
pywikipedia: 8944

Released Version 1.0 with r95: http://fisheye.toolserver.org/changelog/drtrigon/?cs=95

(may be after the run tomorrow, some minor bug fixes will be needed)

Comment by drtrigon [ Tue, 30 Aug 2011 16:12:24 +0000 ]

fixed version missing





19 (copied from wiki)(DRTRIGON-30)

[DRTRIGON-67] Run on all (or several) wikis at once (like interwiki bot) Created: Sat, 2 Oct 2010 15:55:34 +0000 Updated: Sat, 28 Jan 2012 20:33:54 +0000 Resolved: Sat, 28 Jan 2012 20:33:54 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: 1.2,
Security Level: Public (all users)

Type: Sub-task Priority: Minor
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Relation
is related to DRTRIGON-106 Translations (TW) for subster.py Closed

Description

Ev dann den (sum_disc) bot wie interwiki bot auf allen wiki gleichzeitig laufen lassen z.B. mit [1]

(look also at id B45)

[1] http://toolserver.org/~luxo/contributions/contributions.php?user=DrTrigon&blocks=true&lang=


Comments
Comment by drtrigon [ Sat, 24 Dec 2011 08:19:48 +0000 ]

First the tool mentioned was written by Luxo which 'left' the dewiki more than a year ago:

Ich mache nur noch Minimalunterhalt an meinen Toolserver-Tools,
Störungen bitte per Mail melden.--LuxoTS - ja? 20:43, 20. Aug. 2010 (CEST)

so I think it would not be a good idea to use that tool and parse the whole HTML text to
get those info. May be I would make more sence just to run the bot in other languages but
still report everything to your home wiki or home language.

The point is in order to switch the language we have also to adopt things like:

  • checkedit_list
  • ignorepage_list
  • ignorehead_list
  • ...
Comment by drtrigon [ Sat, 28 Jan 2012 20:33:54 +0000 ]

For 'sum_disc.py' somehow interwiki functionality is already implemented through 'http://toolserver.org/~merl/UserPages/query.php' parameter 'globwikinotify_switch'.

For 'subster.py' (and 'subster_irc.py') this is already done since it runs on 'frr' and 'en' wikis also.





pywikipedia framework commits (or patches)(DRTRIGON-38)

[DRTRIGON-66] 'wikipedia' -> 'pywikibot' in watchlist.py Created: Sat, 2 Oct 2010 11:35:13 +0000 Updated: Tue, 30 Aug 2011 16:29:24 +0000 Resolved: Tue, 30 Aug 2011 16:29:24 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Sub-task Priority: Minor
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Attachments: Filebotlist_global.patch      Filewatchlist.patch     
URL: http://lists.wikimedia.org/pipermail/pywikipedia-l/2010-October/006450.html

Description

Adjust watchlist.py according to new framework style.

Add botlist.py analog to watchlist.py for bot user caching.


Comments
Comment by drtrigon [ Sat, 9 Oct 2010 23:26:27 +0000 ]

Done in r8632. Thanks xqt!

Comment by drtrigon [ Sat, 9 Oct 2010 23:44:03 +0000 ]

...only first part was done.

Comment by drtrigon [ Sun, 10 Oct 2010 14:56:18 +0000 ]

Issued on: https://sourceforge.net/tracker/?func=detail&aid=3084735&group_id=93107&atid=603141

Comment by drtrigon [ Thu, 14 Oct 2010 21:08:12 +0000 ]

support for global bot introduced, patch has to be made and sent upstream

look at rev. 46

Comment by drtrigon [ Fri, 15 Oct 2010 16:59:36 +0000 ]

Patch sent upstream.

Comment by drtrigon [ Thu, 21 Oct 2010 19:39:02 +0000 ]

Finally done in r8682 - thanks xqt!

And for DrTrigonBot in r55: https://fisheye.toolserver.org/changelog/drtrigon/?cs=55

Comment by drtrigon [ Tue, 30 Aug 2011 16:13:06 +0000 ]

fixed version missing





[DRTRIGON-65] Some problematic cases Created: Mon, 20 Sep 2010 20:02:12 +0000 Updated: Tue, 30 Aug 2011 16:28:57 +0000 Resolved: Tue, 30 Aug 2011 16:28:57 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Bug Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Issue Links:
Depends
depends on DRTRIGON-32 30 (copied from wiki) Closed
is a dependency for DRTRIGON-68 release new version 0.3 Closed
URL: http://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:DrTrigon/SumDisc&oldid=79347874

Description

This are two pages that should not have been reported at all.

  • New Discussion: [[Wikipedia:Löschkandidaten/25. August 2008]] at [[Wikipedia:Löschkandidaten/25. August 2008#Kategorie:Benutzer:DrTrigon/Kategorie:Vorlagen (erl.)|Kategorie:Benutzer:DrTrigon/Kategorie:Vorlagen (erl.)]] - last edit by [[User:Eynre]] (12:09, 08. Sep. 2008)<noinclude>
    Summary generated from and at: [[Benutzer:DrTrigonBot|DrTrigonBot]] 22:44, 3. Sep. 2009 (CEST)</noinclude>

gets reported because of this is an archive where my signature is present, but not as last entry!

  • New Discussion: [[Benutzer Diskussion:1xSNAFU]] at [[Benutzer Diskussion:1xSNAFU#Deine Monobook|Deine Monobook]] - last edit by [[User:DrTrigonBot]] (13:15, 20. Sep. 2009)<noinclude>
    Summary generated from and at: [[Benutzer:DrTrigonBot|DrTrigonBot]] 04:27, 21. Sep. 2009 (CEST)</noinclude>

gets reported because the page is output page for the bot for '1xSNAFU' and not maintained at all (it seams), thus once the bot reported an entry with my signature.

But after all those two pages should only be reported once, then be put into history and stay there (for ever), preventing the pages to re-appear again...


Comments
Comment by drtrigon [ Sat, 2 Oct 2010 17:12:56 +0000 ]

The second one can be re-reported when 1xSNAFU get news, since then text is added after my 'signature'...

Either improve signature recognition (which will be hard to do) or use DRTRIGON-32.

Comment by drtrigon [ Thu, 3 Feb 2011 22:03:44 +0000 ]

The first issue should be solved by the parameter 'ignorehead_list' introduced in r20: https://fisheye.toolserver.org/changelog/drtrigon/?cs=20

The second issue is not really an issue but a dead account or something like this... If this still persists this ticket should be re-opened. A possible solution could be to trigger on 'BOT MESSAGE' and ignore them OR omit the 'last edit by User:DrTrigon' part in 'BOT MESSAGE's!

Comment by drtrigon [ Tue, 30 Aug 2011 16:13:39 +0000 ]

fixed version missing





[DRTRIGON-64] Wrong byteoffset given by action=parse Created: Mon, 20 Sep 2010 19:45:17 +0000 Updated: Tue, 30 Aug 2011 16:28:19 +0000 Resolved: Tue, 30 Aug 2011 16:28:19 +0000

Status: Resolved
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Support request Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: https://bugzilla.wikimedia.org/show_bug.cgi?id=25203

Description

The mediawiki API for wikipedia:de gives for some pages wrong byteoffset data. This was discussed on [1].

[1] http://de.wikipedia.org/w/index.php?title=Benutzer_Diskussion:Umherirrender&oldid=79265890#MediaWiki_API_und_byteoffset


Comments
Comment by drtrigon [ Thu, 30 Sep 2010 07:16:03 +0000 ]

Another issue with action=parse is the fact that sometimes it does return empty 'item' fields. At the moment this is taken as a sign for parsing problems in the API itself and those sections are merged with the previous one. Without 'item' field the re-try of receiving section data can not be performed (since the number is missing), too.

Comment by drtrigon [ Sat, 2 Oct 2010 17:04:24 +0000 ]

The actual sum_disc and dtbext framework code is able to handle such problematic requests. In the rare cases where not valid section data can be retrieved it either processes multiple headers as one or the whole page as one.

This has the drawback that if a user participates in a discussion which is followed by another one, the user's contributions may be marked as irrelevant and not reported by the bot.

This problem has to be solved in mediawiki software and will cause the bot operate correctly if solved. There is no need to further adapt the bot code, if no major changes are made.

Comment by drtrigon [ Tue, 30 Aug 2011 16:14:15 +0000 ]

fixed version missing





pywikipedia framework commits (or patches)(DRTRIGON-38)

[DRTRIGON-63] Bug fix for userlib.User.__str__ Created: Sun, 19 Sep 2010 19:30:04 +0000 Updated: Tue, 30 Aug 2011 16:27:51 +0000 Resolved: Tue, 30 Aug 2011 16:27:51 +0000

Status: Resolved
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Attachments: Filepatch.diff_userlib     
URL: http://lists.wikimedia.org/pipermail/pywikipedia-l/2010-September/006426.html

Description

A print of an userlib.User() object with special chars (like 'the Bench' is using) throws a 'UnicodeEncodeError'. The given patch can solve this for me.


Comments
Comment by drtrigon [ Wed, 22 Sep 2010 18:32:02 +0000 ]

Fixed by xqt in r8588 (thanks!).

Look also at http://lists.wikimedia.org/pipermail/pywikipedia-svn/2010-September/001968.html

Comment by drtrigon [ Tue, 30 Aug 2011 16:14:47 +0000 ]

fixed version missing





pywikipedia framework commits (or patches)(DRTRIGON-38)

[DRTRIGON-62] Feature request for pagegenerators.RegexFilterPageGenerator Created: Sun, 19 Sep 2010 19:28:05 +0000 Updated: Tue, 30 Aug 2011 16:27:21 +0000 Resolved: Tue, 30 Aug 2011 16:27:21 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Attachments: Filepatch.diff_regex      Filepatch.diff_regex2     
URL: http://lists.wikimedia.org/pipermail/pywikipedia-l/2010-September/006424.html

Description

Enhance 'pagegenerators.RegexFilterPageGenerator' in a way that it can also EXCLUDE pages according to given regexS (plural).


Comments
Comment by drtrigon [ Fri, 8 Oct 2010 06:29:59 +0000 ]

Done in r8609. Thanks xqt!

https://fisheye.toolserver.org/changelog/pywikipedia/?cs=8609

Comment by drtrigon [ Fri, 8 Oct 2010 08:31:06 +0000 ]

http://lists.wikimedia.org/pipermail/pywikipedia-l/2010-October/006462.html

Comment by drtrigon [ Sun, 10 Oct 2010 14:45:47 +0000 ]

Issued on: https://sourceforge.net/tracker/?func=detail&aid=3084727&group_id=93107&atid=603141

Comment by drtrigon [ Thu, 14 Oct 2010 21:04:13 +0000 ]

accepted in r8652

Comment by drtrigon [ Tue, 30 Aug 2011 16:15:19 +0000 ]

fixed version missing





pywikipedia framework commits (or patches)(DRTRIGON-38)

[DRTRIGON-61] Question about getVersionHistory Created: Sun, 19 Sep 2010 19:24:31 +0000 Updated: Tue, 30 Aug 2011 16:26:41 +0000 Resolved: Tue, 30 Aug 2011 16:26:41 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://lists.wikimedia.org/pipermail/pywikipedia-l/2010-September/006415.html

Description

Feature request for 'getVersionHistory' to throw the needed page exceptions.


Comments
Comment by drtrigon [ Sun, 19 Sep 2010 19:25:13 +0000 ]

Now it does (r8571)

Comment by drtrigon [ Sun, 19 Sep 2010 19:26:09 +0000 ]

http://lists.wikimedia.org/pipermail/pywikipedia-l/2010-September/006425.html

Comment by drtrigon [ Mon, 20 Sep 2010 18:43:55 +0000 ]

(switch close <-> resolve management)

Comment by drtrigon [ Tue, 30 Aug 2011 16:15:54 +0000 ]

fixed version missing





pywikipedia framework commits (or patches)(DRTRIGON-38)

[DRTRIGON-60] Question about missing family Created: Sun, 19 Sep 2010 19:22:43 +0000 Updated: Tue, 30 Aug 2011 17:21:56 +0000 Resolved: Tue, 30 Aug 2011 17:21:56 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: 1.1,
Security Level: Public (all users)

Type: Sub-task Priority: Minor
Reporter: drtrigon Assignee: drtrigon
Resolution: Not a bug Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

Attachments: Filewiki_family.py     
URL: http://lists.wikimedia.org/pipermail/pywikipedia-l/2010-September/006414.html

Description

Problem with family 'wiki' either a missing family file or a bug...?!?


Comments
Comment by drtrigon [ Sun, 10 Oct 2010 14:36:14 +0000 ]

Issued on: https://sourceforge.net/tracker/?func=detail&aid=3084721&group_id=93107&atid=603138

Comment by drtrigon [ Sun, 10 Oct 2010 16:11:48 +0000 ]

#wikipedia-de-bots
(18:03:01) xqt: DrTrigon: Logisch ist das nicht. Du nutzt einen Seiteneffekt aus. Du kannst ja eine eigene Klasse myPage(wp.Page) anlegen und dort für den Datenbank-Link einen Container anlegen. Ok. etwas mit Raketen auf Mikroben geschossen
(18:04:40) DrTrigon: xqt: hab ich mir auch so gedacht... aber bitte den seiteneffekt beibehalten, ich finde den sehr nützlich. und auch sehr pythonic: "There should be one-- and preferably only one --obvious way to do it." )
(18:07:39) xqt: DrTrigon: Sicher kannst Du da nicht sein. Du kannst auch so im Seitenobjekt noch was anhängen. wird ja übers framework auch so gemacht.
(18:09:17) xqt: so á la: if not hasattr(self, '_dbname'): _dbname='wiki'
(18:10:23) xqt: * natürlich innerhalb einer Page-Methode

Comment by drtrigon [ Sun, 10 Oct 2010 18:09:02 +0000 ]

Interwiki links can be retrieved by API, but at the moment it does not work very well.

https://bugzilla.wikimedia.org/show_bug.cgi?id=20654
https://bugzilla.wikimedia.org/show_bug.cgi?id=19838

Comment by drtrigon [ Fri, 15 Oct 2010 17:40:09 +0000 ]

Feature request issued as follow up to the closed (invalid) bug report.

https://sourceforge.net/tracker/?func=detail&aid=3088230&group_id=93107&atid=603141

(look also at the attached file)

Comment by drtrigon [ Fri, 15 Oct 2010 17:40:45 +0000 ]

wiki family proposal

Comment by drtrigon [ Tue, 30 Aug 2011 17:11:11 +0000 ]

Essentially 'wiki' in the toolserver DB (like e.g. 'dewiki, 'enwiki') does NOT refer to the same as 'wiki' in pywikipedia and interwiki (where it refers to the first WikiWikiWeb).

For more info look at: http://lists.wikimedia.org/pipermail/pywikipedia-l/2011-August/006944.html

So in pywikipedia replace "wiki:" by:

  • ""
  • "w:"
  • or may be "wikipedia:"

to get working/correct interwiki links.

Comment by drtrigon [ Tue, 30 Aug 2011 17:21:56 +0000 ]

This behaviour is correct since 'wiki' does not refer to the same depending on context pywikipedia framwork/interwiki links or toolserver DB.

Thus the solution chosen in code is not a work-a-round or hack but a correct solution.





pywikipedia framework commits (or patches)(DRTRIGON-38)

[DRTRIGON-59] Console Message for user.contributions() ? Created: Sun, 19 Sep 2010 19:17:20 +0000 Updated: Tue, 30 Aug 2011 16:16:49 +0000 Resolved: Tue, 30 Aug 2011 16:16:49 +0000

Status: Resolved
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: 1.1,
Security Level: Public (all users)

Type: Sub-task Priority: Minor
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://lists.wikimedia.org/pipermail/pywikipedia-l/2010-September/006410.html

Description

There should be a message in analogy to 'Page.getReferences' and all other methods that access the web (e.g. for API access).


Comments
Comment by drtrigon [ Sun, 10 Oct 2010 14:27:41 +0000 ]

Issued on: https://sourceforge.net/tracker/?func=detail&aid=3084717&group_id=93107&atid=603141

Comment by drtrigon [ Sun, 10 Jul 2011 21:31:15 +0000 ]

Did a 'ping?' and 'Assigned' it to xqt. May be this helps because it seams not to get any attention. Anyway this is no critical (or important) issue - more a cosmetic thing, thus this ticket is finished now. A work-a-round is present in the bot code.

"Re-Open" this ticket, if this becomes important or actual again.
"Close" this ticket, if this gets done and solved properly.

Comment by drtrigon [ Tue, 30 Aug 2011 16:16:38 +0000 ]

fixed version missing





pywikipedia framework commits (or patches)(DRTRIGON-38)

[DRTRIGON-58] Detail: get_redirect useless in textlib.py? Created: Sun, 19 Sep 2010 19:11:01 +0000 Updated: Tue, 30 Aug 2011 16:25:06 +0000 Resolved: Tue, 30 Aug 2011 16:25:06 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://lists.wikimedia.org/pipermail/pywikipedia-l/2010-September/006407.html

Description

Just a detail, there was an unused parameter/variable present.


Comments
Comment by drtrigon [ Sun, 19 Sep 2010 19:11:19 +0000 ]

Removed in r8554.

Comment by drtrigon [ Mon, 20 Sep 2010 18:43:52 +0000 ]

(switch close <-> resolve management)

Comment by drtrigon [ Tue, 30 Aug 2011 16:17:12 +0000 ]

fixed version missing





pywikipedia framework commits (or patches)(DRTRIGON-38)

[DRTRIGON-57] Question about 'userlib.py' Created: Sun, 19 Sep 2010 18:58:13 +0000 Updated: Tue, 30 Aug 2011 16:24:27 +0000 Resolved: Tue, 30 Aug 2011 16:24:27 +0000

Status: Closed
Project: [migrated to bugzilla, do not use] DrTrigon's tools
Component/s: DrTrigonBot - pywikipedia,
Affects Version/s: None
Fix Version/s: None
Security Level: Public (all users)

Type: Sub-task Priority: Major
Reporter: drtrigon Assignee: drtrigon
Resolution: Fixed Votes (Watches): 0 (0)
Labels: None
T Remaining Estimate: Not Specified Remaining Estimate: Not Specified
T Time Spent: Not Specified Time Spent: Not Specified
T Original Estimate: Not Specified Original Estimate: Not Specified

URL: http://lists.wikimedia.org/pipermail/pywikipedia-l/2010-September/006397.html

<
Description