Pywikibot: Difference between revisions

Revision as of 12:42, 30 January 2018

Setting this up is confusing as hell, mainly because the documentation is lacking.

Getting, Configuring, Installing

I have the pywikibot software set up with two remotes: one official (Wikimedia gerrit), and one unofficial (my own git repo).

Link to pywikibot on Wikimedia Foundation's gerrit: https://gerrit.wikimedia.org/r/pywikibot/core.git

Link to pywikibot on git.charlesreid1.com: https://charlesreid1.com:3000/wiki/pywikibot

Wikimedia gerrit

Note the official pywikibot repo is also cloned on Github: https://github.com/wikimedia/pywikibot-core/

Start by checking it out:

$ git clone https://gerrit.wikimedia.org/r/pywikibot/core.git pywikibot
$ cd pywikibot

Install all the pip stuff that you may need:

$ pip install -r requirements.txt

Update git submodules:

$ git submodule update --init

Add a custom family file to the big directory of family files:

$ ls pywikibot/families
...
wikivoyage_family.py
wiktionary_family.py
wowwiki_family.py

This is where you will put your custom family file. Here's what the custom family file looks like:

from pywikibot import family

class Family(family.Family):
    def __init__(self):
        family.Family.__init__(self)
        self.name = 'charlesreid1'
        self.langs = {
            'en': 'charlesreid1.com',
        }

Copy and paste this into pwb/pywikibot/families/charlesreid1_family.py (where pwb is the name of the directory where you checked out the git repository).

Now you should be able to log into the wiki as your bot:

$ python pwb.py login
Password for user Bleep bloop on charlesreid1:en (no characters will be shown):

Logging in to charlesreid1:en as Bleep bloop
WARNING: /Users/charles/codes/pywikibot/pywikibot/tools/__init__.py:1717: UserWarning: File /Users/charles/codes/pywikibot/pywikibot.lwp had 644 mode; converted to 600 mode.
Logged in on charlesreid1:en as Bleep bloop.

git.charlesreid1.com

Link to pywikibot on git.charlesreid1.com: https://charlesreid1.com:3000/wiki/pywikibot

To push changes to the pywikibot on git.charlesreid1.com I set up the repo with another remote:

$ git remote add cmr https://charlesreid1.com:3000/wiki/pywikibot
$ git push cmr master

Running Simple Scripts

There are two ways to use pywikibot:

Write your own custom actions
Use a bundle of scripts that come packaged with pywikibot

Prewritten Scripts

Here is a list of all the pre-written scripts for MediaWiki wikis: https://www.mediawiki.org/wiki/Special:MyLanguage/Manual:Pywikibot/Scripts

These are also located in the scripts/ folder of the repository.

To run a given script, you actually run it THROUGH the pwb.py script. See example below.

Redirect.py Script

Suppose we wanted to run the script redirect.py to programmatically deal with redirects on our wiki. We can start by looking at the documentation for this file, which shows us there are many options for this script:

Script to resolve double redirects, and to delete broken redirects.

Requires access to MediaWiki's maintenance pages or to a XML dump file.
Delete function requires adminship.

Syntax:

    python pwb.py redirect action [-arguments ...]

where action can be one of these:

double         Fix redirects which point to other redirects.
do             Shortcut action command is "do".

broken         Tries to fix redirect which point to nowhere by using the last
br             moved target of the destination page. If this fails and the
               -delete option is set, it either deletes the page or marks it
               for deletion depending on whether the account has admin rights.
               It will mark the redirect not for deletion if there is no speedy
               deletion template available. Shortcut action command is "br".

both           Both of the above. Retrieves redirect pages from live wiki,
               not from a special page.

and arguments can be:

-xml           Retrieve information from a local XML dump
               (https://download.wikimedia.org). Argument can also be given as
               "-xml:filename.xml". Cannot be used with -fullscan or -moves.

-fullscan      Retrieve redirect pages from live wiki, not from a special page
               Cannot be used with -xml.

-moves         Use the page move log to find double-redirect candidates. Only
               works with action "double", does not work with -xml.

               NOTE: You may use only one of these options above.
               If neither of -xml -fullscan -moves is given, info will be
               loaded from a special page of the live wiki.

-page:title    Work on a single page

-namespace:n   Namespace to process. Can be given multiple times, for several
               namespaces. If omitted, only the main (article) namespace is
               treated.

-offset:n      With -moves, the number of hours ago to start scanning moved
               pages. With -xml, the number of the redirect to restart with
               (see progress). Otherwise, ignored.

-start:title   The starting page title in each namespace. Page need not exist.

-until:title   The possible last page title in each namespace. Page needs not
               exist.

-total:n       The maximum count of redirects to work upon. If omitted, there
               is no limit.

-delete        Prompt the user whether broken redirects should be deleted (or
               marked for deletion if the account has no admin rights) instead
               of just skipping them.

-sdtemplate:x  Add the speedy deletion template string including brackets.
               This enables overriding the default template via i18n or
               to enable speedy deletion for projects other than wikipedias.

-always        Don't prompt you for each replacement.

Suppose we want to eliminate double-redirects. To do this, we run the redirect script through pwb.py, and pass it the double argument like so:

$ python pwb.py redirect double

Custom Scripts

Site and Page Objects

We'll start out by creating Site and Page objects to obtain pages from the wiki.

Start ipython:

$ ipython

import the pywikibot package:

In [1]: import pwb

if you want to import a custom script in the scripts folder, import it like this (supposing it is in scripts/beavo.py):

In [4]: import scripts.beavo

In [6]: scripts.beavo.test_match_any()

now create a Site object to represent the wiki you're dealing with (you have to have run the login script, to store the credentials to view/edit the wiki):

In [12]: pwb.pywikibot.Site()
Out[12]: APISite("en", "charlesreid1")

next, we can create a Page object to represent pages on the wiki:

In [15]: flag1 = pwb.pywikibot.Page(site,'Template:AircrackFlag')

this Page object has a very long list of available methods:

In [27]: dir(flag1)
Out[27]:
['__class__',
 '__delattr__',
 '__dict__',
 '__dir__',
 '__doc__',
 '__eq__',
 '__format__',
 '__ge__',
 '__getattribute__',
 '__gt__',
 '__hash__',
 '__init__',
 '__init_subclass__',
 '__le__',
 '__lt__',
 '__module__',
 '__ne__',
 '__new__',
 '__reduce__',
 '__reduce_ex__',
 '__repr__',
 '__setattr__',
 '__sizeof__',
 '__str__',
 '__subclasshook__',
 '__unicode__',
 '__weakref__',
 '_cache_attrs',
 '_cmpkey',
 '_cosmetic_changes_hook',
 '_getInternals',
 '_get_parsed_page',
 '_latest_cached_revision',
 '_link',
 '_namespace_obj',
 '_revisions',
 '_save',
 'applicable_protections',
 'aslink',
 'autoFormat',
 'backlinks',
 'botMayEdit',
 'canBeEdited',
 'categories',
 'change_category',
 'clear_cache',
 'content_model',
 'contributingUsers',
 'contributors',
 'coordinates',
 'data_item',
 'data_repository',
 'defaultsort',
 'delete',
 'depth',
 'editTime',
 'embeddedin',
 'encoding',
 'exists',
 'expand_text',
 'extlinks',
 'fullVersionHistory',
 'full_url',
 'get',
 'getCategoryRedirectTarget',
 'getCreator',
 'getDeletedRevision',
 'getLatestEditors',
 'getMovedTarget',
 'getOldVersion',
 'getRedirectTarget',
 'getReferences',
 'getRestrictions',
 'getTemplates',
 'getVersionHistory',
 'getVersionHistoryTable',
 'image_repository',
 'imagelinks',
 'interwiki',
 'isAutoTitle',
 'isCategory',
 'isCategoryRedirect',
 'isDisambig',
 'isEmpty',
 'isFlowPage',
 'isImage',
 'isIpEdit',
 'isRedirectPage',
 'isStaticRedirect',
 'isTalkPage',
 'is_categorypage',
 'is_filepage',
 'is_flow_page',
 'iterlanglinks',
 'itertemplates',
 'langlinks',
 'lastNonBotUser',
 'latestRevision',
 'latest_revision',
 'latest_revision_id',
 'linkedPages',
 'loadDeletedRevisions',
 'markDeletedRevision',
 'merge_history',
 'move',
 'moved_target',
 'namespace',
 'oldest_revision',
 'pageAPInfo',
 'page_image',
 'pageid',
 'permalink',
 'preloadText',
 'previousRevision',
 'previous_revision_id',
 'properties',
 'protect',
 'protection',
 'purge',
 'put',
 'put_async',
 'raw_extracted_templates',
 'removeImage',
 'replaceImage',
 'revision_count',
 'revisions',
 'save',
 'section',
 'sectionFreeTitle',
 'set_redirect_target',
 'site',
 'templates',
 'templatesWithParams',
 'text',
 'title',
 'titleForFilename',
 'titleWithoutNamespace',
 'toggleTalkPage',
 'touch',
 'undelete',
 'urlname',
 'userName',
 'version',
 'watch']

Getting a Page's Version History

In [29]: hist = flag1.fullVersionHistory()
WARNING: /Users/charles/Library/Python/3.6/bin/ipython:1: DeprecationWarning: pywikibot.page.BasePage.fullVersionHistory is deprecated; use Page.revisions(content=True) instead.
  #!/usr/local/opt/python3/bin/python3.6


In [30]: print(type(hist))
<class 'list'>

In [31]: print(type(hist[0]))
<class 'pywikibot.page.FullHistEntry'>

In [32]: item = hist[0]

In [33]: print(dir(item))
['__add__', '__class__', '__contains__', '__delattr__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__getitem__', '__getnewargs__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__iter__', '__le__', '__len__', '__lt__', '__module__', '__mul__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__rmul__', '__setattr__', '__sizeof__', '__slots__', '__str__', '__subclasshook__', '_asdict', '_fields', '_make', '_replace', '_source', 'count', 'index', 'revid', 'rollbacktoken', 'text', 'timestamp', 'user']

In [34]: item.timestamp
Out[34]: Timestamp(2017, 4, 15, 21, 13, 50)

In [36]: item._asdict()
Out[36]:
OrderedDict([('revid', 16270),
             ('timestamp', Timestamp(2017, 4, 15, 21, 13, 50)),
             ('user', 'Admin'),
             ('text',
              "<br />\n\n{{Flag\n|header=aircrack-ng\n|text=a suite of tools for wireless cracking. \n\n\n[[Aircrack]] {{,}} [[:Category:Aircrack]]\n\n\n\n'''aircrack-ng'''\n\nMany Ways to Crack a Wifi: [[Cracking Wifi]]\n\nAircrack Benchmarking: [[Aircrack/Benchmarking]]\n\nWEP Attacks with Aircrack: [[Aircrack/WEP Cracking]]\n\nWPA Attacks with Aircrack: [[Aircrack/WPA Cracking]]\n\nAircrack Hardware: [[Aircrack/Packet Injection Testing]]\n\n[[Harvesting Wireless Network Information]]\n\n\n\n'''airodump-ng'''\n\nBasic Usage of [[Airodump]]\n\n\n\n[[:Category:Security]] {{,}} [[:Category:Wireless]] {{,}} [[:Category:Passwords]]\n\n\n<small>[[Flags]]</small> {{,}} <small>[[Template:AircrackFlag]]</small> {{,}} <small>[http://charlesreid1.com/w/index.php?title=Template:AircrackFlag&action=edit e]</small>\n}}\n\n[[Category:Aircrack]]\n[[Category:Security]]\n[[Category:Wireless]]\n[[Category:Networking]]\n\n<br />"),
             ('rollbacktoken', None)])

In [37]: item.timestamp.isoformat()
Out[37]: '2017-04-15T21:13:50Z'

Getting All Pages on a Wiki

The allpages() method will return a generator object that generates all pages

In [49]: all_pages = list(site.allpages())

In [50]: print(type(all_pages))
<class 'list'>

In [51]: print(len(all_pages))
2292

In [52]: print(type(all_pages[0]))
<class 'pywikibot.page.Page'>

The Page generator object works like a normal Python generator - it won't return page names until it needs to. Wrapping it in a list() call above will walk through every page before constructing and returning the list. Don't do this on big wiki sites like Wikipedia!!!

Flags