Mediawiki/OAI mirror

From Bjoern Hassler's website
Jump to: navigation, search
"Mediawiki mirroring and synchronisation"

Mediawiki mirroring and synchronisation - Mirroring - moving - Offline mediawiki (mvs, MWEclipse) - API/Mirroring - Mwlib - OAI - [Edit]

This page is about using the OAI extension to mirror a wiki.

Unfortunately, the mediawikiwiki:Extension:OAIRepository isn't very well documented, but in principle it can work to create a 'read-only' mirror of a wiki.

The scenario is that there's a public wiki somewhere (that you want to mirror), and that you have another wiki somewhere (e.g. on your laptop, or on a local area network) that acts as a mirror. With the OAI extension, it's possible to create a read-only mirror. Two-way synchronnisation is much harder.

1 Instructions[edit]

There are instructions on setting up the OAI extension on two wikis here: Mediawiki/OAI mirror/OAIRepository.

If you are mirroring to a local area network, you can set up the mirror using a LAMP stack as you would normally set up a wiki. If you are mirroring to your laptop, you could use something like WAMP/MAMP to run a LAMP stack locally.

Do have a look at the list of issues Mediawiki/OAI mirror/OAIRepository#Issues.

2 Uses[edit]

2.1 Mirror one wiki on a LAN[edit]

If the mirror wiki is available e.g. on a LAN, you should lock the db to prevent editing (need to check whether harvesting will still work in those circumstances). You could modify the edit links on the mirror wiki, so that people wanting to edit are redirected to the master wiki.

2.2 Mirror a wiki to your desktop[edit]

A very interesting us of this is also to turn a 'public wiki' into a 'personal copy', running on your desktop. In some ways it turns a public, live website, into a local wiki book (that you can still update).

There are of course some obstacles to this still: The wiki you wan to copy needs to run the OAI extension. Also, you need to run the wiki locally, which (for a novice) is not trivial. The easiest way is probably to use WAMP/MAMP, and install mediawiki / oai extensions. It would be nice to provide a packaged version: a mysql dump and the mediawiki code (including the OAI extension). Then you just get WAMP/MAMP, and you run the installer, and you're done.

What would also be nice would be to mirror several public wikis into a private wiki. This would require some namespace translation, and one owuld need to work out how to make sure a page looks for the templates etc in the right namespace. Nice as though this would be, it's not trivial.

2.3 Two way mirroring[edit]

It's tempting to try to use the extension for two-way mirroring. If there are no conflicting edits, this would probably just work by running the updater on both sides. But there is no method for resolving conflicts, the most recent change would persist.

However, one could do this asymmetrically, as an extra process on the 'client'. Before running the updater on the client, we use the OAI extension to check updates since the last 'checkpoint' (which is recorded in the OAI tables of database), as well as updates on the server. If the two sets of modified pages are disjunct, we simply send client-modified pages to the server, and then run the updater. This is fairly straight forward.

If there are pages that have been modified on client and server, then we need to record the timestamps/versions, and try to merge automatically, or resolve conflicts. (It gets more complicated if pages have been moved, as changes that may need to be merged are now in different histories.)

For an example of this, see pragmatic synchronisation.

2.4 Wikipedia[edit]

If you're hoping to mirror wikipedia, read http://meta.wikimedia.org/wiki/Wikimedia_update_feed_service

3 Links[edit]

Links: See also