Rockbox

Tasklist

FS#4755 - Wikipedia

Attached to Project: Rockbox
Opened by Anonymous Submitter - Wednesday, 01 March 2006, 02:03 GMT
Last edited by sideral (sideral) - Monday, 06 June 2011, 23:26 GMT
Task Type Patches
Category Plugins
Status Unconfirmed   Reopened
Assigned To No-one
Operating System All players
Severity Low
Priority Normal
Reported Version Daily build (which?)
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 11
Private No

Details

As already discussed in a thread [1] at mysticriver it would be very amazing to have a version of Wikipedia on rockbox.

Only a plugin to easily search articles would be needed.
Converted dumps are already available, for example the one of the ipodlinux project [2].

[1] http://www.misticriver.net/showthread.php?t=36924
[2] http://ipodlinux.org/Wikipedia
This task depends upon

Comment by Alexandre Flament (flament) - Sunday, 05 March 2006, 10:26 GMT Comment by Will Robertson (aliask) - Tuesday, 07 March 2006, 11:33 GMT
As far as I can tell, the script on the iPod linux page you linked just converts the files into plain text, which you could open in rockbox anyway - give it a shot.
Comment by Wiki River (wikiriver) - Tuesday, 26 September 2006, 17:49 GMT
If you look at the Misticriver thread and the "Rockipedia" wiki, you'll find information on a plugin of this kind being created.

Hope this helps.


Misticriver thread: http://misticriver.net/showthread.php?t=36924
Plugin wiki: http://rockipedia.techmight.com
Comment by Frederik (freqmod) - Thursday, 26 October 2006, 22:23 GMT
I have created a plugin to search out wikipedia articles, it takes about 5 seconds to search for an article (exact match) on an i-pod and uses b-trees for an index and gzip compression for the articles. (about 2.5GB, split into 1GB files for english retrieved wikipedia a week ago).

I will release the source soon (before 5.nov) when it is a little more finished.
Comment by Izzy (bro2baseball) - Sunday, 29 October 2006, 03:31 GMT
wow! that sounds awesome freqmod! i can't wait!
Comment by Frederik (freqmod) - Saturday, 04 November 2006, 11:15 GMT
Haven't made any more progress, but it works. The archiving functions are split out, but the viewer in mww.c and the parser in wmconv.rb could need a rewrite.
Comment by Izzy (bro2baseball) - Saturday, 04 November 2006, 23:02 GMT
Sorry for being such a noob, but how do i use this?

is that just the source? will there be a patch soon?

great work, btw. i can't wait to use it

bro2
Comment by Izzy (bro2baseball) - Saturday, 04 November 2006, 23:06 GMT
woops... i found the readme file.

i'll try to figure it out from that. though i don't really understand most of it :(

thanks again!

bro2
Comment by Adam Gashlin (AdamGashlin) - Sunday, 05 November 2006, 14:10 GMT
Brilliant freqmod! I'm now testing it on an iPod photo, reading the article about Stephen Hawking while listening to MC Hawking. Are you intending to work on rendering the articles?
Comment by Frederik (freqmod) - Sunday, 05 November 2006, 14:58 GMT
I won't work on rendering now (if someone wants to I will certainly help), but maybe after a while. I second the release of connell's source so I (and others) have a startpoint for the renderer.
Comment by Joe (rockboxer1) - Sunday, 05 November 2006, 16:58 GMT
I patched the rockbox build fine, but I don't know how to dump the wiki.

Are the directions in your readme for linux users?

When it says in your Readme

---run ./Make.sh
run bzcat <wikimedia dump> | ruby mwxmldumpparse.rb /dev/stdin <prepared dump prefix>--- ect...

What does run mean? How do I run it? I have to install Ruby, GCC, and bzip2? Any way to make this a little bit more friendly to newer users :)?

Your work is awesome btw. I've been waiting for this for a long time!

rckbxr
Comment by Frederik (freqmod) - Sunday, 05 November 2006, 21:40 GMT
ok, if you don't want to install unix you could install ruby from http://rubyforge.org/frs/download.php/12751/ruby185-21.exe and compile the ( c ) apps in cygwin/mingw (and upload the binaries here) where you compile rockbox. (the bzip2 file could be decompressed ( http://www.bzip.org/downloads.html ) before it is passed to ruby if you have enough disk space)
Comment by Adam Gashlin (AdamGashlin) - Monday, 06 November 2006, 17:56 GMT
I've rewritten the viewer, it is very basic but working (I was having a lot of trouble with stuff dropping out or cutting off with the earlier one). I'll be adding link following support later today if I can get to it.
   mww.c (9.3 KiB)
Comment by Adam Gashlin (AdamGashlin) - Monday, 06 November 2006, 18:43 GMT
The characters for italics are now ignored, and an issue with articles cutting off too soon has been fixed.
   mww.c (9.4 KiB)
Comment by Adam Gashlin (AdamGashlin) - Tuesday, 07 November 2006, 02:09 GMT
Links are now supported, Select to enter link selection mode, scroll to select a link on screen, Menu to back out of this mode, Select to load the linked article. There were some bugs in gzip which prevented it from working twice, those are now (mostly?) fixed.
The biggest irritants to me right now are: 1) backwards scroll speed 2) text insensitive matching which turns up the wrong pages (see FRance) 3) no "back" feature as one would expect on a browser.
   mww2.zip (13.2 KiB)
Comment by Adam Gashlin (AdamGashlin) - Tuesday, 07 November 2006, 06:15 GMT
You can now no longer scroll beyond the end of the article, and the link selector will stay on screen. Text sensitive matching is now attempted before insensitive matching, if there is an exact match we should use that (France is now accessible). The article title at the start of each article is now skipped. Scrolling may be a litte faster.
Comment by Adam Gashlin (AdamGashlin) - Tuesday, 07 November 2006, 07:44 GMT
Fixed a careless file descriptor leak.
Comment by Adam Gashlin (AdamGashlin) - Tuesday, 07 November 2006, 14:43 GMT
Scrolling backwards is now faster as the state of the last 50 lines is saved. There is a 10-deep history of pages, accessible by pressing Left. When pages are not found the viewer returns to the previous page (or exits if the start page was not found). Links that start before the first line on the screen now work properly. If no links are onscreen select will not enter link selection mode. The "searching..." display is a bit more consistent.

Does anyone care?
Comment by Frederik (freqmod) - Tuesday, 07 November 2006, 18:39 GMT
Wonderful job.

I have updated the MARKUP_* macros to match the values in wmconv.rb, added colored underlining to signalize underlined, italic and bold text. Added a little more memory in inflate.c (to fix some problematic articles).

When it comes to case (in)sensitive searching i don't know if a compare function that is different from the function used to create a b-tree will work, it will not however give false positives so it is no propblem, except that it uses a little more disk access time.
Comment by Adam Gashlin (AdamGashlin) - Tuesday, 07 November 2006, 19:49 GMT
A quick fix: insert
hist[curhist].curline=0;
at line 275 in mww.c.
curline was never being initialized.
Comment by Adam Gashlin (AdamGashlin) - Tuesday, 07 November 2006, 21:26 GMT
That exposed another issue regarding line counting (go to an article, go back, scroll up... jumps back to line 0 because the previous fix clears curline and we lose track of how many lines to seek ahead for backwards scrolling).
Should be good now.
Comment by Adam Gashlin (AdamGashlin) - Wednesday, 08 November 2006, 16:44 GMT
I used the viewer for a few hours yesterday (much easier to use with a large font to reduce eye strain) and came up with a list of things I think need doing (written in the dark at varying angles in red pen on random scraps of paper on my desk...):

* Don't reallocate memory when loading new articles - though I know the current system will produce the same addresses each time the page history relies rather strongly on that fact, which could be a potential problem if things change.
* Case sensitive search only when the article title is produced internally, i.e. from a link (only case-insensitive search for user-entered text)
* Save the index of an article when found, so upon going back to it we don't need to search again.
* Save current status to a file, to come back to it at a later point. This could produce problems if not handled carefully: changing font between runs will change line breaks (easily solved by storing the character offset of the beginning of the first line displayed and recaululating breaks upon loading), the database itself might change (I think this is unlikely enough to not worry about how to handle it, save to ensure it doesn't crash), and other potential pitfalls, I'm sure.
* Menu - to use the save functionality, also to enter a new article title (should be treated as if it was a link followed so we can always go back), perhaps toggle emphasis highlighting (as I don't particularly care for those colored underlines) and other configurables. Should take a look at the text viewer for ideas here.
* Better display of article title (right now it is ignored)

Stuff for the preprocessor (note however that I have not so much as read a line of it, yet):
* Process basic templates - stuff like seealso and main should be handled, as they are quite often used in large articles.
* Images - just put the alt text for the image
* Offsite links - not sure how to handle these... ignorance should be a fine policy
* Cross-mediawiki links - could potentially be supported if the database for the given article is also present, I don't consider this too important, though
* Categories - Generating a page for each category, and processing the template on each article to link to that category, would aid navigation
* HTML Entities - should be replaced with equivalent unicode characters (what about &gt; and &lt; ?)
* Anchors - How to support? Simple half-solution is to just strip off the anchor when following a link.
* Do something nice with ==headings==, lists (on further thought this may not be necessary, both are presented reasonably enough in the current code)
* Do something (even if not nice) with tables
* Do something with the ref tags
* Redirect processing seems to have some problems

It would also be helpful to take a nice long look at how wikipodia works, although I don't intend to prettify things too much and I'd perfer to keep it fairly simple.

Just some ideas I'll be working on. Is there a better place for this kind of discussion than in the feature request?
Comment by Adam Gashlin (AdamGashlin) - Wednesday, 08 November 2006, 16:50 GMT
Sorry, by wikipodia I meant Encyclopodia, it had been a while since I looked thoroughly at that.
Comment by Frederik (freqmod) - Wednesday, 08 November 2006, 19:08 GMT
A system for patching the wikis (to save network bandwidth) (I will try to make it):
* Compares each article with the article in the old version against the new and store diffs (gz-/bz-/lzma- ed) or xdeltas (I will see what that is most efficient, and if diff/patch produces _exactly_ equal files) of the uncompressed article text
* Store a "recepipe" (binary):
* Copy article: <name>
* Patch artilcle: <patch offset> <name>
* Create an ansi c program that compiles on Windows/Mac os X and Linux, with no dependencies that patches a database.

Things that would be nice:
A menu where you could access:
* Page history (a list, with page titles, and stored offsets (in database))
* Document structure (the parser should generate an index with (byte)offsets (localy in the decompressed article)
* Search for new article (with the last article name entered)
* Exit
Comment by Adam Gashlin (AdamGashlin) - Wednesday, 08 November 2006, 19:33 GMT
I understand and agree with all your points but the Document structure menu item, I am not clear what you mean. Headings, anchors, sections? Yes, that makes sense, now that I think of it.
I've made a fix for some issues with multiple whitespaces and wrapping (if there are two spaces at the end of a line one will be consumed for the break, the other will be at the start of the next line, which I don't like, the new version ignores multiple whitespace (with the exception of mutliple newlines)) but I'll wait to post it until I've done a bit more work today.
Comment by Giles (Giles) - Friday, 10 November 2006, 08:15 GMT
Ok, so I understand all about the sources and compiling and all, but try as I might, rockbox won;t compile! I upgraded my make, arm-elf-gcc and gcc and it still gives me errors. Props to anyone who will compile me a build of rockbox for the ipod video containing the latest media wiki viewer, that way I can show off my iPod and get all the chicks.

Please, I will be eternally grateful.
Comment by Frederik (freqmod) - Saturday, 11 November 2006, 20:38 GMT
1 Bug in btsearch.c:87 LOGF("K:%d,%s",utf8strcnmp(((const unsigned char*)key),((const unsigned char*)nd_key),rkeylen,keylen),nd_key); --> LOGF("K:%d,%s",utf8strcnmp(((const unsigned char*)key),((const unsigned char*)nd_key),rkeylen,keylen,casesense),nd_key);
2 my rockbox build http://freqmod.dyndns.org/upload/rockbox.11.11.06.patches.mww.zip (with mww, and a few other patches (some does not work as intended, but should not harm))
Comment by Adam Gashlin (AdamGashlin) - Saturday, 11 November 2006, 23:59 GMT
Rewrite of a substantial portion, now with: menu (save history, new article, exit, clear history), history save/restore, no need to search again when visiting pages in history.
Not too heavily tested yet, though I will surely be using it frequently.
Comment by Adam Gashlin (AdamGashlin) - Sunday, 12 November 2006, 00:17 GMT
Navigating to a new article through the menu didn't reset the scroll offset. Also made abort from keyboard behave properly (the code was there but commented out).
Comment by Izzy (bro2baseball) - Sunday, 12 November 2006, 04:11 GMT
Amazing work, Adam. Don't think you're not appreciated. I wish that I could figure out how to use this! I'm on windows and for some reason I can't figure it out. But I'm gonna keep trying; just wanted to let you know that you're doing great work!

bro2
Comment by Adam Gashlin (AdamGashlin) - Sunday, 12 November 2006, 05:15 GMT
Reinstated the "Loading..." message, renamed save files to .wws.
Izzy, if you can get in touch with me I can possibly walk you through the building, I'm often in #rockbox, my nick is hcs.
Comment by Adam Gashlin (AdamGashlin) - Tuesday, 14 November 2006, 01:07 GMT
I'm in the process of rewriting the xml dump parser in C with expat, as a first step towards improved preprocessing. I'm undecided about what to do with templates so far, though. Some I think it would be best to inline (main, seealso), some could use some special processing (categories), and some I just don't know about (infobox).
Comment by Adam Gashlin (AdamGashlin) - Tuesday, 14 November 2006, 12:42 GMT
A few additional features for the viewer (haven't gotten anything significant done with the preprocessor yet):
-seeking to targets (you know, the # things)
-navigation (the Navigate thing in the menu now displays an outline that you can scroll through, pressing select will jump to that part of the article)
-fixed a few bugs with saving
-history view (a list of the titles in the history)
Comment by Adam Gashlin (AdamGashlin) - Tuesday, 14 November 2006, 23:15 GMT
Faster backwards scrolling in outline navigation, additional option for entering a new article name from scratch, pressing select in the history view will jump back to the topmost displyed article.
Comment by Adam Gashlin (AdamGashlin) - Tuesday, 14 November 2006, 23:49 GMT
fix to make target seeking tolerant of spaces at the end of the heading
Comment by Izzy (bro2baseball) - Saturday, 02 December 2006, 22:23 GMT
Adam, is there anyway I can get the (already compiled) file to actually run the wikimedia dump? I did build rockbox with the mww viewer, but I can't figure out how to compile in VMware. I'm not exactly sure how the actual dump process works, but maybe you can make this a little easier for me (the windows user) :(

thanks for any help

bro2
Comment by Adam Gashlin (AdamGashlin) - Sunday, 10 December 2006, 15:27 GMT
bro2, what target do you have?
Comment by Izzy (bro2baseball) - Sunday, 10 December 2006, 16:37 GMT
Ipod 5g
Comment by Adam Gashlin (AdamGashlin) - Monday, 11 December 2006, 02:03 GMT
http://www.hcs64.com/wiki

Here you will find a somewhat out of date processed wikipedia dump and a build of the viewer from the latest CVS.
Comment by Izzy (bro2baseball) - Tuesday, 12 December 2006, 02:06 GMT
Thanks so much Adam! As I'm downloading these files, I just thought I'd ask where I should put them. They're(the .wwa files) all different parts of the wikipedia, correct? Am I able to search all these parts (the entire wikipedia) at once? Again, thanks a million. Your work is amazing!

bro2
Comment by Adam Gashlin (AdamGashlin) - Tuesday, 12 December 2006, 02:31 GMT
The wwa files are the actual article text, wwi is the index. They can be anywhere in the filesystem but they should all be in the same directory. The plugin actually opens the wwi file, which is then used to access the others.
mww.rock goes in .rockbox/viewers/, viewers.config replaces the one in .rockbox/ (or you can manually add the line
wwi,viewers/mww, 55 55 55 55 55 55
to the existing file)

Also, credit where it is due, the archives are all generated by freqmod's conversion utilities, and the database backend of the viewer is still mostly his.
Comment by Frederik (freqmod) - Tuesday, 12 December 2006, 17:03 GMT
Cleaned up warnings (in all files except mww.c where Adam has done a great job).
Added playbackmenu to the menu.
Comment by Frederik (freqmod) - Tuesday, 12 December 2006, 17:30 GMT
Uploading specification of the wwi and wwa format.

I don't have very much time to work on this project as I am preparing a program for showing videoes on multiple projectors which must be finished before a preformance in the last week of february (and for rehersals). I will problably work more on the patch solution for this viewer before that when I have made some more progress on that project.
Comment by Frederik (freqmod) - Tuesday, 12 December 2006, 17:47 GMT
(the specification was raw text, all numbers were in little endian)

all text lengths are in bytes

Input for btcreate (endiannes like the architecture of btcreate):
In-list:
uint32 data_lo
uint32 data_hi
uint32 title_length
char8*title_length title (in UTF8)
Redirect-list:
uint32 from_length
uint32 to_length
char8*from_length redirect from (in UTF8)
char8*to_length redirect to (in UTF8) (the datapointer is taken from this entry in the in-list)
Comment by emilie dancer (stardancer) - Monday, 01 January 2007, 00:17 GMT
Wow, just installed the viewer and made my own dumps from wikipedia. Thanks for all the amazing work! Can we change the status of this tracker to at least "Alpha"? So people know that it is in development and very functional?
Comment by Matthias Larisch (Matze88m) - Thursday, 18 January 2007, 12:20 GMT
I have problems with my own dump... dewiki becomes a 35mb wwi + 1gb wwa file, but the reader doesnt find ANY article! Some problems are at converting with illegal links, but that are only about 50 or so... The rest of the converting process finishes without any error.

When i download the enwiki dump from the link above it works without problems... When I use my own dump, it says "Searching..." and immediately (within same second, when not same 100th of the second) it says "Not found" and quits the reader.

I have to say that i use iriver h300 (with #define BUTTON_SCROLL_FWD BUTTON_UP and equal ones for SCROLL_DOWN and BUTTON_MENU) and the plugin works fine :) but only with the enwiki dump from above. The Ipodsimulator doesnt work with my dewikidump either.

Any ideas?

Thank you!
Comment by Matthias Larisch (Matze88m) - Thursday, 18 January 2007, 12:21 GMT
Oh if someone wants to look at my .wwi file: http://eow.ath.cx/other/tier/rockbox/dewiki20061130.wwi (currently uploading - finished in 30 minutes - 35MB)
Comment by Frederik (freqmod) - Thursday, 18 January 2007, 15:40 GMT
I have looked at the wwi file. If you could provide (a part of) the wwt file it would help. It seems like all the entries are in the .wwi index but all the values (pointer to the adresses in the wwa file) points to 0 0. If the wwt file also has 0 0 as the index then there is a problem with the ruby converter, else there is a problem with btcreate.

If you want to investigate yourself open the file in an hex editor and use the file format documentation as a reference.
Comment by Matthias Larisch (Matze88m) - Thursday, 18 January 2007, 19:43 GMT
Thanks for your fast response. Oh yes, in wwt-File there is everything (except one byte) between the titles set to 0. The ruby converter of your second post (mediawikiviewer.tar.bz2) is the right one? I'll have a look at the source and try a bit around...
Well there is one step i did different than the README:
instead of using bzcat file | ruby converter /dev/stdin outputfile

i used directly ruby converter <my_decompressed_dewiki.xml> <outputfile> because my cygwin doesnt have a device /dev/stdin.

I'll try doing that directy on a real linux box with readme command next night after looking in the ruby script.
Comment by Matthias Larisch (Matze88m) - Friday, 19 January 2007, 07:16 GMT
It works :) The cygwin ruby reads only zero at $filepointer.len which is used to write the offset to wwt file... I tested this with a debug-print in there. On my real linux machine it works, the generated wwi/wwa files do work too :))
Really nice work!
Maybe I'll write a fast C converter when I have the time (but that would be windows only at first - maybe someone will port it some day)
Comment by Izzy (bro2baseball) - Saturday, 20 January 2007, 22:45 GMT
I'd love if you did that, Matthias. A converter would be amazing! :)
Comment by Matthias Larisch (Matze88m) - Sunday, 21 January 2007, 00:16 GMT
Well I have begun today morning... It is able to create wwr File for now. wwt should be now problem but i have a general C/Pointer problem now :) I'm not very good in C. My code is verrrry dirty, but works and is much quicker than the original ruby code.
For now I have no parsing of any style - I expect a completely working alpha-version tomorrow evening :)
Comment by Izzy (bro2baseball) - Sunday, 21 January 2007, 04:16 GMT
Sweetness. Can't wait :)
Comment by Matthias Larisch (Matze88m) - Sunday, 21 January 2007, 08:22 GMT
But you know that u can also use the converter in freqmods second post? Well it is maybe a bit complicated, but mine wont be much easier ^^
Only problem is that it doesnt seem to run on a cygwin, dont know why. But you should try it with win32-ruby. Real linux should be best :)
My problem with it: pc is simply rebooting at output filesize ~720MB. Maybe that's my crappy PC, maybe a harddiskerror, dont know. Complete Output file is 1000MB in size. Thats the main reason why i'm coding a new program - so that i can use the whole wikipedia :)

What is your problem with freqmods converter?
Comment by Matthias Larisch (Matze88m) - Monday, 22 January 2007, 13:26 GMT
hmm will take a little bit longer, sorry :) Had to cope with a bufferoverflow which i didnt find:) Now everything is clear, work goes on and maybe finished today or tomorrow. But now i have to go to my real work for 5 hours :)

Ah btw a problem with the ruby converter: My PC always crashed at 2GB of the input file. this is what i found out. I didnt use directly decompression with bzcat but converter <wikidump.xml>. Maybe ruby has a problem with the 2GB limit?
Comment by Izzy (bro2baseball) - Monday, 22 January 2007, 21:37 GMT
Actually, I've never suceeded in dumping my own file. I merely used the kinda outdated files on Adam's website http://www.hcs64.com/wiki. I have a working cygwin installation, but the instructions included in freqmod's Readme weren't clear enough for me to finish. A n00b like me needs a step by step guide with everything included :-\\. I was hoping for something that would be a bit easier :-)

bro2
Comment by Matthias Larisch (Matze88m) - Tuesday, 23 January 2007, 00:19 GMT
Okay :) converter is in a useful state now. But it doesnt do ANY layout/article processing. It only creates one big wwa file and the two wwr/wwt files.
Just if anyone needs it now... I would call it 0.1 pre-alpha :) because of the missing main functionality.

how to:
run compile (requires zlib & pcre devel packages)
start xmlconv with wikidump as first argument and output prefix as second argument.
(cygwin or reallinux!)

After that u have to use the btcreate from freqmod.

dont blame me for bad coding... I'm not that good at it ^^

conversion time: dewiki 2,5gb -> 1gb wwa file in exactly 45 minutes on amd athlon xp 1700+ with 512mb sd ram running debian 3.0
Comment by Matthias Larisch (Matze88m) - Wednesday, 24 January 2007, 19:43 GMT
okay here is a new version :)

Conversion time on 1,86GHz Pentium-M Laptop (slow HD...) was 13 minutes for de-wiki. I think thats quite amazing in contrast to original converter :)

limitations:
-doesnt really parse the XML, only wikipedia "style" will be parsed correctly
-Parsing of layout (bold, etc.) is done via char for char comparison... this speeds up and should be expandable later (regexps should work also)
-I dont know how mww expects Links (links work, but not links with other name than the target)
-gzip-compression really bad implemented... writes to temp.gz file on harddisk and rereads it... I couldnt find an easy to implement way of in-memory compression. This should give another speedup :)

I added precompiled binaries compiled on my cygwin machine.
Libz and pcre-development packages are needed for compiling.

Have fun with it :)


Comment by Frederik (freqmod) - Wednesday, 24 January 2007, 21:19 GMT
Links: A_START <link/description> A_END
or: A_START <link> G_MODE <description> A_END
From the wwi-specification:
Name ASCII-code:
A_START 007
A_END 008
G_MODE 015

I hope this helps you :)
Comment by Matthias Larisch (Matze88m) - Wednesday, 24 January 2007, 22:33 GMT
Ahh, G-MODE is the one i look for :) But it is not implemented in mww?

Was a very simple addition to the converter...
It now should work with same features as yours :)
I forgot to say: It automatically strips off html comments (<!-- -->) and converts &gt; and &lt; to > and <.
One weird thing: my article.wwa file was shrinked from 1000MB to 870MB after implementation of the whole conversion...
Okay, for ''''' it is 5:1 compression, but that shouldnt be sooo many savings... Same for html comments... I browsed through a few articles, they were allright.

--snap
found mistake :) I declared comment-endings as --&lt; but they are --&gt; :) so articles with comments should be missing



attachment: new xmlconv.c
-changes version number to 0.2
-fixes comment-end mistake
-added functionality: alternative link-texts are supported
(application/octet-stream)    xmlconv.c (16.4 KiB)
Comment by Matthias Larisch (Matze88m) - Thursday, 25 January 2007, 20:48 GMT
freqmod: there is a problem :) My links wont work now!
After looking to mww.c:
#define MARKUP_BAR 13
this is used for seperating link from description!

In your converter u have PL_END as 13 and use that :)

i'm a little bit confused...
Comment by Frederik (freqmod) - Saturday, 27 January 2007, 21:11 GMT
I understand why you are confused. I don't understand the links myself.
My convertor:

variable declarations:

A_START=''<< 007
A_END=''<< 8
...
PL_END=''<< 013
G_MODE=''<< 015

links (with pipe):

arr[i]=Mwtags::A_START+tmp[0,pipeidx]+Mwtags::G_MODE+tmp[pipeidx+1,tmp.length-pipeidx-1]+Mwtags::A_END
(this line uses G_MODE)

The output makes a link with 13 (PL_END) as separator. I don't know why, it seems like a bug/unwanted feature in my code. However your convertor (Matthias) works just as well and is much more efficient and makes my ruby convertor obsolete. Therefore i have rewritten mww.c to use 15 as G_MODE.

This breaks compatibillity, but this plugin is in no way stable so I think it would be better to change the behavour than to change the "specification".

Updated convertor, to work with xmlconv:
   mww16.tbz (23.4 KiB)
Comment by Frederik (freqmod) - Saturday, 27 January 2007, 21:12 GMT
ehh, updated mww
Comment by Adam Gashlin (AdamGashlin) - Thursday, 15 February 2007, 21:02 GMT
Updated xmlconv.c to support reading from stdin (by using - as the input file name), so I can pipe the output of bzcat directly into it without using up disk space on the uncompressed archive. Full enwiki takes about 1 hour 15 minutes. I'd like to avoid the creation of the temp.gz thing as well but I haven't gotten to it yet.
I hope to get some more work done on mww now that I'm nominally done with NSF and SPC.
   xmlconv.c (16.5 KiB)
Comment by Matthias Larisch (Matze88m) - Thursday, 15 February 2007, 21:55 GMT
nice work :)

i didnt get zlib working like expected (gzip headers etc.) without the temporary file. In-Memory compression without this file should give another speedup of about 10% I think... Did file splitting work correct? I never tested this ^^ dewiki is only 980MB compressed :)
Comment by Adam Gashlin (AdamGashlin) - Thursday, 15 February 2007, 22:17 GMT
Yes, files are split correctly, there are three.
Comment by Adam Gashlin (AdamGashlin) - Friday, 16 February 2007, 08:30 GMT
Updated mww.c to use up and down keys instead of scroll so it'll build for gigabeat, seems to work fine.
Since there are a few of this potentially working on this at the same time, should we set up a sourceforge project for it for version control until it is ready for rockbox?
   mww.c (39.1 KiB)
Comment by Adam Gashlin (AdamGashlin) - Saturday, 17 February 2007, 21:14 GMT
Fixed some problems with redirect processing (fseek was using the wrong file descriptor), more entity handling in xmlconv (quotes, especially), no use of temporary file for gzip but no speed improvement (perhaps need to directly use deflate).
Comment by Adam Gashlin (AdamGashlin) - Monday, 19 February 2007, 02:01 GMT
Here's the version of the conversion tools I'm currently working with. xmlconv would do odd things if there was wikicode inside a comment, I moved comment processing first in the parsing code to combat this.
I'd like to get rid of the ref tags, as well, I don't care too much about the references and they make things a lot less readable.
Packed everything together for ease of management, and also mww17 so one doesn't have to pick things up all over this task...
Comment by Adam Gashlin (AdamGashlin) - Monday, 19 February 2007, 02:06 GMT
reuploading converter04 as .tar.bz2
Comment by Adam Gashlin (AdamGashlin) - Tuesday, 20 February 2007, 07:39 GMT
Reimplemented article parsing with a recursive-descent parser (mostly), should allow much more advanced parsing in the future on this framework. Newly supported things include images (well, a link to the page with info about the image shown as [I], and normal text processing on the caption, which may itself include links). The code is something of a preprocessing monstrosity, but I think it lends itself to being easier to tell what the parser is actually doing. Speed is pretty much the same as the old version.

(If anyone disapproves of my totally ripping apart xmlconv, sticking my name in it, and releasing with a new version number, let me know...)
Comment by Adam Gashlin (AdamGashlin) - Tuesday, 20 February 2007, 07:56 GMT
Oh, and I remove ref tags.
Comment by Adam Gashlin (AdamGashlin) - Tuesday, 20 February 2007, 16:36 GMT
Revised entity support to understand the fact that &ndash; is for instance &amp;ndash; in the dump. Added support for all HTML entities I've been able to find tell of. Despite the fairly large number and an inefficient linear search for them, they are the exception and do not seem to slow down processing at all.
Comment by Adam Gashlin (AdamGashlin) - Wednesday, 21 February 2007, 23:49 GMT
ref checking now part of general text processing (to catch it anywhere), added support for self terminating ref tags.
I'd like to once again redo this, with a stage for each level of translation done (1. xml 2. wikicode 3. html rendering) which I think will present a more logical flow for things and improve some bits that are sort of hackishly scattered in at the moment. Might be a while before I get to it, though.
Comment by Matthias Larisch (Matze88m) - Thursday, 22 February 2007, 06:24 GMT
You are doing very great work, exactly what i hoped to see after my initial "new convertor". I do not have any problem with you ripping away the whole program or with sticking your name in it. As i earlier said, my version is very dirty but already worked :) If you manage to write a converter that does much better job (which it already does) in nearly same time, this is great! I personally wont do much work on this project in future because I have very little time and not the programming skills needed...
Comment by Adam Gashlin (AdamGashlin) - Thursday, 22 February 2007, 21:55 GMT
Those ref tags continue to be a pain, I forgot that html tags are case-insensitive.
I'm wondering if it might be a good idea to try and use the static HTML dumps? That way we wouldn't have to worry about some of the more irritating to implement aspects of wikicode, like templates (though I've given them a lot of thought). The downside is that they are rather infrequently updated. I'm not considering this too seriously.
I think the multi-stage approach, firmly grounded in the structure of the grammars involved, should be fine. I've just got to get some structural aspects of it decided. I'm taking a compilers course now so I'm trying to apply what I'm learning about parsing to this. Wondering if it might be better to just see how mediawiki does it and rewrite in C, though...

Considering templates again, should they be expanded inline or kept as a seperate thing? I'm thinking inline, but I worry it might make things a lot bigger than they need to be.
Comment by Adam Gashlin (AdamGashlin) - Wednesday, 09 May 2007, 16:34 GMT
sync'd, and formulated as a patch for easy application
Comment by Adam Gashlin (AdamGashlin) - Sunday, 17 June 2007, 18:57 GMT
synchronized
Comment by Timo Horstschäfer (x1jmp) - Saturday, 23 June 2007, 11:10 GMT
As this plugin and FS#6697 have almost the same purpose, it would be better if we could merge the projects or work together on one of them.

The advantages of this plugin is its support for links, the gzip compression and binary tree search (maybe I missed something, because I hadn't had a deeper lok at this plugin for some time...).
What I like about my dict plugin is that it isn't limited to accessing the Wikipedia (a lot of other dictionaries as well) and uses a simple fuzzy search.
I think it's also a better idea to create a plugin-independent document viewer for Rockbox.

So what are your opinions?
Comment by Adam Gashlin (AdamGashlin) - Wednesday, 27 June 2007, 05:44 GMT
While you have not had a deeper look for some time, I've never looked at FS #6697, so I really have no idea. I can barely find the time to sync this now and then, but I agree that the effort would probably be best spent on a plugin somewhere between these two. I'm just not able to do any of it myself at the moment.
Comment by Alistair Marshall (amar) - Tuesday, 24 July 2007, 19:29 GMT
should this task be changed to a patch rather than a feature request?
Comment by peter watkins (peterw) - Wednesday, 15 August 2007, 20:20 GMT
Frederik, Adam, et. al. -- thanks for creating this! I'm enjoying having Wikipedia (EN) on my iPod.

Here are a few notes for other users from my experience with this and a fairly recent (week old) Subversion checkout:
- The English Wikipedia dump would uncompress to to something like 11 GB. If you're processing the dump on a filesystem that can't handle files that big (say, the fat32 fs on your DAP), you'll want to use pipes to avoid the need to make an uncompressed file. On Linux, something like the following will allow you to make the article files.
bunzip2 -c enwiki-20070716-pages-articles.xml.bz2 | ./xmlconv /dev/stdin wikipedia
- How it works: this creates a "viewer" for .wwi files. To view/search a processed wikimedia article dump, use the Files browse manu, find the ".wwi" file (both the wwi and wwa files must be on your DAP, as noted above), and open it. If your Rockbox DAP is configured to show only supported files, you'll see the name of only the wwi file.
- Fonts and non-ASCII characters. Not all Rockbox fonts have good support for non-US-ASCII characters. If you find that letters with diacritical marks are not displayed properly, try a different font.

Timo -- I tried your dict plugin (was really looking forward to it), but kept running into errors (iPod 5.5g, 64mb). This wikimedia viewer seems more stable.

Personally, I'd also be interested in a PC app to convert dict files to wikimedia XML format so they could be viewed with this plugin.
Comment by Frank M. (framo) - Sunday, 19 August 2007, 12:30 GMT
Dewiki and enwiki are working fine on my iPod nano and video :)
The patch doesn't compile for Sansa e200 though.
Comment by Adam Gashlin (AdamGashlin) - Wednesday, 29 August 2007, 12:51 GMT
I'm looking at WikiFilter (wikifilter.sf.net) as a starting point for wikicode parsing, and I've downloaded a few dumps to start experimenting with updates.
And at some point I should look at that dict plugin...
Comment by Timo Horstschäfer (x1jmp) - Wednesday, 29 August 2007, 16:31 GMT
I suggest having a look at FlexBisonParse which is listed on http://meta.wikimedia.org/wiki/Alternative_parsers.
It looks quite easy to adapt to custom output.
Comment by Adam Gashlin (AdamGashlin) - Tuesday, 04 September 2007, 14:24 GMT
Updated patch, I don't think there's anything new here but it'll apply to current SVN.
Comment by Simon Wenger (musician72) - Saturday, 08 September 2007, 19:24 GMT
Can someone point me to wikipedia dumps already prepared for this plugin? (The links above give me 404's)
Thanks, Simon
Comment by Adam Gashlin (AdamGashlin) - Sunday, 09 September 2007, 02:17 GMT
Indeed, that page is down. I don't have the capability to upload the 3.8 GB the most recent converted enwiki dump (2007/08/02) takes up, unfortunately, though I do have the web space. If someone else is interested in volunteering I can set up an FTP account to put it up in my space.
Comment by Simon Wenger (musician72) - Sunday, 09 September 2007, 06:51 GMT
Ok. I'm downloading and converting the dewiki (german) now. I might just as well upload it later, after testing. I'll contact you about an FTP account then.
And I'd like to write a short HOWTO on the patching (no details) and the downloading/converting of the dump. Should I just place it in a comment or could we put it in the top (Details section)? It's confusing to have the old converter method and dead links in the earlier posts.
Comment by Simon Wenger (musician72) - Monday, 10 September 2007, 07:07 GMT
The patch applies to the current revision, but doesn't compile.

MAKE in mww
CC mww.c
mww.c: In function ‘set_article_offset’:
mww.c:189: warning: implicit declaration of function ‘printf’
mww.c: In function ‘viewer_init’:
mww.c:936: error: ‘BUTTON_MENU’ undeclared (first use in this function)
mww.c:936: error: (Each undeclared identifier is reported only once
mww.c:936: error: for each function it appears in.)
make[3]: *** [/home/simon/Desktop/iaudio/bleeding/rockbox/build/apps/plugins/mww/mww.o] Error 1
make[2]: *** [mww] Error 2
make[1]: *** [rocks] Error 2
make: *** [build] Fehler 2

From the little I know about programming, I guess it's because of the target? I compile for iaudio X5 (sim) which does not have a menu button.
Am I right. Is there a way out of this?
Comment by Simon Wenger (musician72) - Monday, 10 September 2007, 07:28 GMT
Hmm, can't compile for H300 sim either... and this target knows a menu button (A-B).
Comment by Daniel Dalton (ddalton) - Monday, 10 September 2007, 07:35 GMT
Hmmm it looks like "BUTTON_MENU" hasn't been declared anywhere. You will have to find out what it is called. Maybe BUTTON_MENU is the menu button? (Ab on h300)
Comment by Adam Gashlin (AdamGashlin) - Monday, 10 September 2007, 08:26 GMT
Looking a the keymaps, it looks like BUTTON_MODE|BUTTON_REL is used for the menu function on the h300, so just drop that in there. I should probably find out how to get access to the keymap stuff so mww doesn't have to be too nastily specific to each target.
Regarding making things easier, we're really in a horribly hacky incomplete state now, especially with the converter (though mww is a mess codewise, too). I'd rather it not be too easy to do things the wrong way, especially to avoid transitioning issues when a better way of keeping things updated comes along. This is, however, merely my position, and I'm not doing anything active for this project at the moment anyway (sorry for the grand plans a few messages back), so by all means make things happen if you wish.
Comment by Frank M. (framo) - Wednesday, 12 September 2007, 12:37 GMT
It compiled ok for my Sansa E280 when I replace BUTTON_MENU with BUTTON_POWER in mww23.diff.
The article scrolling is a bit strange (using the up/down button instead of the wheel) but it works.
Comment by Simon Wenger (musician72) - Wednesday, 12 September 2007, 21:03 GMT
Yes, I did almost the same. I replaced BUTTON_MENU with BUTTON_MODE|BUTTON_REL, works perfectly, thanks! An impressive piece of software!
Adam, you could send me an email with the ftp login now, I will then upload the converted dump (german atm, maybe someone else can do others?), and, later a little howto. simon(?)jso.be
Comment by Xinlu Huang (polygonal) - Sunday, 23 September 2007, 23:20 GMT
I encounter a stack over flow when using the xmlconverter:
Exception: STATUS_STACK_OVERFLOW at eip=00403333
eax=005F73BC ebx=00000000 ecx=00032CDC edx=0000004C esi=611001A0 edi=004037A0
ebp=0022CCE8 esp=0022CCD4 program=C:\cygwin\home\...\xmlconv.exe, pid 5788, thread main
cs=001B ds=0023 es=0023 fs=003B gs=0000 ss=0023
Stack trace:
Frame Function Args
0022CCE8 00403333 (00000003, 61169690, 01630090, 00000000)
0022CD98 61006198 (00000000, 0022CDD0, 61005510, 0022CDD0)
61005510 61004416 (0000009C, A02404C7, E8611001, FFFFFF48)
8 [main] xmlconv 5788 _cygtls::handle_exceptions: Error while dumping state (probably corrupted stack)

I'm using cygwin on XP, if that helps. Any idea why this is happening?
Comment by Adam Gashlin (AdamGashlin) - Monday, 24 September 2007, 23:00 GMT
I've never encountered that error, then again I've never tried using the converter in cygwin.
Exactly which mediawiki dump are you using?

In other news, Simon's conversion of dewiki (20070903) is now on my web site.
http://hcs64.com/rockbox/wikipedia/
Comment by Xinlu Huang (polygonal) - Tuesday, 25 September 2007, 02:49 GMT
I use enwiki-latest-pages-articles.xml.bz2 from http://download.wikimedia.org/enwiki/latest/

The link to the converted wiki is interesting, but I don't understand a word of German ;) I'll need to find some way to get the enwiki in the right format...
Comment by Adam Gashlin (AdamGashlin) - Tuesday, 25 September 2007, 06:36 GMT
Aaand, my mirror of dewiki is now down. Went through 250 GB in a few hours (I have 3.6 TB monthly but they won't let me use it all up in a day). I'd try something like a torrent but I don't have anything to host it from.
Comment by Xinlu Huang (polygonal) - Wednesday, 26 September 2007, 21:52 GMT
I got Ubuntu for this and the xmlconv works pretty well on my USB stick ;) There is still quite some tags not parsed - ouststandingly (for me) some tags in the beginning of articles, the quote tags, the math tags (which I of course don't expect a nice integral picture or something, but reading <math></math> is quite annoying, and for most case it will be simple removal of the math tag itself). I wonder are these unparsed tag expected?

Also I'm curious about the coding in the converter: is UTF-8 used?

As for the cygwin problem, I'm guessing that cygwin restrict mem usuage, etc. since it is running in a host enviroment, so maybe that's where the stack overflow comes from.

It is so wonderful to have wikipedia on my ipod :D
Comment by Adam Gashlin (AdamGashlin) - Friday, 28 September 2007, 00:59 GMT
Yes, UTF-8 is used. Regarding the HTML tags, they're handled on a case-by-case basis, and many are not handled at all. The converter needs a rework. Sorry for the extended "proof of concept" phase.
Comment by Fed (Fed) - Saturday, 06 October 2007, 14:09 GMT
Is there a way of getting a peared down set of wiki pages?
Comment by Adam Gashlin (AdamGashlin) - Saturday, 06 October 2007, 16:05 GMT
What do you mean? Like a selection of a smaller number of important pages? There are projects that do that but I don't think they provide XML dumps.
Comment by Fed (Fed) - Saturday, 06 October 2007, 16:17 GMT
Where am I supposed to put the xml file on the sansa?
Comment by Fed (Fed) - Saturday, 06 October 2007, 18:16 GMT
That's exactly what I mean. Can I save some pages, and make my own xml file and use that?
Comment by Simon Wenger (musician72) - Saturday, 06 October 2007, 22:10 GMT
Here's the very first version of a really simple HOWTO. It took me a while to figure out, where to get and how to convert the wiki dumps and how to compile for other targets than ipod. Hope others can make a shorter way.
Please feel free to send me any improvements. I will love to include them.
I suggest to make the document available in the head of this page, as it has become very long and complex by now.
Thanks to Matze88m for the work on the converter and freqmod and AdamGashlin for this fantastic plugin, I've spent at least 20 hours in the past month, learning my head off....
Comment by Fed (Fed) - Sunday, 07 October 2007, 09:10 GMT
Thanks a lot for the howto!!!
Comment by Fed (Fed) - Sunday, 07 October 2007, 13:22 GMT
I get an error when I try to compile. It looks like pcre.h is missing.
Comment by Xinlu Huang (polygonal) - Sunday, 07 October 2007, 13:32 GMT
Your linux distro (or cygwin?) does not have the libpcre package. You have to install it.
Comment by Fed (Fed) - Sunday, 07 October 2007, 13:38 GMT
I am using a mac. Do you know what I have to install?
Comment by Fed (Fed) - Sunday, 07 October 2007, 13:49 GMT
I seem to have found a pcre.c for mac. Does this seem right to you? Where should I put it in the file tree?
   pcre.h (12.1 KiB)
Comment by Fed (Fed) - Monday, 08 October 2007, 15:13 GMT
I am sorry about all the posts. I have finally gotten everything to work.
The pcre for mac is available at http://pcre.darwinports.com/

I use a Sansa, and in addition to "replace BUTTON_MENU with BUTTON_POWER (Thanks framo!)", you should replace BUTTON_SCROLL_BACK with BUTTON_SCROLL_UP, and BUTTON_SCROLL_FWD with BUTTON_SCROLL_DOWN

I also added "rb->backlight_set_timeout(1);" after "rb = api;" as well as
"rb->backlight_set_timeout(rb->global_settings->backlight_timeout);" before "return PLUGIN_OK;" so that the backlight stays on while I read, and goes back to the user settings when the plugin closes. I think if this could be set up as an option it would be very useful.

Would it be possible to have a function to copy text to a text file? (ie export a selection)

Also, could someone add a bookmarking function? I don't even know where to start for that.

Thanks again for such a great program!



Comment by Simon Wenger (musician72) - Monday, 08 October 2007, 17:06 GMT
Glad it worked! Bookmarking is implemented. Press rec (in my case) to get the menu, then save the history. Now the last article is reopened when restarted and you can view the history and jump back. As long as you don't resave the history it will remain unchanged.
The backlight thing would be really cool!
Comment by Fed (Fed) - Monday, 08 October 2007, 21:21 GMT
What about copying text. Any idea how that could be done?

The problem I find with the bookmarking as it is now is that you can't pick and choose what to keep. You have to keep it all.
Comment by Fed (Fed) - Monday, 08 October 2007, 22:47 GMT
I am trying to make a faster scroll (using the rec button), and I think I found an error in the code.
Should line 224 be:
advance_scrollback(1,(dorender && i==a-1));
it is currently
advance_scrollback(1,(dorender && i==-a-1));

Comment by Adam Gashlin (AdamGashlin) - Monday, 08 October 2007, 23:39 GMT
Bookmarking for individual articles would be nice.
Yes, that would appear to be a bug on line 224. Doesn't show itself because I never use multiline scroll except when scrolling backwards past the scrollback buffer. I guess I just never noticed when that failed to render as it only happens on the one line when the buffer must be regenerated, and I only ever tested that when scrolling back quickly so a single frame went unnoticed. I just tested this now and it is in fact a problem in the current build.
When you've enabled your faster scrolling and the backlight fix (as that had bothered me as well) could you post an updated patch? If not I could, I guess.
Comment by Fed (Fed) - Tuesday, 09 October 2007, 00:29 GMT
Here is the new mmw.c
I haven't made the patch because I have a lot of other changes, and it is too confusing. Can you make the patch with this?
   mww.c (40.2 KiB)
Comment by Fed (Fed) - Tuesday, 09 October 2007, 13:38 GMT
I was thinking about how to save text.
How about using the right key to advance the text (ie in the 'normal mode') and at the same time append the 'current line' to a file named [title of the article].txt
Comment by Fed (Fed) - Wednesday, 10 October 2007, 08:54 GMT
Here is my latest version of mww.c
It does a more universal page scroll.

Also,when you click the right arrow, it saves the current line of text to a file (in a wiki folder which you have to make prior to using it), but I only get the first 3 letters for some reason. I really cannot figure out why. Can anyone help?

Thanks
   mww.c (41.9 KiB)
Comment by Fed (Fed) - Thursday, 11 October 2007, 04:58 GMT
This version creates a wiki folder, where the 'recorded' lines are stored. The lines are stored, one at a time as you press the rec button, in a file titled by the wiki page you were viewing.
   mww.c (41.8 KiB)
Comment by Fed (Fed) - Thursday, 11 October 2007, 05:45 GMT
I cannot figure out how to make a diff file for this. I have tried a number of things, but no luck. Could someone let me know how or do it themselves? It seems that "svn diff > mmw24.diff" doesn't recognize the mww folder.

My addition works very well now, and I would like to add it to the project.
Comment by Adam Gashlin (AdamGashlin) - Thursday, 11 October 2007, 19:37 GMT
You need to do
svn add rockbox/apps/plugins/mww
first (minus the first rockbox/ if you're already in the rockbox/ directory, of course).
Comment by Fed (Fed) - Thursday, 11 October 2007, 23:57 GMT
Thanks. Here you go. I hope I did it right!
Comment by Fed (Fed) - Sunday, 14 October 2007, 05:25 GMT
Bookmarks done. Also, right arrow scrolls forward in history.
   mww25.diff (105.7 KiB)
Comment by Adam Gashlin (AdamGashlin) - Sunday, 14 October 2007, 11:44 GMT
Your patches are a bit odd, all the files but mww.c are in rocbox/whatever, mww.c is just in apps/plugins/mww/
Comment by Fed (Fed) - Sunday, 14 October 2007, 15:11 GMT
Sorry. I fixed it.
   mww25.diff (105.7 KiB)
Comment by Fed (Fed) - Tuesday, 23 October 2007, 00:27 GMT
I am having a problem trying to make the recording feature record the top line of the screen rather than the second line (as it is now). In order to do this, I need to place the contents of the 'current line' into a string.

ie I need to copy

buf[(i<=LINEBUFLEN)?i:LINEBUFLEN]='\0'

into a global string. I have tried many things but no luck. Can anyone help?
Comment by Fed (Fed) - Tuesday, 23 October 2007, 18:00 GMT
Major fixes to the recording function, and a few minor fixes to scrolling.

To get a gap in the recording between areas where you record the text, press and hold the select button.

Also, I am using the select button for scrolling down one page because it is easier that way on the Sansa.

   mww26.diff (107.1 KiB)
Comment by Adam Gashlin (AdamGashlin) - Sunday, 04 November 2007, 05:29 GMT
Updated to build once again on Gigabeat F and iPod 4G, as well as the e200 and c200 Sansa. It now uses a set of #defines near the top to set the keymappings based on the device so we don't have to go flipping those around.
I don't think I understand the "gap in recording" feature, though.
Comment by Fed (Fed) - Sunday, 04 November 2007, 15:28 GMT
It is useful for when you record a few lines of text and then move to another area in the article to record another few lines. The gap adds "\n\r" to separate the two sets of recorded text. Otherwise, you would have continuous text, and it would be more difficult to read and find the individual records.
Comment by Adam Gashlin (AdamGashlin) - Monday, 05 November 2007, 06:16 GMT
Ahh, I see now. I misnamed the macros, then, sorry.
Comment by kimangroo (kimangroo) - Wednesday, 14 November 2007, 20:59 GMT
Thanks a lot for all the work put into this, I'm an avid wikipedia user so it will be great to have it on my ipod.

Unfortunately I haven't been quite successful yet though...

1) got the same stack overflow problem as Xinlu with the converter in Cygwin in XP. Has anyone had success using Cygwin and xmlconv?

2) am now trying to use the converter with a Linux live CD (knoppix or ubuntu) but I don't understand how I can convert an 11+ Gb file on a FAT32 drive... How big would a converted file be? Does anyone have any pointers?
Comment by Adam Gashlin (AdamGashlin) - Thursday, 15 November 2007, 09:58 GMT
The converter is designed to output .wwa files of only up to 1 GB in size.
Comment by kimangroo (kimangroo) - Thursday, 15 November 2007, 19:58 GMT
^^Duh! Sorry for be dumb!

Knoppix live cd made short work of the converting and now it's all up and running on my ipod so thanks a million.

Can't seem to get the links working though. Pressing select works as page down scrolling. Have I got the wrong patch?

Anyway thanks again so much to everyone who put work into this!
Comment by Fed (Fed) - Thursday, 15 November 2007, 21:38 GMT
Arrow down activates the links. I found that page down was used more frequently, so I use "select" for it. Any suggestions which is better?
Comment by Alistair Marshall (amar) - Friday, 16 November 2007, 09:09 GMT
It would make much more sense arrow down to scroll down and select to activate link. Try to keep them as obvious and natural as possible
Comment by kimangroo (kimangroo) - Friday, 16 November 2007, 21:17 GMT
Hmm, I can't work out what I'm doing wrong here! When I press Arrow down I scroll down line by line, when I press select I do page down. Can't seem to work out how to select links.

I tried to patch the source again and recompile but now cygwin's telling me "cannot find file to patch at input line 5, Perhaps you used the wrong -p or --strip option? blah blah blah..."

As you may have guessed I'm a little new to this, I'm sure I'm missing something really stupid. ;)

When/if I do get it working I agree with Alistair that it would probably make more sense to have arrow down to scroll page down and select to activate a link.
Comment by Fed (Fed) - Wednesday, 21 November 2007, 12:46 GMT
I tried converting the wiktionary.xml file I downloaded, but I get many "unsupported entity" errors. Also, after using btcreate, I get "SCRLD".

The converted file seems to work on my Sansa, however. Is there a problem, or should I just ignore these "unsupported entity" errors?
Comment by Adam Gashlin (AdamGashlin) - Wednesday, 21 November 2007, 16:52 GMT
I don't remember what "SCRLD" was for, but the "unsupported entity" messages simply indicate that there is more work to do in the parser.
Comment by Fed (Fed) - Thursday, 22 November 2007, 13:20 GMT
The wiktionary seems to work fine. I would like to make the "find article" function find all articles that start with the search string, not just an exact match. I cannot figure out how to implement that, though. I looked through the code, and I think the function to adapt is "search_btree" in "btsearch.c". I just cannot figure this one out. Any help would be great.
Comment by kimangroo (kimangroo) - Friday, 23 November 2007, 06:48 GMT
Still can't get this to work properly unfortunately. Has anyone else got the links to work with an 80G Ipod video? Or is anyone else having the Arrow down scroll down as well?

If anyone has got a working patched Rockbox.zip for 80Gb Ipod, do they feel like sharing it with me?! My email is my username + "@gmail.com"

Thanks! ;)
Comment by kimangroo (kimangroo) - Friday, 23 November 2007, 06:50 GMT
Or you could upload it here. Just noticed that you can attach a file of up to 2 MiB which should be just enough!
Comment by Fed (Fed) - Saturday, 24 November 2007, 05:56 GMT
I think it would be nice to implement a new search that lists all articles from the .wwi list starting with the search string. For example, searching for 'car' will produce a list starting with 'car', then 'card' then 'care'. You could then scroll to the entry you want, and load the article.

The reason I don't do this is that I cannot figure out how to get the wwi data extracted. Any help would be appreciated.
Comment by kimangroo (kimangroo) - Saturday, 24 November 2007, 06:36 GMT
Fed I think that's a great idea and would really increase the usefulness of the viewer. I'd love to help you but as you can see I don't have the first clue about programming, rockbox or anything much!
Comment by almargo (almargo) - Saturday, 24 November 2007, 17:49 GMT
I have made a new mww.c which replaces the page down with select a link. It also includes the keymaps for iriver h100/h300.
   mww.c (51.2 KiB)
Comment by Fed (Fed) - Tuesday, 27 November 2007, 01:12 GMT
I am still trying to get a write a new 'find' function, but I cannot seem to figure out how to access the article titles. I think they are in the wwi file. Is this correct? Also, how can I access these titles individually? I cannot seem to figure out the functions that were written for this ap. I think the function in question is "search_btree" in "btsearch.c" Any help would be great.
Comment by Isaac (Izzeh) - Tuesday, 27 November 2007, 09:32 GMT
Damn this looks awesome... I'm fearing reading through that huge wall of text to understand how to patch it in, but I'll give it a shot.

Hah.. figured the file size would be huge (currently capped and downloading at >1kbps)
Comment by Isaac (Izzeh) - Tuesday, 27 November 2007, 22:50 GMT
Hmm I tried patching the file in and it doesn't seem to have worked. After further investigation I found that it's placing a Rockbox folder inside the Rockbox folder, rather than placing directly into the apps?
Comment by Timo Horstschäfer (x1jmp) - Thursday, 06 December 2007, 21:43 GMT
Have you considered using a hash table instead of the b-tree? It's is simpler and a lot faster than a b-tree so maybe a "search while typing" would be possible.

The "find articles that begin with" is available in my dict plugin (FS#6697), but there seems to be a bug on some players and I'm too lazy at the moment to continue development.
Comment by Isaac (Izzeh) - Saturday, 12 January 2008, 00:27 GMT
Ok after much work I've managed to get it all completed up to the point of converting the xml. Pcre library is installed and i have xmlconv.exe and after running the command ($ ./xmlconv enwiki-20080103-pages-articles.xml enwiki-20080103) I received this error...

16354 [main] xmlconv 2368 _cygtls::handle_exceptions: Error while dumping stat
e (probably corrupted stack)
Segmentation fault (core dumped)

Anyone can help me? despite this thread appearing dead...
Comment by Adam Gashlin (AdamGashlin) - Saturday, 12 January 2008, 06:19 GMT
Reading up a bit, it looks like the converter doesn't work well in cygwin with something as large as enwiki.
Comment by Isaac (Izzeh) - Saturday, 12 January 2008, 06:54 GMT
Any other suggestion? VMWare doesn't exactly allow file sizes that big and I'm on dialup and have no desire to download a whole linux OS..
Comment by Xinlu Huang (polygonal) - Saturday, 12 January 2008, 14:51 GMT
Check out some smaller distros - like puppylinux, feather linux, damn small linux, etc. They usually come in around 100MB - should not wear your dial-up out.
Comment by Frank M. (framo) - Sunday, 03 February 2008, 00:13 GMT
Attached a torrent for the German version of the converted wiki download from January 21st, 2008, size is 1.5 GB.
After downloading the file, please leave it in seeding mode for a while to share it with other rockbox users.
Comment by Frank M. (framo) - Sunday, 03 February 2008, 01:44 GMT
When compiling mww28.diff on my Sansa E280 I got an error with two unkown buttons.
It compiled fine when I replaced in "mmw.c" the two entries BUTTON_SCROLL_UP with BUTTON_SCROLL_BACK, and BUTTON_SCROLL_DOWN with BUTTON_SCROLL_FWD.
I also had to increase the value for "backlight_set_timeout" from 1 to 15 because the LCD kept switching off after 1 second when opening the wiki plugin.
Comment by Frank M. (framo) - Sunday, 03 February 2008, 15:29 GMT
Attached the torrent of the English WIKI dump from January 3rd 2008, size is 4.2 GB.
Hope this helps, again please continue to share the folder after downloading it.
Comment by Fed (Fed) - Sunday, 03 February 2008, 16:58 GMT
Thanks.
Comment by Adam Gashlin (AdamGashlin) - Friday, 08 February 2008, 11:41 GMT
Here's a new patch. I do backlist_set_timeout(0) and I fixed the keymappings (at least it builds with e200 and c200 now).

Thanks for the torrents, framo, but they don't seem to be working (getting a connection refused from the tracker and not seeing anyone over DHT).
Comment by Pascal Briehl (ColdSphinX) - Saturday, 09 February 2008, 00:24 GMT
there is my cowon x5 keymapping:

#elif (CONFIG_KEYPAD == IAUDIO_X5M5_PAD)
#define MWW_MENU BUTTON_POWER
#define MWW_SELECT BUTTON_SELECT
#define MWW_END_RECORD (BUTTON_SELECT|BUTTON_REPEAT)
#define MWW_FWD BUTTON_DOWN
#define MWW_FWD_REPEAT (BUTTON_DOWN|BUTTON_REPEAT)
//#define MWW_FWD2 BUTTON_RC_VOL_UP
//#define MWW_FWD2_REPEAT (BUTTON_RC_VOL_UP|BUTTON_REPEAT)
#define MWW_BACK BUTTON_UP
#define MWW_BACK_REPEAT (BUTTON_UP|BUTTON_REPEAT)
//#define MWW_BACK2 BUTTON_RC_VOL_DOWN
//#define MWW_BACK2_REPEAT (BUTTON_RC_VOL_DOWN|BUTTON_REPEAT)
#define MWW_PREV BUTTON_LEFT
#define MWW_NEXT BUTTON_RIGHT
#define MWW_RECORD BUTTON_REC
#define MWW_RECORD_REPEAT (BUTTON_REC|BUTTON_REPEAT)
Comment by Frank M. (framo) - Saturday, 09 February 2008, 00:35 GMT
Sorry for the outage, the tracker is now back online.
I have also compiled and tested the latest svn with Adam's mww29.diff for Sansa E2x0, Ipod nano G1 and Ipod video 32Mb.
All these torrents (including the wiki compilations) are available on http://fim.ath.cx:6969
Comment by Alex (Sanek) - Saturday, 09 February 2008, 09:59 GMT
Can someone post a screenshot of this please just to give me the idea of what this looks like. Could someone also please post the size of uncompressed English wiki dump, just to know if I can even fit it into my 30GB ipod :P

Thank you in advance!
Comment by Adam Gashlin (AdamGashlin) - Saturday, 09 February 2008, 11:09 GMT
The converter recompresses each article, so you don't need a fully uncompressed dump on your device. The English wiki dump that framo is providing via torrent is 4.4 GB.
I've attached a screenshot from my iPod Color, using Unifont. It is in link select mode, where you scroll and select a link to follow, which is shown highlighted.
Comment by Isaac (Izzeh) - Saturday, 09 February 2008, 14:56 GMT
Ok.. well after memory issues with cygwin, complete distaste for VMWare and its infernal samba file system and finally countless errors upon issue upon errors running feather linux, including it's still complete lack of NTFS writing capability, Fat32's inability to even attempt to hold a large file and basically EVERYTHING short of buying a new hard drive to convert the damn thing.. I officially GIVE UP... sigh I guess some people are just destined not to have wikipedia on their iPods...
Comment by Frank M. (framo) - Saturday, 09 February 2008, 16:58 GMT
No need to give up yet, have a look at the latest messages where you can find torrents for converted wiki dumps and some precompiled rockbox versions.
Comment by Isaac (Izzeh) - Saturday, 09 February 2008, 22:20 GMT
.. I'm on dial up - The torrent is a 13 and a half day download...
Comment by Frank M. (framo) - Saturday, 09 February 2008, 22:35 GMT
Fine with me, I can seed until mid March (which is when I move and will be down for a while).
Comment by Isaac (Izzeh) - Monday, 11 February 2008, 06:14 GMT
Ok I've had the torrent downloading at school directly onto my iPod (I plugged it into a computer that hasnt currently got a monitor, so no one else can access it) I believe it's almost done.. thanks Frank and Adam! this will REALLY come in handy for my history course at uni next year - even if all lecturers hate wikipedia :P
Comment by kimangroo (kimangroo) - Sunday, 17 February 2008, 08:33 GMT
Hey guys, thanks for the latest patch. Having problems patching though, I keep getting this error:

<code>
can't find file to patch at input line 5
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|Index: rockbox/apps/plugins/viewers.config
|===================================================================
|--- rockbox/apps/plugins/viewers.config (revision 16246)
|+++ rockbox/apps/plugins/viewers.config (working copy)
--------------------------
</code>

I can't find the right file to patch and when I skip I get the same error about other files as well.

I'm sure I'm doing something noobishly wrong, can someone walk me throught the patching process for ipod 80G?!

Thanks!
Comment by kimangroo (kimangroo) - Sunday, 17 February 2008, 09:05 GMT
btw I'm using Ubuntu Gutsy if that makes a difference
Comment by Isaac (Izzeh) - Sunday, 17 February 2008, 09:31 GMT
Uhm from what I've found you need to have the .diff in the folder above where you usually put patches. It seems to add everything into a rockbox directory... try moving it up a level...
Comment by kimangroo (kimangroo) - Sunday, 17 February 2008, 19:37 GMT
Thanks Isaac, but don't worry it's just me, I'm dumb.

First of all it turns out I was trying to patch an already compiled version of rockbox (!), and then I had to rename the untarred source code from rockbox2008blabla to just rockbox. Then it all worked smoothly and (touch wood) I finally seem to have wikipedia on my ipod with working links.

Thanks a million people for all the hard work that went into this!
Comment by Rafa? (RvX) - Wednesday, 20 February 2008, 00:33 GMT
The search function needs modification to find articles on the same purpose as in FS#6697. It would be almost perfect then, after implementing that.
Comment by Fed (Fed) - Saturday, 15 March 2008, 16:11 GMT
I think there should be a faster scrolling method like I implemented way back. It was removed for some reason. This way I think it takes too long to just scan an article. Any ideas?
Comment by Isaac (Izzeh) - Monday, 17 March 2008, 10:32 GMT
Nar I don't think its necessary... the text moving that fast already causes my music to drop out if I scroll too long.. I just use the navigate menu to get around... My only suggestion for this mod is to get it put into the daily build already! It's certainly ready for it.. and will encourage more people to download the dump :P
Comment by Fed (Fed) - Monday, 17 March 2008, 13:05 GMT
If you could scroll a whole page at a time, your music wouldn't skip, because it is done in one step rather than many, as with scrolling one line at a time. Also, you could scan an article more easily. I know this because I did it, and it works well. But for some reason this feature was removed. I think it should be put back.

Does anyone else have an opinion on this?
Comment by Adam Gashlin (AdamGashlin) - Wednesday, 19 March 2008, 22:14 GMT
It was removed because I've been posting the patches I use in my own builds, and I happen to prefer single line scrolling. I understand why you might prefer whole page scrolling, though. It seems like it would be a good thing for a configuration option.
Comment by Fed (Fed) - Thursday, 20 March 2008, 00:40 GMT
I agree. Do you know how to make a configuration option?
Comment by Philip Hansen (Philip Hansen) - Friday, 28 March 2008, 22:08 GMT
Hello. I've just discovered this wikiviewer, but I can't compile it for my iPod Video 5.5g 30GB. Everything is compiled except wikiviewer. I keep getting many error messages. I compile in Cygwin for Xp using the files from "mediawikiviewer.tar.bz2" found at the top on this site, and I follow the instructions in the README file. Yet, to no avail. :o( Does anyone have a working copy of this viewer, or help me otherwise?

I think it sounds like a great idea and I plan to make a reference work myself.
Comment by Adam Gashlin (AdamGashlin) - Saturday, 29 March 2008, 00:24 GMT
The latest patch on this page is mww29.diff.bz2, and there are two torrents right above that for English and German converted dumps. I think the latest converter was converter011.tar.bz2 (which you'll only need for converting something new).
There is a "HOWTO wikipedia.html" for conversion halfway up the page with up to date directions.
Comment by Philip Hansen (Philip Hansen) - Sunday, 30 March 2008, 10:09 GMT
To anyone who might benefit from my experience of compiling RB to include mww: Get mww29.diff. Download the patch utility and install (download from here, http://gnuwin32.sourceforge.net/packages/patch.htm and choose the "complete package, except sources"). Create a folder named "mww" and put the diff file and patch.exe in there.

I use Cygwin to patch. In Cygwin go to the mww folder and type "patch < mww29.diff". Patch now creates the needed mww files in this folder.

In my case it asked me where the viewers.config and SUBDIRS files were. So I patched these manually by doing this:
Open rockbox/apps/plugins/viewers.config in TextEdit (I'm using a mac) or another textedit programme, and insert this line "wwi,viewers,mww,-" after "wav,viewers/test_codec,-". Save and exit.
Open rockbox/apps/plugins/SUBDIRS and insert this line "mww" after "reversi". Save and exit.
Open rockbox/apps/plugins/CATEGORIES and insert this line "mww,viewers" after "mpegplayer,viewers". Save and exit.

Next, move the "mww" folder to rockbox/apps/plugins and start compiling.

I used this procedure and it finally worked! And I was very happy :-) Originally I mixed up the old mww files from "mediawikiviewer.tar.bz2" and that messed everything up. Finally I got the picture. Trial and error.

Now I just have to figure out how to convert dumps, because I can't compile the converter in Cygwin. Gonna try Linux on my mac and see if that helps.
Comment by Adam Gashlin (AdamGashlin) - Wednesday, 02 April 2008, 05:17 GMT
In case anyone is curious about where I'd like to see the project go, here's a suggested todo list I wrote up for someone.
Comment by Andrew Slavozchikz (ytifle) - Wednesday, 23 April 2008, 19:37 GMT
A program to manage the dumps, syncs and updates would be VERY useful (like AvantGO for the pdas, in that style.

Like, i.e., call the sub revision in the wiki of one article, and is newer, download and copy the dump to the plugin mww folder
Comment by Delyan Kratunov (archivator) - Wednesday, 23 April 2008, 20:08 GMT
The plugin requires a sync after the recent change in the plugin menu playback control. I lack the development experience to do the change, though.
Comment by Alexander Papst (DerPapst) - Friday, 25 April 2008, 18:35 GMT
change every call of playback_control(rb); to playback_control(rb, NULL);
Comment by Adam Gashlin (AdamGashlin) - Friday, 25 April 2008, 19:24 GMT
Something else needs changing, it compiles alright but I don't know where the binary is going.
Comment by Frank Gevaerts (fg) - Friday, 25 April 2008, 20:38 GMT
Fixed to make it compile on recent (r17248) svn. No other changes, and I don't know if it works
Comment by Adam Gashlin (AdamGashlin) - Saturday, 26 April 2008, 00:27 GMT
Works fine on my iPod, thanks.
Comment by Delyan Kratunov (archivator) - Sunday, 18 May 2008, 10:17 GMT
After a recent update, mww yields tons of redefinition errors. A sync would be much appreciated.
Comment by Adam Gashlin (AdamGashlin) - Monday, 19 May 2008, 17:55 GMT
*Tons* of redefinition errors?

Sync'd.
Comment by Thomas Martitz (kugel.) - Saturday, 24 May 2008, 18:43 GMT
Delian, you have probably applied the patch twice, so that you have the files which get created with the patch twice (twice in one file). Get a new source, or remove the patch and delete the files created by this patch manually.
Comment by Isaac (Izzeh) - Monday, 21 July 2008, 13:35 GMT
Getting an error on hunk #1 or something to do with subdir when trying to patch. Need resync?
Comment by Marcin Łukasik (Moarc) - Sunday, 21 September 2008, 14:54 GMT
When I'm trying to compile Rockbox with this patch, I receive compilation error:
MAKE in mww
CC mww.c
mww.c: In function 'set_article_offset':
mww.c:386: warning: implicit declaration of function 'printf'
mww.c: In function 'viewer_init':
mww.c:1103: error: too many arguments to function 'rb->splash'
make[3]: *** [/home/Moarc/rockbox/build/apps/plugins/mww/mww.o] Error 1
make[2]: *** [mww] Error 2
make[1]: *** [rocks] Error 2
make: *** [build] Error 2

How can I fix that? Will you fix that in new version of plugin?
Comment by Thomas Schott (scotty) - Sunday, 21 September 2008, 20:57 GMT
Moarc: sync to SVN ;-)

The wwi viewer doesn't work in simulators because of:
> dlopen(archos/.rockbox/rocks/viewers/mww.rock): archos/.rockbox/rocks/viewers/mww.rock: undefined symbol: sim_write
at runtime. I couldn't find the sim_write call in the mmw sources, seems to be in an included file.

But anyway, it works on my iPod and I love having wikipedia in my pocket :-) THX!
Comment by Xinlu Huang (polygonal) - Saturday, 22 November 2008, 20:05 GMT
The recent revamp of the make system (r19146) broke this patch; make fails with
make: *** No rule to make target `/data/rock/rockbox/apps/plugins/mww/mww.make'. Stop.

I know next to nothing about makefiles etc. (always avoid that like plague... ;) ). Can someone who knows about makefile fix this (looks like a quick fix for a makefile guru)? Many thanks!
Comment by Thomas Schott (scotty) - Saturday, 29 November 2008, 10:02 GMT
Sync to SVN.
Comment by Björn Stenberg (zagor) - Monday, 01 December 2008, 13:11 GMT
I'm prepared to nurse this into SVN. However there is a couple of things that needs fixing before it is ready for prime time:

1) MWW is a cryptic name. We don't like cryptic names. Please rename the plugin wikiviewer or wikireader or some such. The Mediawiki reference is not relevant since the plugin is not reading the mediawiki file format anyway, only the converter is.

2) Add keymaps for all targets in http://build.rockbox.org, or at least add #ifdefs in apps/plugins/SUBDIRS so the plugin isn't compiled on targets where it will fail.

3) Clean up the code:
3a) Remove all #define read rb->read and such. simprint too.
3b) The code is too dense. Keep one statement per line, sprinkle with whitespace and split long lines at 80 columns.
3c) Indent all code with spaces, not tabs.
3d) Remove unused code.
3f) Replace // comments with /* */
3g) Change #include "../lib/.." to simply #include "lib/..."

4) Include the converter source code in the patch, since it is essential to the plugin. Put it in a subdirectory under tools.

5) Make sure each sourcefile has a rockbox header (except the imported gzip source files).
Comment by Adam Gashlin (AdamGashlin) - Monday, 01 December 2008, 18:33 GMT
Is anyone interested in tackling this? As much of the viewer is my code I'd probably have an easier time of it, but if someone else volunteers it'll probably get done sooner (can't even guess when I'd get to it).
Comment by Matthias Larisch (Matze88m) - Monday, 01 December 2008, 19:56 GMT
I already started doing some parts of Bjorns list. I will add a patch today or tomorrow that covers some points. Rest of work is up to another one :-)
Comment by Xinlu Huang (polygonal) - Monday, 01 December 2008, 23:34 GMT
I can do the rest within two weeks or so (after my rounds of final exams) - if no one gets to that before.
Comment by Matthias Larisch (Matze88m) - Tuesday, 02 December 2008, 12:50 GMT
I forgot to ask how we should call this plugin? I would stick to wikiviewer as suggested by Björn but as it mainly is the work of Adam he has to decide the name :-)

Adam, what do you think?
Comment by Matthias Larisch (Matze88m) - Tuesday, 02 December 2008, 18:43 GMT
Here it is. I renamed everything to wikiviewer.
My changes:
-rename
-some cleanup (defines, more white spaces, line breaks, spaces instead of tabs, removed unused code, replaced all comments, changed includes)
-included converter code (with new makefile, some cleanup)
-added some rockbox headers (maybe not really correct - please check me)

Compiled code ran on my H320 as the original (this should be like that because I did not change anything)
Comment by Matthias Larisch (Matze88m) - Tuesday, 02 December 2008, 19:03 GMT
Sorry for the incomplete diff. svn under cygwin is really crap for developing. Seems that I should change to linux soon :-)
Here is a diff generated by TortoiseSVN, not perfect but works:
Comment by Adam Gashlin (AdamGashlin) - Tuesday, 02 December 2008, 21:23 GMT
Yeah wikiviewer sounds like a good name, glad to see this getting somewhere finally.
Comment by Edward Hyer (ejhyer) - Wednesday, 10 December 2008, 04:54 GMT
Excited by this project!! Got patch, re-compiled, so far so good.
Could not compile converter (Ubuntu Hardy):
jo:~/rockbox_ipod55/rockbox_source/rockbox/tools/converter$ make
gcc -o xmlconv -O3 xmlconv.c -lpcre -lz -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE
xmlconv.c:33:18: error: pcre.h: No such file or directory
Comment by Marcin Łukasik (Moarc) - Wednesday, 10 December 2008, 05:03 GMT
ejhyer, you need to install pcre-dev or libpcre-dev. (I don't know how it's named in Ubuntu)
Comment by Edward Hyer (ejhyer) - Wednesday, 10 December 2008, 05:44 GMT
OK, got the libpcre, was able to compile the converter. Now it's cranking, and I'll find out in the morning what I have.
Comment by Edward Hyer (ejhyer) - Thursday, 11 December 2008, 03:52 GMT
OK, one source of confusion. If I have the new stardict plugin, and I have used the converter, do I actually need the wikiviewer plugin?
Comment by Edward Hyer (ejhyer) - Thursday, 11 December 2008, 04:14 GMT
Hmmm... if I *don't* use the stardict plugin, where do I put the converted wikipedia files?
Comment by Adam Gashlin (AdamGashlin) - Thursday, 11 December 2008, 04:24 GMT
I don't know how the stardict plugin works, but the wikipedia viewer compresses the files, and the stardict plugin probably doesn't support it (nor the indexing system).
To use the files, put the generated .wwi and .wwa files in the same directory on your device. The viewer should be set up as a viewer for .wwi, just open that file and you should be able to use it
Comment by Edward Hyer (ejhyer) - Thursday, 11 December 2008, 05:56 GMT
Success! I got it working! Very, very happy. Now I'll complain: the text entry system (ipod video) is just terrible. Can I modify it?
Anyway, thanks everyone for your work on this and help with installing, I'm going to play a bit now.
Comment by Adam Gashlin (AdamGashlin) - Thursday, 11 December 2008, 06:28 GMT
Sure you can modify it, you have the source. The index tree should lend itself naturally to a nicer system that searches as you're typing, with the bonus of autocompletion, just scroll among the possible letters for each place.
Comment by Edward Hyer (ejhyer) - Thursday, 11 December 2008, 07:39 GMT
OK, here are my observations after playing a little. Most of these are Rockbox-in-general, but some maybe can be fixed in wikiviewer, and hopefully some I'm just doing it wrong, and someone can set me straight.
1) On startup, I get the keypad to enter a search term. Why can't I go directly to the menu, where I can start at a bookmark or something?
2) Search is case-sensitive. Must this be?
3) The only font I could find which was unicode-capable was the gnu-unifont. Are there any other unicode-capable fonts ported to Rockbox?

OK, that's all I got for right now. Thanks again for this super-cool piece of hackery, I'm having lots of fun.
Comment by Adam Gashlin (AdamGashlin) - Thursday, 11 December 2008, 15:55 GMT
1) You have to "save history" in the menu if you want to return to a session in progress next time
2) I don't think search is case sensitive
3) not that I know of, but I haven't been paying attention for a while
Comment by Timo Horstschäfer (x1jmp) - Thursday, 01 January 2009, 23:15 GMT
I had a look at this plugin again and really liked it! Great work!

I used to have my own stardict plugin for rockbox to read wikipedia and of course stardict dictionaries on my player.
To keep these dictionaries with wikiviewer I just wrote a hackish stardict to wikiviewer converter. It's name is stardictconv and does the same job as xmlconv does but for stardicts (get them at http://stardict.sf.net).
It contains a lot of files from the original stardict, so I put them in an archive instead of a single patch (and the stardict files are GPLv3 so I'm not sure if they fit into RB anyways...).

If you're interested I could put an archive of stardicts in wikiviewer format online.
Comment by Robert Kukla (roolku) - Sunday, 18 January 2009, 14:23 GMT
- resync
- don't include bbfuncs.c, but compile separately and link it
- buttons for x5, c200 and mr100
Comment by Seth (froggyman) - Wednesday, 04 February 2009, 22:21 GMT
does any one have a pre patched version of this(or maybe even compiled) for the iPod Video 5.5G 30GB, because i cant get the .patch patcher program to work at all? If some one has a pre-patched version i can compile it!
If no-one does though, i understand, I know its kind of "beggish", =).
Comment by Adam Gashlin (AdamGashlin) - Wednesday, 04 February 2009, 23:07 GMT
Here it is as a .tar.gz of the changed and new files.
Comment by Seth (froggyman) - Thursday, 05 February 2009, 03:22 GMT
wait, what is that, is it just the wikipedia convertor and wikipedia plugin for Rockox, just want to make sure.
Comment by Adam Gashlin (AdamGashlin) - Thursday, 05 February 2009, 08:07 GMT
Yes, you would extract that into the clean rockbox source directory, overwriting existing files and creating new ones.
Comment by Seth (froggyman) - Thursday, 05 February 2009, 13:10 GMT
followed by compiling!
Thanks abunch for doing that, i have wanted wikipedia on my ipod for like ever. Now i have to try and figure out how to download like a 2Gb+ file with dial-up, hmm..... maybe my neighbors will let me download it on their computer(with road runner turbo).... i'lll manage to get it done!
Comment by Seth (froggyman) - Thursday, 26 March 2009, 12:20 GMT
ok, i have another question with this patch, for i finally took the time to compile it and load it to my iPod. SO now, my iPod can view the wiki files, but i dont know how to convert the wikipedia dumps into the correct .wwi and .wwa formats. Is there a conversion program out there OR maybe instructions on how to convert the wikipedia dumps OR is there a dump pre converted that can be downloaded?
Comment by Adam Gashlin (AdamGashlin) - Saturday, 28 March 2009, 06:05 GMT
I think this was the latest converter: http://www.rockbox.org/tracker/task/4755#comment13476
Building the converter should just be a matter of running the "compile" script to build the xml processor (xmlconv) and "Make.sh" to build the index builder (btcreate).
For converting, the first step is:
bunzip2 -c enwiki-XXX-pages-articles.xml.bz2 | ./xmlconv - wikipedia
assuming "enwiki-XXX-pages-articles.xml.bz2" is the dump, and that you want the resulting files to be called "wikipedia". That will build the wikipediaN.wwa files (containing compressed article text), wikipedia.wwt (titles and indexes into the article data) and wikipedia.wwr (redirects).
The second step is:
./btcreate wikipedia.wwt wikipedia.wwr wikipedia.wwi
Which creates the index file wikipedia.wwi. Then just copy the .wwi and .wwa files to your device.

Sorry for editing this so many times.
Comment by Seth (froggyman) - Sunday, 29 March 2009, 23:46 GMT
ok thanks, but wouldnt it be more simple for everyone if we just put a pre converted wiki dump up on like megaupload or rapidshare? i mean, it would surely save a lot of processing power and quicker than converting it yourself (especialy for those of us with a slower computer)?

IDK, just my thought.

But thanks anyways for giving me those instructions on how to convert it.
Comment by Jerry Chapman (syn4pse) - Wednesday, 01 July 2009, 16:14 GMT
Needs a resync
Comment by Jonathan (JesusFreak) - Sunday, 05 July 2009, 20:35 GMT
In case anyone is interested, I'd be willing to upload my recent Wikipedia dump (7/02/09) in wwa and wwi format if someone can suggest a good filehost.
Comment by Cristián Quezada (CQK) - Monday, 06 July 2009, 02:43 GMT
I don't know any filehost that allows files that big, but if you consider making a torrent I can seed it for some time.
Comment by Jonathan (JesusFreak) - Monday, 06 July 2009, 02:48 GMT
Okay. I'll try to get to it soon, and maybe send you a pm with the torrent link. Don't forget though, the max size of each file is 1 gig, so maybe I could do it still.
Comment by Steve Bavin (pondlife) - Monday, 06 July 2009, 07:09 GMT
Could you not post the link here? More people = better torrenting...
Comment by Jonathan (JesusFreak) - Monday, 06 July 2009, 16:00 GMT
Sure, but I need a suggestion for a site. I don't do much file hosting, much less torrents.
Comment by Adam Gashlin (AdamGashlin) - Monday, 06 July 2009, 18:57 GMT
I don't know of any reason not to use a major site like The Pirate Bay, unless you have concerns over guilt by association. Maybe try the various fun stuff listed here:
http://azureuswiki.com/index.php/Legal_torrent_sites
but I've used tpb for wiki stuff in the past.
Comment by Maurus Cuelenaere (mcuelenaere) - Tuesday, 07 July 2009, 12:17 GMT
This is a cleaned up version (removed TABs, make lots of functions static, added (basic) keymap for touchscreen targets and remove unneeded code) + some optimisations (html entities are now binary searched).
Comment by Jonathan (JesusFreak) - Tuesday, 07 July 2009, 19:04 GMT
Tell me if this worked. I have never created a torrent before. I'll probably be seeding all day. If someone else could download it and do it right, it would be much appreciated.
Comment by Adam Gashlin (AdamGashlin) - Tuesday, 07 July 2009, 23:04 GMT
It seems to be working fine, thanks, I'll keep it seeded, too. It would help if in the future you include the dump date and language info in the torrent, preferably in the directory name or a comment.
Comment by Jonathan (JesusFreak) - Tuesday, 07 July 2009, 23:18 GMT
Ok, now I know. Yeah, it would have been good to include that there. I did say it on this page, but not everyone will see that. I'm just glad to finally contribute something to Rockbox. :)
Comment by Frank M. (framo) - Friday, 10 July 2009, 19:01 GMT
Hi Jonathan, the torrent is stuck at 84% - could you kindly seed a bit longer?
Thanks, Frank
Comment by Jonathan (JesusFreak) - Saturday, 11 July 2009, 16:48 GMT
Yeah, sorry. Whenever I shut it down at night, it stops, and I often forget to turn it back on the next day. It's up again.
Comment by Cristián Quezada (CQK) - Saturday, 11 July 2009, 23:10 GMT
The new Wikipedia dump works great with the updated patch. Thank you Jonathan and Maurus! I was out of town so I couldn't seed, but now I'm back.
Comment by Dave Chapman (linuxstb) - Saturday, 11 July 2009, 23:34 GMT
A few comments if the intention is to commit this patch:

1) tools/ is probably not the best place for the converter program (it's intended for tools required to build Rockbox). Maybe utils/ would be better?

2) "converter" is too general a name - how about something like "wikiconv" ?

3) Lots of files are lacking names of authors/copyright holders. I don't think they can be committed to Rockbox like that.

4) Documentation refers to "freqmod" - can that be changed to a real name?

5) Some files still contain the old version of the Rockbox license text (before the change to explicitly say "v2 of the GPL or later"). Other files don't include that text at all.
Comment by Maurus Cuelenaere (mcuelenaere) - Sunday, 12 July 2009, 02:51 GMT
I suppose in the end, the intention is to commit this (it still needs reworking).

1) I agree
2) sounds better
3) yes, I also wondered about that; there's lots of anonymous/not credited code
5) should be doable
Comment by Gman (Thecoolgman) - Monday, 13 July 2009, 20:50 GMT
I have problems compiling this on my 5th Gen.
Comment by Gman (Thecoolgman) - Wednesday, 15 July 2009, 22:07 GMT
It seems it needs a resync?
Comment by Nick G (gersto) - Wednesday, 05 August 2009, 23:50 GMT
ok i'm trying compiling Rockbox for the first time since i'd like to try some of these patches out there. I've read through how to compile and successfully compiled the standard SVN from today without error. Since i'm new to all this, what is the base file from here that i need to start with regarding the wikipedia patch. Do i need to grab every file from the 1st posted and go in sequence?
Comment by Nick G (gersto) - Thursday, 06 August 2009, 03:06 GMT
well actually i'm reading the diff and i see this:
diff --git a/apps/plugins/CATEGORIES b/apps/plugins/CATEGORIES
index 8e4bb7d..d8235c1 100644
--- a/apps/plugins/CATEGORIES
+++ b/apps/plugins/CATEGORIES

i noticed there is no a or b directory. Could that be why? Please excuse my noobishness
.
Comment by Nick G (gersto) - Thursday, 06 August 2009, 04:53 GMT
sorry about the double-post.

I decided to give it a shot and remove references to the directories a/ and b/ this is what i got at compiling the 8/5 SVN with the patch:

LD lua.rock
make: *** No rule to make target `/root/rockbox/ipod/liplayback_control.h', needed by `/root/rockbox/ipod/apps/plugins/wikiviewer/wikiviewer.o'. Stop
obviously i'm missing something...
Comment by Thomas Martitz (kugel.) - Thursday, 06 August 2009, 11:19 GMT
a/ and b/ comes from git (a version control system) when creating patches with it. You need to run "patch -p1 < file.patch" to strip a/ and b/ automatically.

Comment by Nick G (gersto) - Thursday, 06 August 2009, 16:50 GMT
ok did that but now i'm getting something different:

CC apps/plugins/wikiviewer/wikiviewer.c
/home/nicholas/Projects/rockbox/apps/plugins/wikiviewer/wikiviewer.c: In function ‘render’:
/home/nicholas/Projects/rockbox/apps/plugins/wikiviewer/wikiviewer.c:621: warning: unused variable ‘colorem’
/home/nicholas/Projects/rockbox/apps/plugins/wikiviewer/wikiviewer.c: In function ‘plugin_start’:
/home/nicholas/Projects/rockbox/apps/plugins/wikiviewer/wikiviewer.c:1765: error: ‘struct plugin_api’ has no member named ‘strncpy’
make: *** [/home/nicholas/Projects/rockbox/ipod/apps/plugins/wikiviewer/wikiviewer.o] Error 1

btw i'm building the ipod 5g (30gb) variation
Comment by Maurus Cuelenaere (mcuelenaere) - Thursday, 06 August 2009, 17:25 GMT
The rb->strncpy() error is due to the replacement of strncpy() with strlcpy() in core, you can either change it to use the pluginlib one or make it use rb->strlcpy().
Comment by Vergil Weatherford (theverge) - Sunday, 09 August 2009, 18:34 GMT
Maurus and Nick G,

I'm compiling the latest wikiviewer patch (posted by Maurus on July 7) for the first time, with today's rockbox build. I also ran into the 'strncpy' error, saw the comments here, and changed rb->strncpy() to rb->strlcpy() in wikiviewer.c and recompiled.

upon recompiling, I ran into another error:

In function `render':
wikiviewer.c:(.text+0x19a0): undefined reference to `LCD_RGBPACK'

any ideas what I might do to fix this error?

Comment by Maurus Cuelenaere (mcuelenaere) - Sunday, 09 August 2009, 23:33 GMT
LCD_RGBPACK is a macro which is defined in lcd.h I think, so adding #include "lcd.h" to wikiviewer.c should do the trick.

I wonder though how I didn't get that error, what target are you compiling for?
Comment by Vergil Weatherford (theverge) - Monday, 10 August 2009, 00:25 GMT
ipod mini 2nd generation, with an upgraded (extended life) battery and a 16gb flash card instead of the micro-drive.

6gb=wikipedia
10gb=music

sweet!
Comment by Vergil Weatherford (theverge) - Monday, 10 August 2009, 00:44 GMT
Maurus,

I added #include "lcd.h" to wikiviewer.c and then recompiled, but I still get the same error. Thanks for taking a look.
Comment by Jeff Wood (woodi) - Monday, 10 August 2009, 02:24 GMT
I am trying to compile the latest version of this viewer for an ipod 5g and keep running into troubles. Make keeps failing with the error:
make: ***No rule to make target '/home/Jeff/rockbox/apps/plugins/lib/oldmenuapi.h' needed by '/home/jeff/rockbox/wiki/apps/plugins/wikiviewer/wikiviewer.o'.
Stop.
Any idea's why this is happening something simple and stupid i'm missing that's causing it? I'll admit to having pretty much no idea what i'm doing. I ran enough linux software in the past to have no trouble getting the enivronment setup and making a basic build for my ipod but as soon as the compiler starts throwing errors I get lost.
Comment by Nick G (gersto) - Monday, 10 August 2009, 03:37 GMT
Jeff - Look at Thomas's post just below mine on august 6th ^_^. Changed rb->strncpy() to rb->strlcpy() in wikiviewer.c, like Maurus says, and patch it with the command Thomas mentioned and you'll be good to go to compile. That worked for me and it compiled fine :)
Comment by Jeff Wood (woodi) - Monday, 10 August 2009, 16:25 GMT
Nick - I tried changing rb->strncpy() this morning before heading to work but still got the same error. I think when I get home i'm gonna grab a fresh copy of the source code, patch as instructed above, and make the rb->strncpy() change. Then I'll try a fresh compile and see where I get.
Comment by Jeff Wood (woodi) - Monday, 10 August 2009, 22:08 GMT
Started from scratch and it worked this time. Thanks for the help. In case others who don't know what they're doing have this same problem and can't manage to read like me. I did:

1. Got a fresh copy of the source from svn (the one i screwed up patching originally I couldn't seem to get to work)
2. patched using the wikiviewer.diff above using patch -p1 > wikiviewer.diff
3. edited apps/plugins/wikiviewer/wikiviewer.c and changed rb->strncpy() to rb->strlcpy()
3. compile rockbox as usual

Played with it and it's definitely very cool to have on the ipod. Now I just gotta figure out a better keymap or someway to adjust that so it's not impossible to type in what I'm searching for. Great plugin thanks all.
Comment by Rosso Maltese (asettico) - Monday, 10 August 2009, 22:21 GMT
The right command is: patch -p1 < wikiviewer.diff ("less than" instead of "greater than").

As general note: since I don't make a normal development activity, I wrote a script that first update my local source tree, then applies my selected patches, then build, and finally remove all the patches (if you apply a patch with "patch < file", you can remove it with "patch -R < file").
Comment by Vergil Weatherford (theverge) - Monday, 10 August 2009, 22:25 GMT
Still having troubles with the LCD_RGBPACK error on compile. Does anyone have any suggestions? On my ipod mini I don't have a color screen, so is there anything wrong with commenting out that section in wikiviewer.c? I'm not much of a programmer, so I am not sure if that would have a ripple-effect the rest of the compile.

Would love to get this up and running! Thanks in advance for the help.
Comment by TruthTaco (TruthTaco) - Monday, 10 August 2009, 23:31 GMT
I've tried to install this patch several times, done the things suggested above, and i still get these errors http://pastebin.com/m46ec1a5f
Any suggestions?
Comment by G.P. (dracayr) - Tuesday, 11 August 2009, 08:09 GMT
synced (changed strncpy to strlcpy), and added sansa fuze support. Record doesn't work on the Fuze, because all the Buttons were already used. Any Suggestions on how to record on the Fuze? (I'm new at Rockbox)

TruthTaco, it looks like you are building for an unsupported target (or for the sansa fuze, which is now supported)
Comment by TruthTaco (TruthTaco) - Tuesday, 11 August 2009, 17:47 GMT
Thanks fuze support was what i needed. Record function has not been added to the fuze yet.
I still get a few errors http://pastebin.com/m1baf2f70 but i think im just going to get a friend to compile it for me.
Comment by Vergil Weatherford (theverge) - Friday, 14 August 2009, 01:13 GMT
So trying again today, I started from scratch with the latest patch from TruthTaco. I added "#include "lcd.h" to wikiviewer.c, and compiled. Unfortunately, I ran into the same error as before: (undefined reference to `LCD_RGBPACK'). Is this because I'm trying to build for an unsupported target (ipod mini 2nd gen)? Is there any way to work around this?
Comment by Gman (Thecoolgman) - Monday, 17 August 2009, 23:48 GMT
I get this error when compiling it on my 5G, it is something about a, "wikiviewer.make" It stops it from compiling. Why is this?
Comment by Jerry Chapman (syn4pse) - Tuesday, 18 August 2009, 00:09 GMT
It looks like the last part of wikiviewer.c got clipped from the patch made on Tuesday, 07 July 2009, 07:17 GMT-5 by Maurus Cuelenaere (mcuelenaere). If you scroll back and get a copy of that patch, you will see what I mean. I was getting the same error as TruthTaco until I copied and pasted the last few lines of Maurus' old patch.

It is in the last function, starts with:
enum plugin_status plugin_start(const void* file)

Just copy that function over from the old patch to wikiviewer.c Don't forget to change the one instance of rb->strncpy() to rb->strlcpy()
Comment by Jerry Chapman (syn4pse) - Tuesday, 18 August 2009, 00:52 GMT
Ok, I prepared a patch so you don't have to copy and paste. This compiles for me cleanly now on my Ipod Video.
Comment by Gman (Thecoolgman) - Tuesday, 18 August 2009, 01:16 GMT
Thanks man!
Comment by Marianne Arnold (pixelma) - Tuesday, 01 September 2009, 00:16 GMT
> Comment by Dave Chapman (linuxstb) - Saturday, 11 July 2009, 23:34 GMT
> 4) Documentation refers to "freqmod" - can that be changed to a real name?

I don't know if the problem is getting the real name or about actually doing the work to change it. If it's the former: freqmod's real name is Frederik Vestre and he's already in the credits for a Norwegian update, see http://svn.rockbox.org/viewvc.cgi?view=rev&revision=9930 - and later contributed a few more small patches. Just when I wrote this, I could still see him around in IRC - #rockbox and a some other channels, I'm quite sure it's him so if you really want to contact him, there would be a way...
Comment by Seth (froggyman) - Wednesday, 09 September 2009, 21:30 GMT
has anybody here tried converting a copy of simple Wikipedia or Wiktionary with the converters. I would, but I dont know how to use the converter program (havent found any instructions on how to use it if someone could give exact instructions I wouldnt have a problem converting it, and uploading it somewhere).

The reason I say this is because both of those would be smaller, and the simple wikipedia can be useful in a situation where you have a small player/ don't have a lot of space you want to give up and for times when you want to just get a more generalized feel of the article/topic.

The converter should work, as they both use the same wiki type and are both in the XML format which the standard Wikipedia uses.
Comment by Gman (Thecoolgman) - Thursday, 10 September 2009, 05:13 GMT
I would too like to know how to work the converter :(
Comment by Robin Bertram (berti) - Friday, 11 September 2009, 12:31 GMT
in the converter folder, there is a README file that helped me compiling and converting

I had to install zlib and pcre first to be able to compile the converter succesfully, but perhaps that's not neccesary... you'll see
I also compiled the stardict converter that can be found here: http://www.rockbox.org/tracker/task/4755#comment27464

now the new dict plugin is redundant for me... although I like it's search method better

PS: I can upload dewiki and some converted dictionaries, if needed
Comment by MKL (Excalibur) - Wednesday, 16 September 2009, 16:18 GMT
I can't patch the dict file, do I need to have the source code or does the dict.rock file work with cygwin?

Might it be possible to upload a finished file (dict.rock) and the dewiki or frwiki dict files

(I would prefer some ready-to-copy-on-device-files, because i can't patch
Comment by Robin Bertram (berti) - Sunday, 18 October 2009, 22:57 GMT
I uploaded the converted wikipedia dump (2009-08-10) (about 2.1GB)

here the netload and the rapidshare links
http://netfolder.in/folder.php?folder_id=X5og6rk
http://rapidshare.com/files/294818716/dewiki-20090810.txt
Comment by Robin Bertram (berti) - Sunday, 18 October 2009, 23:14 GMT Comment by Robin Bertram (berti) - Sunday, 18 October 2009, 23:15 GMT
oh sry... forgot to mention

it's only dewiki
Comment by Seth (froggyman) - Sunday, 08 November 2009, 15:20 GMT
i know earlier on in this FS, it was mentioned that it is not case sensitive when you search for things.

If you type in 'GUI' on rockbox's wikipedia it brings you to a food, because the case is ignored. Whereas you would have wanted it to go to "Graphical User Interface". Thats just one example, I am sure there are more examples.

One thing I thought about how to solve this is to first search for the article in the case that it was entered. So it would search for GUI first, and then if nothing appears remove the case sensitivity and search for the article.
Comment by Adam Gashlin (AdamGashlin) - Sunday, 08 November 2009, 18:16 GMT
Well, it is supposed to do that. On 07 November 2006 I said: "Text sensitive matching is now attempted before insensitive matching, if there is an exact match we should use that". It may be that the search is messed up, but what you suggest was the intention.
Comment by Seth (froggyman) - Sunday, 08 November 2009, 22:23 GMT
hmmm....

I just tried it again to verify what I had previously reported. It did the same thing when I searched for "GUI", it brought me to some korean dish/food. Then one i went to "find similar article" in the menu, it brought me to the Virtual Keyboard, displaying "Gui", even though GUI was entered. On Wikipedia, GUI goes to: http://en.wikipedia.org/wiki/GUI and "Gui" goes to http://en.wikipedia.org/wiki/Gui .
Can you or someone else verify this?

Any ideas why that might be happening then?
Comment by Maurus Cuelenaere (mcuelenaere) - Monday, 04 January 2010, 01:03 GMT
I've made some changes to this in my git tree ( http://gitorious.org/mcuelenaere-rockbox/mcuelenaere-rockbox/commits/wiki_viewer ).

I'm planning to clean the patch up, add touchscreen support and perhaps add LZMA support.
Comment by Robin Bertram (berti) - Friday, 19 February 2010, 15:59 GMT
here is a new version of dewiki

I only seed as long as my Notebook is on
Comment by Alex (Darkknight512) - Saturday, 27 February 2010, 23:59 GMT
Does anyone have a pre-compiled version of the Stardict -> Wiki converter?
Comment by David Lorch (dlorch) - Monday, 29 March 2010, 23:15 GMT
This plugin is amazing. I hope that it will eventually make it into SVN!

To everyone who's interested: I have uploaded a conversion (.wwi and .wwa files, 7.4GB) of a March 2010 English Wikipedia dump to RapidShare.
The archive consists of 37 parts, 200MB each, the links to which are given in the attached file. The easiest way to download all of them is to copy&paste the links into jDownloader (www.jdownloader.org).

The source XML file was: http://download.wikimedia.org/enwiki/20100312/enwiki-20100312-pages-articles.xml.bz2
Comment by Robin Bertram (berti) - Saturday, 19 June 2010, 14:23 GMT
thanks mcuelenaere for your work in your git tree
I synced your changes and removed an ifdef because ONDA_VX747_PAD is not defined in svn
Comment by Robin Bertram (berti) - Saturday, 19 June 2010, 16:41 GMT
ONDA_VX747_PAD is ONDAVX747_PAD in svn
I think I should change that instead of removing it
Comment by Rosso Maltese (asettico) - Thursday, 08 July 2010, 10:50 GMT
Sync against r27345.
Comment by Tom (mind_the_gap) - Monday, 12 July 2010, 18:21 GMT
Thanks Rosso for the updated patch. I creaderd a new rockbox github project and in one branch lives that patch:
http://github.com/tomka/rockbox/tree/wikiviewer
Comment by Matteo Italia (MItaly) - Saturday, 28 August 2010, 15:45 GMT
The last patch gives problems with the current build, since PLUGINLIB_EXIT_INIT does no longer exist, replaced with some new machinery that involves the new plugin_crt0.c. If I understood correctly what it did, I think it could be removed without much fear, but since I'm quite new to RockBox development I'd like to have confirmation from someone who knows before posting a new patch.
Comment by Tom (mind_the_gap) - Friday, 08 October 2010, 16:08 GMT
No rockbox guru here as well, but besides the suggestions you made Matteo I think some more USB-Event and exit related things have to be changed. With git it is commit 950ac28c3ff7e339bd388314ddb0a7f12bc0c74 which introduced plugin.crt0.c. With viewing at the changes of that commit I saw a couple of things that were changed in other plugins with similar structure. Well, I've added them and my changes are available on github repo from two posts ago. I try to veryfy that these changes are correct.
Comment by Rosso Maltese (asettico) - Friday, 12 November 2010, 15:38 GMT
Synced against r28563.
Comment by Haim (haim) - Saturday, 12 March 2011, 22:16 GMT
Hi, It will be possible to add support for RTL languages​​? (Like Hebrew and Arabic ) so the text will be aligned to the right instead of left
If so, it will help me a lot,
thanks, Haim.
Comment by sideral (sideral) - Monday, 06 June 2011, 23:27 GMT
Looks like we actually do know the original contributor's real name – to be confirmed.
Comment by Justin Hannigan (Chronon) - Wednesday, 15 June 2011, 19:57 GMT
Haim, RTL support is in the core but isn't available to plugins at the moment.
Comment by Haim (haim) - Thursday, 16 June 2011, 21:02 GMT
Justin, Yes, I know, I tried to change the code of this plugin to add RTL support for this plugin, but I couldn't do it, so I asked if anyone knows the code of this plugin can do it for me (if it's not too hard...)
thanks again.
Comment by Christoph (Christoph) - Sunday, 01 April 2012, 12:39 GMT
I just added keymappings for the Sansa Fuze+. Find them in the attached patch (if anybody is interested). It is based upon 4755-plugins-Wikiviewer.0.patch.

Thanks for providing this plugin. I have been using it on my Sansa e250 and am now using it on my Sansa Fuze+ and it is working perfectly fine on both. It might be worthwile to consider adding this to official rockbox releases.
Comment by Haim (haim) - Sunday, 06 May 2012, 02:02 GMT
Not working on 88aeef9 build
Comment by Avi (ikeboy) - Tuesday, 19 March 2013, 17:47 GMT
If anyone is interested in the converted dumps , I have them and am trying to uploading them now .
The wwi file for February 4 ,2013 (466.8MB) is at http://www.freefilehosting.net/wikipedia .
When I upload the wwa files I will try to post links then.
The full dump is 10 files that are 1GB each and 1 that is 209MB.
It took about an hour and a half to convert (not including downloading and extracting) on a dell core-i7 with 8GB RAM.
You'll need about 55-60 GB free to do it yourself
Comment by Avi (ikeboy) - Tuesday, 19 March 2013, 17:53 GMT
By the way the patch program didn't patch the last few lines of wikiviewer.c or viewers.config .
I did it manually and have included them.
After using the patch, replace /apps/plugins/wikiviewer/wikiviewer.c and /apps/plugins/viewers.config with my file.
Comment by Avi (ikeboy) - Tuesday, 19 March 2013, 18:44 GMT
This should be the folder link (right now has the first 2 of 11 files) :
https://mega.co.nz/#F!pg431CDB!aeAo_HT3YngdsxBzYuoksg
Please let me know if it doesn't work.
It will get updated automatically when the rest are finished.
Comment by Avi (ikeboy) - Wednesday, 20 March 2013, 17:41 GMT
All the files should be up now.
Comment by Avi (ikeboy) - Monday, 27 May 2013, 03:58 GMT
This is a patch that should apply clean. Also I'm uploading the may 3 2013 dumps of wikipedia now, it should be on the mega link above but it's not all there yet.
Comment by G.P. (dracayr) - Friday, 31 May 2013, 06:25 GMT
> As you can see at http://www.rockbox.org/irc/log-20130527#04:20:33 I am
> trying to put the wikiviewer into the rockbox core and they need official
> permission to use your code. I got 8 your emails addresses from
> http://www.rockbox.org/tracker/task/4755 . If any of you have objections
> to
> this please email me. If you are fine with it and release your code under
> the GPL, then the rockbox project needs your full names. Also please post
> any response to the tracker page . Thank you.


I hardly did anything on this patch, but for what it's worth:

I have no problems with the release of this code under the GPL license. My name is Gregor Pohl.
Comment by Thomas Martitz (kugel.) - Friday, 31 May 2013, 08:24 GMT
The mallocer.c should be removed before inclusion. Rockbox has a real malloc() for plugins, so if it this plugin relies on malloc() it should use this instead of creating a dumb malloc as the patch currently does. (I haven't examined why it relies on malloc but it would be best if it didn't need it at all)

Loading...