Rockbox

Attached to Project: Rockbox
Opened by Anonymous Submitter - 2006-03-01
Last edited by Willliam W - 2019-08-04

FS#4755 - Wikipedia

As already discussed in a thread [1] at mysticriver it would be very amazing to have a version of Wikipedia on rockbox.

Only a plugin to easily search articles would be needed.
Converted dumps are already available, for example the one of the ipodlinux project [2].

[1] http://www.misticriver.net/showthread.php?t=36924 [2] http://ipodlinux.org/Wikipedia

The task blocks this from closing
ID Project Summary Priority Severity Assigned To Progress
13048 Rockbox  FS#13048 - Wikipedia on rockbox  Very Low Low
100%
Closed by  Willliam W
2019-08-04 19:07
Reason for closing:  Accepted
Alexandre Flament commented on 2006-03-05 10:26
Will Robertson commented on 2006-03-07 11:33

As far as I can tell, the script on the iPod linux page you linked just converts the files into plain text, which you could open in rockbox anyway - give it a shot.

Wiki River commented on 2006-09-26 17:49

If you look at the Misticriver thread and the “Rockipedia” wiki, you’ll find information on a plugin of this kind being created.

Hope this helps.

Misticriver thread: http://misticriver.net/showthread.php?t=36924 Plugin wiki: http://rockipedia.techmight.com

Frederik commented on 2006-10-26 22:23

I have created a plugin to search out wikipedia articles, it takes about 5 seconds to search for an article (exact match) on an i-pod and uses b-trees for an index and gzip compression for the articles. (about 2.5GB, split into 1GB files for english retrieved wikipedia a week ago).

I will release the source soon (before 5.nov) when it is a little more finished.

Izzy commented on 2006-10-29 03:31

wow! that sounds awesome freqmod! i can’t wait!

Frederik commented on 2006-11-04 11:15

Haven’t made any more progress, but it works. The archiving functions are split out, but the viewer in mww.c and the parser in wmconv.rb could need a rewrite.

Izzy commented on 2006-11-04 23:02

Sorry for being such a noob, but how do i use this?

is that just the source? will there be a patch soon?

great work, btw. i can’t wait to use it

bro2

Izzy commented on 2006-11-04 23:06

woops… i found the readme file.

i’ll try to figure it out from that. though i don’t really understand most of it :(

thanks again!

bro2

Adam Gashlin commented on 2006-11-05 14:10

Brilliant freqmod! I’m now testing it on an iPod photo, reading the article about Stephen Hawking while listening to MC Hawking. Are you intending to work on rendering the articles?

Frederik commented on 2006-11-05 14:58

I won’t work on rendering now (if someone wants to I will certainly help), but maybe after a while. I second the release of connell’s source so I (and others) have a startpoint for the renderer.

Joe commented on 2006-11-05 16:58

I patched the rockbox build fine, but I don’t know how to dump the wiki.

Are the directions in your readme for linux users?

When it says in your Readme

—run ./Make.sh
run bzcat <wikimedia dump> | ruby mwxmldumpparse.rb /dev/stdin <prepared dump prefix>— ect…

What does run mean? How do I run it? I have to install Ruby, GCC, and bzip2? Any way to make this a little bit more friendly to newer users :)?

Your work is awesome btw. I’ve been waiting for this for a long time!

rckbxr

Frederik commented on 2006-11-05 21:40

ok, if you don’t want to install unix you could install ruby from http://rubyforge.org/frs/download.php/12751/ruby185-21.exe and compile the ( c ) apps in cygwin/mingw (and upload the binaries here) where you compile rockbox. (the bzip2 file could be decompressed ( http://www.bzip.org/downloads.html ) before it is passed to ruby if you have enough disk space)

Adam Gashlin commented on 2006-11-06 17:56

I’ve rewritten the viewer, it is very basic but working (I was having a lot of trouble with stuff dropping out or cutting off with the earlier one). I’ll be adding link following support later today if I can get to it.

   mww.c (9.3 KiB)
Adam Gashlin commented on 2006-11-06 18:43

The characters for italics are now ignored, and an issue with articles cutting off too soon has been fixed.

   mww.c (9.4 KiB)
Adam Gashlin commented on 2006-11-07 02:09

Links are now supported, Select to enter link selection mode, scroll to select a link on screen, Menu to back out of this mode, Select to load the linked article. There were some bugs in gzip which prevented it from working twice, those are now (mostly?) fixed.
The biggest irritants to me right now are: 1) backwards scroll speed 2) text insensitive matching which turns up the wrong pages (see FRance) 3) no “back” feature as one would expect on a browser.

   mww2.zip (13.2 KiB)
Adam Gashlin commented on 2006-11-07 06:15

You can now no longer scroll beyond the end of the article, and the link selector will stay on screen. Text sensitive matching is now attempted before insensitive matching, if there is an exact match we should use that (France is now accessible). The article title at the start of each article is now skipped. Scrolling may be a litte faster.

Adam Gashlin commented on 2006-11-07 07:44

Fixed a careless file descriptor leak.

Adam Gashlin commented on 2006-11-07 14:43

Scrolling backwards is now faster as the state of the last 50 lines is saved. There is a 10-deep history of pages, accessible by pressing Left. When pages are not found the viewer returns to the previous page (or exits if the start page was not found). Links that start before the first line on the screen now work properly. If no links are onscreen select will not enter link selection mode. The “searching…” display is a bit more consistent.

Does anyone care?

Frederik commented on 2006-11-07 18:39

Wonderful job.

I have updated the MARKUP_* macros to match the values in wmconv.rb, added colored underlining to signalize underlined, italic and bold text. Added a little more memory in inflate.c (to fix some problematic articles).

When it comes to case (in)sensitive searching i don’t know if a compare function that is different from the function used to create a b-tree will work, it will not however give false positives so it is no propblem, except that it uses a little more disk access time.

Adam Gashlin commented on 2006-11-07 19:49

A quick fix: insert
hist[curhist].curline=0;
at line 275 in mww.c.
curline was never being initialized.

Adam Gashlin commented on 2006-11-07 21:26

That exposed another issue regarding line counting (go to an article, go back, scroll up… jumps back to line 0 because the previous fix clears curline and we lose track of how many lines to seek ahead for backwards scrolling).
Should be good now.

Adam Gashlin commented on 2006-11-08 16:44

I used the viewer for a few hours yesterday (much easier to use with a large font to reduce eye strain) and came up with a list of things I think need doing (written in the dark at varying angles in red pen on random scraps of paper on my desk…):

* Don’t reallocate memory when loading new articles - though I know the current system will produce the same addresses each time the page history relies rather strongly on that fact, which could be a potential problem if things change.
* Case sensitive search only when the article title is produced internally, i.e. from a link (only case-insensitive search for user-entered text)
* Save the index of an article when found, so upon going back to it we don’t need to search again.
* Save current status to a file, to come back to it at a later point. This could produce problems if not handled carefully: changing font between runs will change line breaks (easily solved by storing the character offset of the beginning of the first line displayed and recaululating breaks upon loading), the database itself might change (I think this is unlikely enough to not worry about how to handle it, save to ensure it doesn’t crash), and other potential pitfalls, I’m sure.
* Menu - to use the save functionality, also to enter a new article title (should be treated as if it was a link followed so we can always go back), perhaps toggle emphasis highlighting (as I don’t particularly care for those colored underlines) and other configurables. Should take a look at the text viewer for ideas here.
* Better display of article title (right now it is ignored)

Stuff for the preprocessor (note however that I have not so much as read a line of it, yet):
* Process basic templates - stuff like seealso and main should be handled, as they are quite often used in large articles.
* Images - just put the alt text for the image
* Offsite links - not sure how to handle these… ignorance should be a fine policy
* Cross-mediawiki links - could potentially be supported if the database for the given article is also present, I don’t consider this too important, though
* Categories - Generating a page for each category, and processing the template on each article to link to that category, would aid navigation
* HTML Entities - should be replaced with equivalent unicode characters (what about &gt; and &lt; ?)
* Anchors - How to support? Simple half-solution is to just strip off the anchor when following a link.
* Do something nice with ==headings==, lists (on further thought this may not be necessary, both are presented reasonably enough in the current code)
* Do something (even if not nice) with tables
* Do something with the ref tags
* Redirect processing seems to have some problems

It would also be helpful to take a nice long look at how wikipodia works, although I don’t intend to prettify things too much and I’d perfer to keep it fairly simple.

Just some ideas I’ll be working on. Is there a better place for this kind of discussion than in the feature request?

Adam Gashlin commented on 2006-11-08 16:50

Sorry, by wikipodia I meant Encyclopodia, it had been a while since I looked thoroughly at that.

Frederik commented on 2006-11-08 19:08

A system for patching the wikis (to save network bandwidth) (I will try to make it):

  • Compares each article with the article in the old version against the new and store diffs (gz-/bz-/lzma- ed) or xdeltas (I will see what that is most efficient, and if diff/patch produces _exactly_ equal files) of the uncompressed article text
  • Store a “recepipe” (binary):
    • Copy article: <name>
    • Patch artilcle: <patch offset> <name>
  • Create an ansi c program that compiles on Windows/Mac os X and Linux, with no dependencies that patches a database.

Things that would be nice:
A menu where you could access:

  • Page history (a list, with page titles, and stored offsets (in database))
  • Document structure (the parser should generate an index with (byte)offsets (localy in the decompressed article)
  • Search for new article (with the last article name entered)
  • Exit
Adam Gashlin commented on 2006-11-08 19:33

I understand and agree with all your points but the Document structure menu item, I am not clear what you mean. Headings, anchors, sections? Yes, that makes sense, now that I think of it.
I’ve made a fix for some issues with multiple whitespaces and wrapping (if there are two spaces at the end of a line one will be consumed for the break, the other will be at the start of the next line, which I don’t like, the new version ignores multiple whitespace (with the exception of mutliple newlines)) but I’ll wait to post it until I’ve done a bit more work today.

Giles commented on 2006-11-10 08:15

Ok, so I understand all about the sources and compiling and all, but try as I might, rockbox won;t compile! I upgraded my make, arm-elf-gcc and gcc and it still gives me errors. Props to anyone who will compile me a build of rockbox for the ipod video containing the latest media wiki viewer, that way I can show off my iPod and get all the chicks.

Please, I will be eternally grateful.

Frederik commented on 2006-11-11 20:38

1 Bug in btsearch.c:87 LOGF(”K:%d,%s”,utf8strcnmp1)

1) (const unsigned char*)key),((const unsigned char*)nd_key),rkeylen,keylen),nd_key); –> LOGF(”K:%d,%s”,utf8strcnmp(((const unsigned char*)key),((const unsigned char*)nd_key),rkeylen,keylen,casesense),nd_key);
2 my rockbox build http://freqmod.dyndns.org/upload/rockbox.11.11.06.patches.mww.zip (with mww, and a few other patches (some does not work as intended, but should not harm
Adam Gashlin commented on 2006-11-11 23:59

Rewrite of a substantial portion, now with: menu (save history, new article, exit, clear history), history save/restore, no need to search again when visiting pages in history.
Not too heavily tested yet, though I will surely be using it frequently.

Adam Gashlin commented on 2006-11-12 00:17

Navigating to a new article through the menu didn’t reset the scroll offset. Also made abort from keyboard behave properly (the code was there but commented out).

Izzy commented on 2006-11-12 04:11

Amazing work, Adam. Don’t think you’re not appreciated. I wish that I could figure out how to use this! I’m on windows and for some reason I can’t figure it out. But I’m gonna keep trying; just wanted to let you know that you’re doing great work!

bro2

Adam Gashlin commented on 2006-11-12 05:15

Reinstated the “Loading…” message, renamed save files to .wws.
Izzy, if you can get in touch with me I can possibly walk you through the building, I’m often in #rockbox, my nick is hcs.

Adam Gashlin commented on 2006-11-14 01:07

I’m in the process of rewriting the xml dump parser in C with expat, as a first step towards improved preprocessing. I’m undecided about what to do with templates so far, though. Some I think it would be best to inline (main, seealso), some could use some special processing (categories), and some I just don’t know about (infobox).

Adam Gashlin commented on 2006-11-14 12:42

A few additional features for the viewer (haven’t gotten anything significant done with the preprocessor yet):
-seeking to targets (you know, the # things)
-navigation (the Navigate thing in the menu now displays an outline that you can scroll through, pressing select will jump to that part of the article)
-fixed a few bugs with saving
-history view (a list of the titles in the history)

Adam Gashlin commented on 2006-11-14 23:15

Faster backwards scrolling in outline navigation, additional option for entering a new article name from scratch, pressing select in the history view will jump back to the topmost displyed article.

Adam Gashlin commented on 2006-11-14 23:49

fix to make target seeking tolerant of spaces at the end of the heading

Izzy commented on 2006-12-02 22:23

Adam, is there anyway I can get the (already compiled) file to actually run the wikimedia dump? I did build rockbox with the mww viewer, but I can’t figure out how to compile in VMware. I’m not exactly sure how the actual dump process works, but maybe you can make this a little easier for me (the windows user) :(

thanks for any help

bro2

Adam Gashlin commented on 2006-12-10 15:27

bro2, what target do you have?

Izzy commented on 2006-12-10 16:37

Ipod 5g

Adam Gashlin commented on 2006-12-11 02:03

http://www.hcs64.com/wiki

Here you will find a somewhat out of date processed wikipedia dump and a build of the viewer from the latest CVS.

Izzy commented on 2006-12-12 02:06

Thanks so much Adam! As I’m downloading these files, I just thought I’d ask where I should put them. They’re(the .wwa files) all different parts of the wikipedia, correct? Am I able to search all these parts (the entire wikipedia) at once? Again, thanks a million. Your work is amazing!

bro2

Adam Gashlin commented on 2006-12-12 02:31

The wwa files are the actual article text, wwi is the index. They can be anywhere in the filesystem but they should all be in the same directory. The plugin actually opens the wwi file, which is then used to access the others.
mww.rock goes in .rockbox/viewers/, viewers.config replaces the one in .rockbox/ (or you can manually add the line
wwi,viewers/mww, 55 55 55 55 55 55
to the existing file)

Also, credit where it is due, the archives are all generated by freqmod’s conversion utilities, and the database backend of the viewer is still mostly his.

Frederik commented on 2006-12-12 17:03

Cleaned up warnings (in all files except mww.c where Adam has done a great job).
Added playbackmenu to the menu.

Frederik commented on 2006-12-12 17:30

Uploading specification of the wwi and wwa format.

I don’t have very much time to work on this project as I am preparing a program for showing videoes on multiple projectors which must be finished before a preformance in the last week of february (and for rehersals). I will problably work more on the patch solution for this viewer before that when I have made some more progress on that project.

Frederik commented on 2006-12-12 17:47

(the specification was raw text, all numbers were in little endian)

all text lengths are in bytes

Input for btcreate (endiannes like the architecture of btcreate):
In-list:
uint32 data_lo
uint32 data_hi
uint32 title_length
char8*title_length title (in UTF8)
Redirect-list:
uint32 from_length
uint32 to_length
char8*from_length redirect from (in UTF8)
char8*to_length redirect to (in UTF8) (the datapointer is taken from this entry in the in-list)

emilie dancer commented on 2007-01-01 00:17

Wow, just installed the viewer and made my own dumps from wikipedia. Thanks for all the amazing work! Can we change the status of this tracker to at least “Alpha”? So people know that it is in development and very functional?

Matthias Larisch commented on 2007-01-18 12:20

I have problems with my own dump… dewiki becomes a 35mb wwi + 1gb wwa file, but the reader doesnt find ANY article! Some problems are at converting with illegal links, but that are only about 50 or so… The rest of the converting process finishes without any error.

When i download the enwiki dump from the link above it works without problems… When I use my own dump, it says “Searching…” and immediately (within same second, when not same 100th of the second) it says “Not found” and quits the reader.

I have to say that i use iriver h300 (with #define BUTTON_SCROLL_FWD BUTTON_UP and equal ones for SCROLL_DOWN and BUTTON_MENU) and the plugin works fine :) but only with the enwiki dump from above. The Ipodsimulator doesnt work with my dewikidump either.

Any ideas?

Thank you!

Matthias Larisch commented on 2007-01-18 12:21

Oh if someone wants to look at my .wwi file: http://eow.ath.cx/other/tier/rockbox/dewiki20061130.wwi (currently uploading - finished in 30 minutes - 35MB)

Frederik commented on 2007-01-18 15:40

I have looked at the wwi file. If you could provide (a part of) the wwt file it would help. It seems like all the entries are in the .wwi index but all the values (pointer to the adresses in the wwa file) points to 0 0. If the wwt file also has 0 0 as the index then there is a problem with the ruby converter, else there is a problem with btcreate.

If you want to investigate yourself open the file in an hex editor and use the file format documentation as a reference.

Matthias Larisch commented on 2007-01-18 19:43

Thanks for your fast response. Oh yes, in wwt-File there is everything (except one byte) between the titles set to 0. The ruby converter of your second post (mediawikiviewer.tar.bz2) is the right one? I’ll have a look at the source and try a bit around… Well there is one step i did different than the README:
instead of using bzcat file | ruby converter /dev/stdin outputfile

i used directly ruby converter <my_decompressed_dewiki.xml> <outputfile> because my cygwin doesnt have a device /dev/stdin.

I’ll try doing that directy on a real linux box with readme command next night after looking in the ruby script.

Matthias Larisch commented on 2007-01-19 07:16

It works :) The cygwin ruby reads only zero at $filepointer.len which is used to write the offset to wwt file… I tested this with a debug-print in there. On my real linux machine it works, the generated wwi/wwa files do work too :))
Really nice work!
Maybe I’ll write a fast C converter when I have the time (but that would be windows only at first - maybe someone will port it some day)

Izzy commented on 2007-01-20 22:45

I’d love if you did that, Matthias. A converter would be amazing! :)

Matthias Larisch commented on 2007-01-21 00:16

Well I have begun today morning… It is able to create wwr File for now. wwt should be now problem but i have a general C/Pointer problem now :) I’m not very good in C. My code is verrrry dirty, but works and is much quicker than the original ruby code.
For now I have no parsing of any style - I expect a completely working alpha-version tomorrow evening :)

Izzy commented on 2007-01-21 04:16

Sweetness. Can’t wait :)

Matthias Larisch commented on 2007-01-21 08:22

But you know that u can also use the converter in freqmods second post? Well it is maybe a bit complicated, but mine wont be much easier ^^
Only problem is that it doesnt seem to run on a cygwin, dont know why. But you should try it with win32-ruby. Real linux should be best :)
My problem with it: pc is simply rebooting at output filesize ~720MB. Maybe that’s my crappy PC, maybe a harddiskerror, dont know. Complete Output file is 1000MB in size. Thats the main reason why i’m coding a new program - so that i can use the whole wikipedia :)

What is your problem with freqmods converter?

Matthias Larisch commented on 2007-01-22 13:26

hmm will take a little bit longer, sorry :) Had to cope with a bufferoverflow which i didnt find:) Now everything is clear, work goes on and maybe finished today or tomorrow. But now i have to go to my real work for 5 hours :)

Ah btw a problem with the ruby converter: My PC always crashed at 2GB of the input file. this is what i found out. I didnt use directly decompression with bzcat but converter <wikidump.xml>. Maybe ruby has a problem with the 2GB limit?

Izzy commented on 2007-01-22 21:37

Actually, I’ve never suceeded in dumping my own file. I merely used the kinda outdated files on Adam’s website http://www.hcs64.com/wiki. I have a working cygwin installation, but the instructions included in freqmod’s Readme weren’t clear enough for me to finish. A n00b like me needs a step by step guide with everything included :-\\. I was hoping for something that would be a bit easier :-)

bro2

Matthias Larisch commented on 2007-01-23 00:19

Okay :) converter is in a useful state now. But it doesnt do ANY layout/article processing. It only creates one big wwa file and the two wwr/wwt files.
Just if anyone needs it now… I would call it 0.1 pre-alpha :) because of the missing main functionality.

how to:
run compile (requires zlib & pcre devel packages)
start xmlconv with wikidump as first argument and output prefix as second argument.
(cygwin or reallinux!)

After that u have to use the btcreate from freqmod.

dont blame me for bad coding… I’m not that good at it ^^

conversion time: dewiki 2,5gb → 1gb wwa file in exactly 45 minutes on amd athlon xp 1700+ with 512mb sd ram running debian 3.0

Matthias Larisch commented on 2007-01-24 19:43

okay here is a new version :)

Conversion time on 1,86GHz Pentium-M Laptop (slow HD…) was 13 minutes for de-wiki. I think thats quite amazing in contrast to original converter :)

limitations:
-doesnt really parse the XML, only wikipedia “style” will be parsed correctly
-Parsing of layout (bold, etc.) is done via char for char comparison… this speeds up and should be expandable later (regexps should work also)
-I dont know how mww expects Links (links work, but not links with other name than the target)
-gzip-compression really bad implemented… writes to temp.gz file on harddisk and rereads it… I couldnt find an easy to implement way of in-memory compression. This should give another speedup :)

I added precompiled binaries compiled on my cygwin machine.
Libz and pcre-development packages are needed for compiling.

Have fun with it :)

Frederik commented on 2007-01-24 21:19

Links: A_START <link/description> A_END
or: A_START <link> G_MODE <description> A_END
From the wwi-specification:
Name ASCII-code:
A_START 007
A_END 008
G_MODE 015

I hope this helps you :)

Matthias Larisch commented on 2007-01-24 22:33

Ahh, G-MODE is the one i look for :) But it is not implemented in mww?

Was a very simple addition to the converter… It now should work with same features as yours :)
I forgot to say: It automatically strips off html comments (<!– –>) and converts &gt; and &lt; to > and <.
One weird thing: my article.wwa file was shrinked from 1000MB to 870MB after implementation of the whole conversion… Okay, for ’ it is 5:1 compression, but that shouldnt be sooo many savings… Same for html comments… I browsed through a few articles, they were allright.

–snap
found mistake :) I declared comment-endings as –&lt; but they are –&gt; :) so articles with comments should be missing

attachment: new xmlconv.c
-changes version number to 0.2
-fixes comment-end mistake
-added functionality: alternative link-texts are supported

(application/octet-stream)    xmlconv.c (16.4 KiB)
Matthias Larisch commented on 2007-01-25 20:48

freqmod: there is a problem :) My links wont work now!
After looking to mww.c:
#define MARKUP_BAR 13
this is used for seperating link from description!

In your converter u have PL_END as 13 and use that :)

i’m a little bit confused…

Frederik commented on 2007-01-27 21:11

I understand why you are confused. I don’t understand the links myself.
My convertor:

variable declarations:

A_START=« 007
A_END=
« 8
… PL_END=« 013
G_MODE=
« 015

links (with pipe):

arr[i]=Mwtags::A_START+tmp[0,pipeidx]+Mwtags::G_MODE+tmp[pipeidx+1,tmp.length-pipeidx-1]+Mwtags::A_END
(this line uses G_MODE)

The output makes a link with 13 (PL_END) as separator. I don’t know why, it seems like a bug/unwanted feature in my code. However your convertor (Matthias) works just as well and is much more efficient and makes my ruby convertor obsolete. Therefore i have rewritten mww.c to use 15 as G_MODE.

This breaks compatibillity, but this plugin is in no way stable so I think it would be better to change the behavour than to change the “specification”.

Updated convertor, to work with xmlconv:

   mww16.tbz (23.4 KiB)
Frederik commented on 2007-01-27 21:12

ehh, updated mww

Adam Gashlin commented on 2007-02-15 21:02

Updated xmlconv.c to support reading from stdin (by using - as the input file name), so I can pipe the output of bzcat directly into it without using up disk space on the uncompressed archive. Full enwiki takes about 1 hour 15 minutes. I’d like to avoid the creation of the temp.gz thing as well but I haven’t gotten to it yet.
I hope to get some more work done on mww now that I’m nominally done with NSF and SPC.

   xmlconv.c (16.5 KiB)
Matthias Larisch commented on 2007-02-15 21:55

nice work :)

i didnt get zlib working like expected (gzip headers etc.) without the temporary file. In-Memory compression without this file should give another speedup of about 10% I think… Did file splitting work correct? I never tested this ^^ dewiki is only 980MB compressed :)

Adam Gashlin commented on 2007-02-15 22:17

Yes, files are split correctly, there are three.

Adam Gashlin commented on 2007-02-16 08:30

Updated mww.c to use up and down keys instead of scroll so it’ll build for gigabeat, seems to work fine.
Since there are a few of this potentially working on this at the same time, should we set up a sourceforge project for it for version control until it is ready for rockbox?

   mww.c (39.1 KiB)
Adam Gashlin commented on 2007-02-17 21:14

Fixed some problems with redirect processing (fseek was using the wrong file descriptor), more entity handling in xmlconv (quotes, especially), no use of temporary file for gzip but no speed improvement (perhaps need to directly use deflate).

Adam Gashlin commented on 2007-02-19 02:01

Here’s the version of the conversion tools I’m currently working with. xmlconv would do odd things if there was wikicode inside a comment, I moved comment processing first in the parsing code to combat this.
I’d like to get rid of the ref tags, as well, I don’t care too much about the references and they make things a lot less readable.
Packed everything together for ease of management, and also mww17 so one doesn’t have to pick things up all over this task…

Adam Gashlin commented on 2007-02-19 02:06

reuploading converter04 as .tar.bz2

Adam Gashlin commented on 2007-02-20 07:39

Reimplemented article parsing with a recursive-descent parser (mostly), should allow much more advanced parsing in the future on this framework. Newly supported things include images (well, a link to the page with info about the image shown as [I], and normal text processing on the caption, which may itself include links). The code is something of a preprocessing monstrosity, but I think it lends itself to being easier to tell what the parser is actually doing. Speed is pretty much the same as the old version.

(If anyone disapproves of my totally ripping apart xmlconv, sticking my name in it, and releasing with a new version number, let me know…)

Adam Gashlin commented on 2007-02-20 07:56

Oh, and I remove ref tags.

Adam Gashlin commented on 2007-02-20 16:36

Revised entity support to understand the fact that &ndash; is for instance &amp;ndash; in the dump. Added support for all HTML entities I’ve been able to find tell of. Despite the fairly large number and an inefficient linear search for them, they are the exception and do not seem to slow down processing at all.

Adam Gashlin commented on 2007-02-21 23:49

ref checking now part of general text processing (to catch it anywhere), added support for self terminating ref tags.
I’d like to once again redo this, with a stage for each level of translation done (1. xml 2. wikicode 3. html rendering) which I think will present a more logical flow for things and improve some bits that are sort of hackishly scattered in at the moment. Might be a while before I get to it, though.

Matthias Larisch commented on 2007-02-22 06:24

You are doing very great work, exactly what i hoped to see after my initial “new convertor”. I do not have any problem with you ripping away the whole program or with sticking your name in it. As i earlier said, my version is very dirty but already worked :) If you manage to write a converter that does much better job (which it already does) in nearly same time, this is great! I personally wont do much work on this project in future because I have very little time and not the programming skills needed…

Adam Gashlin commented on 2007-02-22 21:55

Those ref tags continue to be a pain, I forgot that html tags are case-insensitive.
I’m wondering if it might be a good idea to try and use the static HTML dumps? That way we wouldn’t have to worry about some of the more irritating to implement aspects of wikicode, like templates (though I’ve given them a lot of thought). The downside is that they are rather infrequently updated. I’m not considering this too seriously.
I think the multi-stage approach, firmly grounded in the structure of the grammars involved, should be fine. I’ve just got to get some structural aspects of it decided. I’m taking a compilers course now so I’m trying to apply what I’m learning about parsing to this. Wondering if it might be better to just see how mediawiki does it and rewrite in C, though…

Considering templates again, should they be expanded inline or kept as a seperate thing? I’m thinking inline, but I worry it might make things a lot bigger than they need to be.

Adam Gashlin commented on 2007-05-09 16:34

sync’d, and formulated as a patch for easy application

Adam Gashlin commented on 2007-06-17 18:57

synchronized

Timo Horstschäfer commented on 2007-06-23 11:10

As this plugin and FS#6697 have almost the same purpose, it would be better if we could merge the projects or work together on one of them.

The advantages of this plugin is its support for links, the gzip compression and binary tree search (maybe I missed something, because I hadn’t had a deeper lok at this plugin for some time…).
What I like about my dict plugin is that it isn’t limited to accessing the Wikipedia (a lot of other dictionaries as well) and uses a simple fuzzy search.
I think it’s also a better idea to create a plugin-independent document viewer for Rockbox.

So what are your opinions?

Adam Gashlin commented on 2007-06-27 05:44

While you have not had a deeper look for some time, I’ve never looked at FS #6697, so I really have no idea. I can barely find the time to sync this now and then, but I agree that the effort would probably be best spent on a plugin somewhere between these two. I’m just not able to do any of it myself at the moment.

Alistair Marshall commented on 2007-07-24 19:29

should this task be changed to a patch rather than a feature request?

peter watkins commented on 2007-08-15 20:20

Frederik, Adam, et. al. – thanks for creating this! I’m enjoying having Wikipedia (EN) on my iPod.

Here are a few notes for other users from my experience with this and a fairly recent (week old) Subversion checkout:
- The English Wikipedia dump would uncompress to to something like 11 GB. If you’re processing the dump on a filesystem that can’t handle files that big (say, the fat32 fs on your DAP), you’ll want to use pipes to avoid the need to make an uncompressed file. On Linux, something like the following will allow you to make the article files.
bunzip2 -c enwiki-20070716-pages-articles.xml.bz2 | ./xmlconv /dev/stdin wikipedia
- How it works: this creates a “viewer” for .wwi files. To view/search a processed wikimedia article dump, use the Files browse manu, find the “.wwi” file (both the wwi and wwa files must be on your DAP, as noted above), and open it. If your Rockbox DAP is configured to show only supported files, you’ll see the name of only the wwi file.
- Fonts and non-ASCII characters. Not all Rockbox fonts have good support for non-US-ASCII characters. If you find that letters with diacritical marks are not displayed properly, try a different font.

Timo – I tried your dict plugin (was really looking forward to it), but kept running into errors (iPod 5.5g, 64mb). This wikimedia viewer seems more stable.

Personally, I’d also be interested in a PC app to convert dict files to wikimedia XML format so they could be viewed with this plugin.

Frank M. commented on 2007-08-19 12:30

Dewiki and enwiki are working fine on my iPod nano and video :)
The patch doesn’t compile for Sansa e200 though.

Adam Gashlin commented on 2007-08-29 12:51

I’m looking at WikiFilter (wikifilter.sf.net) as a starting point for wikicode parsing, and I’ve downloaded a few dumps to start experimenting with updates.
And at some point I should look at that dict plugin…

Timo Horstschäfer commented on 2007-08-29 16:31

I suggest having a look at FlexBisonParse which is listed on http://meta.wikimedia.org/wiki/Alternative_parsers.
It looks quite easy to adapt to custom output.

Adam Gashlin commented on 2007-09-04 14:24

Updated patch, I don’t think there’s anything new here but it’ll apply to current SVN.

Simon Wenger commented on 2007-09-08 19:24

Can someone point me to wikipedia dumps already prepared for this plugin? (The links above give me 404’s)
Thanks, Simon

Adam Gashlin commented on 2007-09-09 02:17

Indeed, that page is down. I don’t have the capability to upload the 3.8 GB the most recent converted enwiki dump (2007/08/02) takes up, unfortunately, though I do have the web space. If someone else is interested in volunteering I can set up an FTP account to put it up in my space.

Simon Wenger commented on 2007-09-09 06:51

Ok. I’m downloading and converting the dewiki (german) now. I might just as well upload it later, after testing. I’ll contact you about an FTP account then.
And I’d like to write a short HOWTO on the patching (no details) and the downloading/converting of the dump. Should I just place it in a comment or could we put it in the top (Details section)? It’s confusing to have the old converter method and dead links in the earlier posts.

Simon Wenger commented on 2007-09-10 07:07

The patch applies to the current revision, but doesn’t compile.

MAKE in mww
CC mww.c
mww.c: In function ‘set_article_offset’:
mww.c:189: warning: implicit declaration of function ‘printf’
mww.c: In function ‘viewer_init’:
mww.c:936: error: ‘BUTTON_MENU’ undeclared (first use in this function)
mww.c:936: error: (Each undeclared identifier is reported only once
mww.c:936: error: for each function it appears in.)
make[3]: * [/home/simon/Desktop/iaudio/bleeding/rockbox/build/apps/plugins/mww/mww.o] Error 1
make[2]:
* [mww] Error 2
make[1]: * [rocks] Error 2
make:
* [build] Fehler 2

From the little I know about programming, I guess it’s because of the target? I compile for iaudio X5 (sim) which does not have a menu button.
Am I right. Is there a way out of this?

Simon Wenger commented on 2007-09-10 07:28

Hmm, can’t compile for H300 sim either… and this target knows a menu button (A-B).

Daniel Dalton commented on 2007-09-10 07:35

Hmmm it looks like “BUTTON_MENU” hasn’t been declared anywhere. You will have to find out what it is called. Maybe BUTTON_MENU is the menu button? (Ab on h300)

Adam Gashlin commented on 2007-09-10 08:26

Looking a the keymaps, it looks like BUTTON_MODE|BUTTON_REL is used for the menu function on the h300, so just drop that in there. I should probably find out how to get access to the keymap stuff so mww doesn’t have to be too nastily specific to each target.
Regarding making things easier, we’re really in a horribly hacky incomplete state now, especially with the converter (though mww is a mess codewise, too). I’d rather it not be too easy to do things the wrong way, especially to avoid transitioning issues when a better way of keeping things updated comes along. This is, however, merely my position, and I’m not doing anything active for this project at the moment anyway (sorry for the grand plans a few messages back), so by all means make things happen if you wish.

Frank M. commented on 2007-09-12 12:37

It compiled ok for my Sansa E280 when I replace BUTTON_MENU with BUTTON_POWER in mww23.diff.
The article scrolling is a bit strange (using the up/down button instead of the wheel) but it works.

Simon Wenger commented on 2007-09-12 21:03

Yes, I did almost the same. I replaced BUTTON_MENU with BUTTON_MODE|BUTTON_REL, works perfectly, thanks! An impressive piece of software!
Adam, you could send me an email with the ftp login now, I will then upload the converted dump (german atm, maybe someone else can do others?), and, later a little howto. simon(?)jso.be

Xinlu Huang commented on 2007-09-23 23:20

I encounter a stack over flow when using the xmlconverter:
Exception: STATUS_STACK_OVERFLOW at eip=00403333
eax=005F73BC ebx=00000000 ecx=00032CDC edx=0000004C esi=611001A0 edi=004037A0
ebp=0022CCE8 esp=0022CCD4 program=C:\cygwin\home\…\xmlconv.exe, pid 5788, thread main
cs=001B ds=0023 es=0023 fs=003B gs=0000 ss=0023
Stack trace:
Frame Function Args
0022CCE8 00403333 (00000003, 61169690, 01630090, 00000000)
0022CD98 61006198 (00000000, 0022CDD0, 61005510, 0022CDD0)
61005510 61004416 (0000009C, A02404C7, E8611001, FFFFFF48)

    8 [main] xmlconv 5788 _cygtls::handle_exceptions: Error while dumping state (probably corrupted stack)

I’m using cygwin on XP, if that helps. Any idea why this is happening?

Adam Gashlin commented on 2007-09-24 23:00

I’ve never encountered that error, then again I’ve never tried using the converter in cygwin.
Exactly which mediawiki dump are you using?

In other news, Simon’s conversion of dewiki (20070903) is now on my web site.
http://hcs64.com/rockbox/wikipedia/

Xinlu Huang commented on 2007-09-25 02:49

I use enwiki-latest-pages-articles.xml.bz2 from http://download.wikimedia.org/enwiki/latest/

The link to the converted wiki is interesting, but I don’t understand a word of German ;) I’ll need to find some way to get the enwiki in the right format…

Adam Gashlin commented on 2007-09-25 06:36

Aaand, my mirror of dewiki is now down. Went through 250 GB in a few hours (I have 3.6 TB monthly but they won’t let me use it all up in a day). I’d try something like a torrent but I don’t have anything to host it from.

Xinlu Huang commented on 2007-09-26 21:52

I got Ubuntu for this and the xmlconv works pretty well on my USB stick ;) There is still quite some tags not parsed - ouststandingly (for me) some tags in the beginning of articles, the quote tags, the math tags (which I of course don’t expect a nice integral picture or something, but reading <math></math> is quite annoying, and for most case it will be simple removal of the math tag itself). I wonder are these unparsed tag expected?

Also I’m curious about the coding in the converter: is UTF-8 used?

As for the cygwin problem, I’m guessing that cygwin restrict mem usuage, etc. since it is running in a host enviroment, so maybe that’s where the stack overflow comes from.

It is so wonderful to have wikipedia on my ipod :D

Adam Gashlin commented on 2007-09-28 00:59

Yes, UTF-8 is used. Regarding the HTML tags, they’re handled on a case-by-case basis, and many are not handled at all. The converter needs a rework. Sorry for the extended “proof of concept” phase.

Fed commented on 2007-10-06 14:09

Is there a way of getting a peared down set of wiki pages?

Adam Gashlin commented on 2007-10-06 16:05

What do you mean? Like a selection of a smaller number of important pages? There are projects that do that but I don’t think they provide XML dumps.

Fed commented on 2007-10-06 16:17

Where am I supposed to put the xml file on the sansa?

Fed commented on 2007-10-06 18:16

That’s exactly what I mean. Can I save some pages, and make my own xml file and use that?

Simon Wenger commented on 2007-10-06 22:10

Here’s the very first version of a really simple HOWTO. It took me a while to figure out, where to get and how to convert the wiki dumps and how to compile for other targets than ipod. Hope others can make a shorter way.
Please feel free to send me any improvements. I will love to include them.
I suggest to make the document available in the head of this page, as it has become very long and complex by now.
Thanks to Matze88m for the work on the converter and freqmod and AdamGashlin for this fantastic plugin, I’ve spent at least 20 hours in the past month, learning my head off….

Fed commented on 2007-10-07 09:10

Thanks a lot for the howto!!!

Fed commented on 2007-10-07 13:22

I get an error when I try to compile. It looks like pcre.h is missing.

Xinlu Huang commented on 2007-10-07 13:32

Your linux distro (or cygwin?) does not have the libpcre package. You have to install it.

Fed commented on 2007-10-07 13:38

I am using a mac. Do you know what I have to install?

Fed commented on 2007-10-07 13:49

I seem to have found a pcre.c for mac. Does this seem right to you? Where should I put it in the file tree?

   pcre.h (12.1 KiB)
Fed commented on 2007-10-08 15:13

I am sorry about all the posts. I have finally gotten everything to work.
The pcre for mac is available at http://pcre.darwinports.com/

I use a Sansa, and in addition to “replace BUTTON_MENU with BUTTON_POWER (Thanks framo!)”, you should replace BUTTON_SCROLL_BACK with BUTTON_SCROLL_UP, and BUTTON_SCROLL_FWD with BUTTON_SCROLL_DOWN

I also added “rb→backlight_set_timeout(1);” after “rb = api;” as well as
“rb→backlight_set_timeout(rb→global_settings→backlight_timeout);” before “return PLUGIN_OK;” so that the backlight stays on while I read, and goes back to the user settings when the plugin closes. I think if this could be set up as an option it would be very useful.

Would it be possible to have a function to copy text to a text file? (ie export a selection)

Also, could someone add a bookmarking function? I don’t even know where to start for that.

Thanks again for such a great program!

Simon Wenger commented on 2007-10-08 17:06

Glad it worked! Bookmarking is implemented. Press rec (in my case) to get the menu, then save the history. Now the last article is reopened when restarted and you can view the history and jump back. As long as you don’t resave the history it will remain unchanged.
The backlight thing would be really cool!

Fed commented on 2007-10-08 21:21

What about copying text. Any idea how that could be done?

The problem I find with the bookmarking as it is now is that you can’t pick and choose what to keep. You have to keep it all.

Fed commented on 2007-10-08 22:47

I am trying to make a faster scroll (using the rec button), and I think I found an error in the code.
Should line 224 be:

          advance_scrollback(1,(dorender && i==a-1));

it is currently

          advance_scrollback(1,(dorender && i==-a-1));
Adam Gashlin commented on 2007-10-08 23:39

Bookmarking for individual articles would be nice.
Yes, that would appear to be a bug on line 224. Doesn’t show itself because I never use multiline scroll except when scrolling backwards past the scrollback buffer. I guess I just never noticed when that failed to render as it only happens on the one line when the buffer must be regenerated, and I only ever tested that when scrolling back quickly so a single frame went unnoticed. I just tested this now and it is in fact a problem in the current build.
When you’ve enabled your faster scrolling and the backlight fix (as that had bothered me as well) could you post an updated patch? If not I could, I guess.

Fed commented on 2007-10-09 00:29

Here is the new mmw.c
I haven’t made the patch because I have a lot of other changes, and it is too confusing. Can you make the patch with this?

   mww.c (40.2 KiB)
Fed commented on 2007-10-09 13:38

I was thinking about how to save text.
How about using the right key to advance the text (ie in the ‘normal mode’) and at the same time append the ‘current line’ to a file named [title of the article].txt

Fed commented on 2007-10-10 08:54

Here is my latest version of mww.c
It does a more universal page scroll.

Also,when you click the right arrow, it saves the current line of text to a file (in a wiki folder which you have to make prior to using it), but I only get the first 3 letters for some reason. I really cannot figure out why. Can anyone help?

Thanks

   mww.c (41.9 KiB)
Fed commented on 2007-10-11 04:58

This version creates a wiki folder, where the ‘recorded’ lines are stored. The lines are stored, one at a time as you press the rec button, in a file titled by the wiki page you were viewing.

   mww.c (41.8 KiB)
Fed commented on 2007-10-11 05:45

I cannot figure out how to make a diff file for this. I have tried a number of things, but no luck. Could someone let me know how or do it themselves? It seems that “svn diff > mmw24.diff” doesn’t recognize the mww folder.

My addition works very well now, and I would like to add it to the project.

Adam Gashlin commented on 2007-10-11 19:37

You need to do
svn add rockbox/apps/plugins/mww
first (minus the first rockbox/ if you’re already in the rockbox/ directory, of course).

Fed commented on 2007-10-11 23:57

Thanks. Here you go. I hope I did it right!

Fed commented on 2007-10-14 05:25

Bookmarks done. Also, right arrow scrolls forward in history.

   mww25.diff (105.7 KiB)
Adam Gashlin commented on 2007-10-14 11:44

Your patches are a bit odd, all the files but mww.c are in rocbox/whatever, mww.c is just in apps/plugins/mww/

Fed commented on 2007-10-14 15:11

Sorry. I fixed it.

   mww25.diff (105.7 KiB)
Fed commented on 2007-10-23 00:27

I am having a problem trying to make the recording feature record the top line of the screen rather than the second line (as it is now). In order to do this, I need to place the contents of the ‘current line’ into a string.

ie I need to copy

 buf[(i<=LINEBUFLEN)?i:LINEBUFLEN]='\0'

into a global string. I have tried many things but no luck. Can anyone help?

Fed commented on 2007-10-23 18:00

Major fixes to the recording function, and a few minor fixes to scrolling.

To get a gap in the recording between areas where you record the text, press and hold the select button.

Also, I am using the select button for scrolling down one page because it is easier that way on the Sansa.

   mww26.diff (107.1 KiB)
Adam Gashlin commented on 2007-11-04 05:29

Updated to build once again on Gigabeat F and iPod 4G, as well as the e200 and c200 Sansa. It now uses a set of #defines near the top to set the keymappings based on the device so we don’t have to go flipping those around.
I don’t think I understand the “gap in recording” feature, though.

Fed commented on 2007-11-04 15:28

It is useful for when you record a few lines of text and then move to another area in the article to record another few lines. The gap adds “\n\r” to separate the two sets of recorded text. Otherwise, you would have continuous text, and it would be more difficult to read and find the individual records.

Adam Gashlin commented on 2007-11-05 06:16

Ahh, I see now. I misnamed the macros, then, sorry.

kimangroo commented on 2007-11-14 20:59

Thanks a lot for all the work put into this, I’m an avid wikipedia user so it will be great to have it on my ipod.

Unfortunately I haven’t been quite successful yet though…

1) got the same stack overflow problem as Xinlu with the converter in Cygwin in XP. Has anyone had success using Cygwin and xmlconv?

2) am now trying to use the converter with a Linux live CD (knoppix or ubuntu) but I don’t understand how I can convert an 11+ Gb file on a FAT32 drive… How big would a converted file be? Does anyone have any pointers?

Adam Gashlin commented on 2007-11-15 09:58

The converter is designed to output .wwa files of only up to 1 GB in size.

kimangroo commented on 2007-11-15 19:58

Knoppix live cd made short work of the converting and now it’s all up and running on my ipod so thanks a million.

Can’t seem to get the links working though. Pressing select works as page down scrolling. Have I got the wrong patch?

Anyway thanks again so much to everyone who put work into this!

Fed commented on 2007-11-15 21:38

Arrow down activates the links. I found that page down was used more frequently, so I use “select” for it. Any suggestions which is better?

Alistair Marshall commented on 2007-11-16 09:09

It would make much more sense arrow down to scroll down and select to activate link. Try to keep them as obvious and natural as possible

kimangroo commented on 2007-11-16 21:17

Hmm, I can’t work out what I’m doing wrong here! When I press Arrow down I scroll down line by line, when I press select I do page down. Can’t seem to work out how to select links.

I tried to patch the source again and recompile but now cygwin’s telling me “cannot find file to patch at input line 5, Perhaps you used the wrong -p or –strip option? blah blah blah…”

As you may have guessed I’m a little new to this, I’m sure I’m missing something really stupid. ;)

When/if I do get it working I agree with Alistair that it would probably make more sense to have arrow down to scroll page down and select to activate a link.

Fed commented on 2007-11-21 12:46

I tried converting the wiktionary.xml file I downloaded, but I get many “unsupported entity” errors. Also, after using btcreate, I get “SCRLD”.

The converted file seems to work on my Sansa, however. Is there a problem, or should I just ignore these “unsupported entity” errors?

Adam Gashlin commented on 2007-11-21 16:52

I don’t remember what “SCRLD” was for, but the “unsupported entity” messages simply indicate that there is more work to do in the parser.

Fed commented on 2007-11-22 13:20

The wiktionary seems to work fine. I would like to make the “find article” function find all articles that start with the search string, not just an exact match. I cannot figure out how to implement that, though. I looked through the code, and I think the function to adapt is “search_btree” in “btsearch.c”. I just cannot figure this one out. Any help would be great.

kimangroo commented on 2007-11-23 06:48

Still can’t get this to work properly unfortunately. Has anyone else got the links to work with an 80G Ipod video? Or is anyone else having the Arrow down scroll down as well?

If anyone has got a working patched Rockbox.zip for 80Gb Ipod, do they feel like sharing it with me?! My email is my username + “@gmail.com”

Thanks! ;)

kimangroo commented on 2007-11-23 06:50

Or you could upload it here. Just noticed that you can attach a file of up to 2 MiB which should be just enough!

Fed commented on 2007-11-24 05:56

I think it would be nice to implement a new search that lists all articles from the .wwi list starting with the search string. For example, searching for ‘car’ will produce a list starting with ‘car’, then ‘card’ then ‘care’. You could then scroll to the entry you want, and load the article.

The reason I don’t do this is that I cannot figure out how to get the wwi data extracted. Any help would be appreciated.

kimangroo commented on 2007-11-24 06:36

Fed I think that’s a great idea and would really increase the usefulness of the viewer. I’d love to help you but as you can see I don’t have the first clue about programming, rockbox or anything much!

almargo commented on 2007-11-24 17:49

I have made a new mww.c which replaces the page down with select a link. It also includes the keymaps for iriver h100/h300.

   mww.c (51.2 KiB)
Fed commented on 2007-11-27 01:12

I am still trying to get a write a new ‘find’ function, but I cannot seem to figure out how to access the article titles. I think they are in the wwi file. Is this correct? Also, how can I access these titles individually? I cannot seem to figure out the functions that were written for this ap. I think the function in question is “search_btree” in “btsearch.c” Any help would be great.

Isaac commented on 2007-11-27 09:32

Damn this looks awesome… I’m fearing reading through that huge wall of text to understand how to patch it in, but I’ll give it a shot.

Hah.. figured the file size would be huge (currently capped and downloading at >1kbps)

Isaac commented on 2007-11-27 22:50

Hmm I tried patching the file in and it doesn’t seem to have worked. After further investigation I found that it’s placing a Rockbox folder inside the Rockbox folder, rather than placing directly into the apps?

Timo Horstschäfer commented on 2007-12-06 21:43

Have you considered using a hash table instead of the b-tree? It’s is simpler and a lot faster than a b-tree so maybe a “search while typing” would be possible.

The “find articles that begin with” is available in my dict plugin (FS#6697), but there seems to be a bug on some players and I’m too lazy at the moment to continue development.

Isaac commented on 2008-01-12 00:27

Ok after much work I’ve managed to get it all completed up to the point of converting the xml. Pcre library is installed and i have xmlconv.exe and after running the command ($ ./xmlconv enwiki-20080103-pages-articles.xml enwiki-20080103) I received this error…

 16354 [main] xmlconv 2368 _cygtls::handle_exceptions: Error while dumping stat

e (probably corrupted stack)
Segmentation fault (core dumped)

Anyone can help me? despite this thread appearing dead…

Adam Gashlin commented on 2008-01-12 06:19

Reading up a bit, it looks like the converter doesn’t work well in cygwin with something as large as enwiki.

Isaac commented on 2008-01-12 06:54

Any other suggestion? VMWare doesn’t exactly allow file sizes that big and I’m on dialup and have no desire to download a whole linux OS..

Xinlu Huang commented on 2008-01-12 14:51

Check out some smaller distros - like puppylinux, feather linux, damn small linux, etc. They usually come in around 100MB - should not wear your dial-up out.

Frank M. commented on 2008-02-03 00:13

Attached a torrent for the German version of the converted wiki download from January 21st, 2008, size is 1.5 GB.
After downloading the file, please leave it in seeding mode for a while to share it with other rockbox users.

Frank M. commented on 2008-02-03 01:44

When compiling mww28.diff on my Sansa E280 I got an error with two unkown buttons.
It compiled fine when I replaced in “mmw.c” the two entries BUTTON_SCROLL_UP with BUTTON_SCROLL_BACK, and BUTTON_SCROLL_DOWN with BUTTON_SCROLL_FWD.
I also had to increase the value for “backlight_set_timeout” from 1 to 15 because the LCD kept switching off after 1 second when opening the wiki plugin.

Frank M. commented on 2008-02-03 15:29

Attached the torrent of the English WIKI dump from January 3rd 2008, size is 4.2 GB.
Hope this helps, again please continue to share the folder after downloading it.

Fed commented on 2008-02-03 16:58

Thanks.

Adam Gashlin commented on 2008-02-08 11:41

Here’s a new patch. I do backlist_set_timeout(0) and I fixed the keymappings (at least it builds with e200 and c200 now).

Thanks for the torrents, framo, but they don’t seem to be working (getting a connection refused from the tracker and not seeing anyone over DHT).

Pascal Briehl commented on 2008-02-09 00:24

there is my cowon x5 keymapping:

#elif (CONFIG_KEYPAD == IAUDIO_X5M5_PAD)
#define MWW_MENU BUTTON_POWER
#define MWW_SELECT BUTTON_SELECT
#define MWW_END_RECORD (BUTTON_SELECT|BUTTON_REPEAT)
#define MWW_FWD BUTTON_DOWN
#define MWW_FWD_REPEAT (BUTTON_DOWN|BUTTON_REPEAT)
#define MWW_FWD2 BUTTON_RC_VOL_UP
#define MWW_FWD2_REPEAT (BUTTON_RC_VOL_UP|BUTTON_REPEAT)
#define MWW_BACK BUTTON_UP
#define MWW_BACK_REPEAT (BUTTON_UP|BUTTON_REPEAT)
#define MWW_BACK2 BUTTON_RC_VOL_DOWN
#define MWW_BACK2_REPEAT (BUTTON_RC_VOL_DOWN|BUTTON_REPEAT)
#define MWW_PREV BUTTON_LEFT
#define MWW_NEXT BUTTON_RIGHT
#define MWW_RECORD BUTTON_REC
#define MWW_RECORD_REPEAT (BUTTON_REC|BUTTON_REPEAT)

Frank M. commented on 2008-02-09 00:35

Sorry for the outage, the tracker is now back online.
I have also compiled and tested the latest svn with Adam’s mww29.diff for Sansa E2x0, Ipod nano G1 and Ipod video 32Mb.
All these torrents (including the wiki compilations) are available on http://fim.ath.cx:6969

Alex commented on 2008-02-09 09:59

Can someone post a screenshot of this please just to give me the idea of what this looks like. Could someone also please post the size of uncompressed English wiki dump, just to know if I can even fit it into my 30GB ipod :P

Thank you in advance!

Adam Gashlin commented on 2008-02-09 11:09

The converter recompresses each article, so you don’t need a fully uncompressed dump on your device. The English wiki dump that framo is providing via torrent is 4.4 GB.
I’ve attached a screenshot from my iPod Color, using Unifont. It is in link select mode, where you scroll and select a link to follow, which is shown highlighted.

Isaac commented on 2008-02-09 14:56

Ok.. well after memory issues with cygwin, complete distaste for VMWare and its infernal samba file system and finally countless errors upon issue upon errors running feather linux, including it’s still complete lack of NTFS writing capability, Fat32’s inability to even attempt to hold a large file and basically EVERYTHING short of buying a new hard drive to convert the damn thing.. I officially GIVE UP… sigh I guess some people are just destined not to have wikipedia on their iPods…

Frank M. commented on 2008-02-09 16:58

No need to give up yet, have a look at the latest messages where you can find torrents for converted wiki dumps and some precompiled rockbox versions.

Isaac commented on 2008-02-09 22:20

.. I’m on dial up - The torrent is a 13 and a half day download…

Frank M. commented on 2008-02-09 22:35

Fine with me, I can seed until mid March (which is when I move and will be down for a while).

Isaac commented on 2008-02-11 06:14

Ok I’ve had the torrent downloading at school directly onto my iPod (I plugged it into a computer that hasnt currently got a monitor, so no one else can access it) I believe it’s almost done.. thanks Frank and Adam! this will REALLY come in handy for my history course at uni next year - even if all lecturers hate wikipedia :P

kimangroo commented on 2008-02-17 08:33

Hey guys, thanks for the latest patch. Having problems patching though, I keep getting this error:

can't find file to patch at input line 5
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|Index: rockbox/apps/plugins/viewers.config
|===================================================================
|--- rockbox/apps/plugins/viewers.config        (revision 16246)
|+++ rockbox/apps/plugins/viewers.config        (working copy)
--------------------------

I can’t find the right file to patch and when I skip I get the same error about other files as well.

I’m sure I’m doing something noobishly wrong, can someone walk me throught the patching process for ipod 80G?!

Thanks!

kimangroo commented on 2008-02-17 09:05

btw I’m using Ubuntu Gutsy if that makes a difference

Isaac commented on 2008-02-17 09:31

Uhm from what I’ve found you need to have the .diff in the folder above where you usually put patches. It seems to add everything into a rockbox directory… try moving it up a level…

kimangroo commented on 2008-02-17 19:37

Thanks Isaac, but don’t worry it’s just me, I’m dumb.

First of all it turns out I was trying to patch an already compiled version of rockbox (!), and then I had to rename the untarred source code from rockbox2008blabla to just rockbox. Then it all worked smoothly and (touch wood) I finally seem to have wikipedia on my ipod with working links.

Thanks a million people for all the hard work that went into this!

Rafa? commented on 2008-02-20 00:33

The search function needs modification to find articles on the same purpose as in FS#6697. It would be almost perfect then, after implementing that.

Fed commented on 2008-03-15 16:11

I think there should be a faster scrolling method like I implemented way back. It was removed for some reason. This way I think it takes too long to just scan an article. Any ideas?

Isaac commented on 2008-03-17 10:32

Nar I don’t think its necessary… the text moving that fast already causes my music to drop out if I scroll too long.. I just use the navigate menu to get around… My only suggestion for this mod is to get it put into the daily build already! It’s certainly ready for it.. and will encourage more people to download the dump :P

Fed commented on 2008-03-17 13:05

If you could scroll a whole page at a time, your music wouldn’t skip, because it is done in one step rather than many, as with scrolling one line at a time. Also, you could scan an article more easily. I know this because I did it, and it works well. But for some reason this feature was removed. I think it should be put back.

Does anyone else have an opinion on this?

Adam Gashlin commented on 2008-03-19 22:14

It was removed because I’ve been posting the patches I use in my own builds, and I happen to prefer single line scrolling. I understand why you might prefer whole page scrolling, though. It seems like it would be a good thing for a configuration option.

Fed commented on 2008-03-20 00:40

I agree. Do you know how to make a configuration option?

Philip Hansen commented on 2008-03-28 22:08

Hello. I’ve just discovered this wikiviewer, but I can’t compile it for my iPod Video 5.5g 30GB. Everything is compiled except wikiviewer. I keep getting many error messages. I compile in Cygwin for Xp using the files from “mediawikiviewer.tar.bz2” found at the top on this site, and I follow the instructions in the README file. Yet, to no avail. :o( Does anyone have a working copy of this viewer, or help me otherwise?

I think it sounds like a great idea and I plan to make a reference work myself.

Adam Gashlin commented on 2008-03-29 00:24

The latest patch on this page is mww29.diff.bz2, and there are two torrents right above that for English and German converted dumps. I think the latest converter was converter011.tar.bz2 (which you’ll only need for converting something new).
There is a “HOWTO wikipedia.html” for conversion halfway up the page with up to date directions.

Philip Hansen commented on 2008-03-30 10:09

To anyone who might benefit from my experience of compiling RB to include mww: Get mww29.diff. Download the patch utility and install (download from here, http://gnuwin32.sourceforge.net/packages/patch.htm and choose the “complete package, except sources”). Create a folder named “mww” and put the diff file and patch.exe in there.

I use Cygwin to patch. In Cygwin go to the mww folder and type “patch < mww29.diff”. Patch now creates the needed mww files in this folder.

In my case it asked me where the viewers.config and SUBDIRS files were. So I patched these manually by doing this:
Open rockbox/apps/plugins/viewers.config in TextEdit (I’m using a mac) or another textedit programme, and insert this line “wwi,viewers,mww,-” after “wav,viewers/test_codec,-”. Save and exit.
Open rockbox/apps/plugins/SUBDIRS and insert this line “mww” after “reversi”. Save and exit.
Open rockbox/apps/plugins/CATEGORIES and insert this line “mww,viewers” after “mpegplayer,viewers”. Save and exit.

Next, move the “mww” folder to rockbox/apps/plugins and start compiling.

I used this procedure and it finally worked! And I was very happy :-) Originally I mixed up the old mww files from “mediawikiviewer.tar.bz2” and that messed everything up. Finally I got the picture. Trial and error.

Now I just have to figure out how to convert dumps, because I can’t compile the converter in Cygwin. Gonna try Linux on my mac and see if that helps.

Adam Gashlin commented on 2008-04-02 05:17

In case anyone is curious about where I’d like to see the project go, here’s a suggested todo list I wrote up for someone.

Andrew Slavozchikz commented on 2008-04-23 19:37

A program to manage the dumps, syncs and updates would be VERY useful (like AvantGO for the pdas, in that style.

Like, i.e., call the sub revision in the wiki of one article, and is newer, download and copy the dump to the plugin mww folder

Delyan Kratunov commented on 2008-04-23 20:08

The plugin requires a sync after the recent change in the plugin menu playback control. I lack the development experience to do the change, though.

Alexander Papst commented on 2008-04-25 18:35

change every call of playback_control(rb); to playback_control(rb, NULL);

Adam Gashlin commented on 2008-04-25 19:24

Something else needs changing, it compiles alright but I don’t know where the binary is going.

Admin
Frank Gevaerts commented on 2008-04-25 20:38

Fixed to make it compile on recent (r17248) svn. No other changes, and I don’t know if it works

Adam Gashlin commented on 2008-04-26 00:27

Works fine on my iPod, thanks.

Delyan Kratunov commented on 2008-05-18 10:17

After a recent update, mww yields tons of redefinition errors. A sync would be much appreciated.

Adam Gashlin commented on 2008-05-19 17:55

*Tons* of redefinition errors?

Sync’d.

Thomas Martitz commented on 2008-05-24 18:43

Delian, you have probably applied the patch twice, so that you have the files which get created with the patch twice (twice in one file). Get a new source, or remove the patch and delete the files created by this patch manually.

Isaac commented on 2008-07-21 13:35

Getting an error on hunk #1 or something to do with subdir when trying to patch. Need resync?

Marcin Łukasik commented on 2008-09-21 14:54

When I’m trying to compile Rockbox with this patch, I receive compilation error:
MAKE in mww
CC mww.c
mww.c: In function ‘set_article_offset’:
mww.c:386: warning: implicit declaration of function ‘printf’ mww.c: In function ‘viewer_init’:
mww.c:1103: error: too many arguments to function ‘rb→splash’ make[3]: * [/home/Moarc/rockbox/build/apps/plugins/mww/mww.o] Error 1
make[2]:
* [mww] Error 2
make[1]: * [rocks] Error 2
make:
* [build] Error 2

How can I fix that? Will you fix that in new version of plugin?

Thomas Schott commented on 2008-09-21 20:57

Moarc: sync to SVN ;-)

The wwi viewer doesn’t work in simulators because of:
> dlopen(archos/.rockbox/rocks/viewers/mww.rock): archos/.rockbox/rocks/viewers/mww.rock: undefined symbol: sim_write
at runtime. I couldn’t find the sim_write call in the mmw sources, seems to be in an included file.

But anyway, it works on my iPod and I love having wikipedia in my pocket :-) THX!

Xinlu Huang commented on 2008-11-22 20:05

The recent revamp of the make system (r19146) broke this patch; make fails with
make: *** No rule to make target `/data/rock/rockbox/apps/plugins/mww/mww.make’. Stop.

I know next to nothing about makefiles etc. (always avoid that like plague… ;) ). Can someone who knows about makefile fix this (looks like a quick fix for a makefile guru)? Many thanks!

Thomas Schott commented on 2008-11-29 10:02

Sync to SVN.

Admin
Björn Stenberg commented on 2008-12-01 13:11

I’m prepared to nurse this into SVN. However there is a couple of things that needs fixing before it is ready for prime time:

1) MWW is a cryptic name. We don’t like cryptic names. Please rename the plugin wikiviewer or wikireader or some such. The Mediawiki reference is not relevant since the plugin is not reading the mediawiki file format anyway, only the converter is.

2) Add keymaps for all targets in http://build.rockbox.org, or at least add #ifdefs in apps/plugins/SUBDIRS so the plugin isn’t compiled on targets where it will fail.

3) Clean up the code:
3a) Remove all #define read rb→read and such. simprint too.
3b) The code is too dense. Keep one statement per line, sprinkle with whitespace and split long lines at 80 columns.
3c) Indent all code with spaces, not tabs.
3d) Remove unused code.
3f) Replace // comments with /* */
3g) Change #include “../lib/..” to simply #include “lib/…”

4) Include the converter source code in the patch, since it is essential to the plugin. Put it in a subdirectory under tools.

5) Make sure each sourcefile has a rockbox header (except the imported gzip source files).

Adam Gashlin commented on 2008-12-01 18:33

Is anyone interested in tackling this? As much of the viewer is my code I’d probably have an easier time of it, but if someone else volunteers it’ll probably get done sooner (can’t even guess when I’d get to it).

Matthias Larisch commented on 2008-12-01 19:56

I already started doing some parts of Bjorns list. I will add a patch today or tomorrow that covers some points. Rest of work is up to another one :-)

Xinlu Huang commented on 2008-12-01 23:34

I can do the rest within two weeks or so (after my rounds of final exams) - if no one gets to that before.

Matthias Larisch commented on 2008-12-02 12:50

I forgot to ask how we should call this plugin? I would stick to wikiviewer as suggested by Björn but as it mainly is the work of Adam he has to decide the name :-)

Adam, what do you think?

Matthias Larisch commented on 2008-12-02 18:43

Here it is. I renamed everything to wikiviewer.
My changes:
-rename
-some cleanup (defines, more white spaces, line breaks, spaces instead of tabs, removed unused code, replaced all comments, changed includes)
-included converter code (with new makefile, some cleanup)
-added some rockbox headers (maybe not really correct - please check me)

Compiled code ran on my H320 as the original (this should be like that because I did not change anything)

Matthias Larisch commented on 2008-12-02 19:03

Sorry for the incomplete diff. svn under cygwin is really crap for developing. Seems that I should change to linux soon :-) Here is a diff generated by TortoiseSVN, not perfect but works:

Adam Gashlin commented on 2008-12-02 21:23

Yeah wikiviewer sounds like a good name, glad to see this getting somewhere finally.

Edward Hyer commented on 2008-12-10 04:54

Excited by this project!! Got patch, re-compiled, so far so good.
Could not compile converter (Ubuntu Hardy):
jo:~/rockbox_ipod55/rockbox_source/rockbox/tools/converter$ make
gcc -o xmlconv -O3 xmlconv.c -lpcre -lz -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE
xmlconv.c:33:18: error: pcre.h: No such file or directory

Marcin Łukasik commented on 2008-12-10 05:03

ejhyer, you need to install pcre-dev or libpcre-dev. (I don’t know how it’s named in Ubuntu)

Edward Hyer commented on 2008-12-10 05:44

OK, got the libpcre, was able to compile the converter. Now it’s cranking, and I’ll find out in the morning what I have.

Edward Hyer commented on 2008-12-11 03:52

OK, one source of confusion. If I have the new stardict plugin, and I have used the converter, do I actually need the wikiviewer plugin?

Edward Hyer commented on 2008-12-11 04:14

Hmmm… if I *don’t* use the stardict plugin, where do I put the converted wikipedia files?

Adam Gashlin commented on 2008-12-11 04:24

I don’t know how the stardict plugin works, but the wikipedia viewer compresses the files, and the stardict plugin probably doesn’t support it (nor the indexing system).
To use the files, put the generated .wwi and .wwa files in the same directory on your device. The viewer should be set up as a viewer for .wwi, just open that file and you should be able to use it

Edward Hyer commented on 2008-12-11 05:56

Success! I got it working! Very, very happy. Now I’ll complain: the text entry system (ipod video) is just terrible. Can I modify it?
Anyway, thanks everyone for your work on this and help with installing, I’m going to play a bit now.

Adam Gashlin commented on 2008-12-11 06:28

Sure you can modify it, you have the source. The index tree should lend itself naturally to a nicer system that searches as you’re typing, with the bonus of autocompletion, just scroll among the possible letters for each place.

Edward Hyer commented on 2008-12-11 07:39

OK, here are my observations after playing a little. Most of these are Rockbox-in-general, but some maybe can be fixed in wikiviewer, and hopefully some I’m just doing it wrong, and someone can set me straight.
1) On startup, I get the keypad to enter a search term. Why can’t I go directly to the menu, where I can start at a bookmark or something?
2) Search is case-sensitive. Must this be?
3) The only font I could find which was unicode-capable was the gnu-unifont. Are there any other unicode-capable fonts ported to Rockbox?

OK, that’s all I got for right now. Thanks again for this super-cool piece of hackery, I’m having lots of fun.

Adam Gashlin commented on 2008-12-11 15:55

1) You have to “save history” in the menu if you want to return to a session in progress next time
2) I don’t think search is case sensitive
3) not that I know of, but I haven’t been paying attention for a while

Timo Horstschäfer commented on 2009-01-01 23:15

I had a look at this plugin again and really liked it! Great work!

I used to have my own stardict plugin for rockbox to read wikipedia and of course stardict dictionaries on my player.
To keep these dictionaries with wikiviewer I just wrote a hackish stardict to wikiviewer converter. It’s name is stardictconv and does the same job as xmlconv does but for stardicts (get them at http://stardict.sf.net).
It contains a lot of files from the original stardict, so I put them in an archive instead of a single patch (and the stardict files are GPLv3 so I’m not sure if they fit into RB anyways…).

If you’re interested I could put an archive of stardicts in wikiviewer format online.

Robert Kukla commented on 2009-01-18 14:23

- resync
- don’t include bbfuncs.c, but compile separately and link it
- buttons for x5, c200 and mr100

Seth commented on 2009-02-04 22:21

does any one have a pre patched version of this(or maybe even compiled) for the iPod Video 5.5G 30GB, because i cant get the .patch patcher program to work at all? If some one has a pre-patched version i can compile it!
If no-one does though, i understand, I know its kind of “beggish”, =).

Adam Gashlin commented on 2009-02-04 23:07

Here it is as a .tar.gz of the changed and new files.

Seth commented on 2009-02-05 03:22

wait, what is that, is it just the wikipedia convertor and wikipedia plugin for Rockox, just want to make sure.

Adam Gashlin commented on 2009-02-05 08:07

Yes, you would extract that into the clean rockbox source directory, overwriting existing files and creating new ones.

Seth commented on 2009-02-05 13:10

followed by compiling!
Thanks abunch for doing that, i have wanted wikipedia on my ipod for like ever. Now i have to try and figure out how to download like a 2Gb+ file with dial-up, hmm….. maybe my neighbors will let me download it on their computer(with road runner turbo)…. i’lll manage to get it done!

Seth commented on 2009-03-26 12:20

ok, i have another question with this patch, for i finally took the time to compile it and load it to my iPod. SO now, my iPod can view the wiki files, but i dont know how to convert the wikipedia dumps into the correct .wwi and .wwa formats. Is there a conversion program out there OR maybe instructions on how to convert the wikipedia dumps OR is there a dump pre converted that can be downloaded?

Adam Gashlin commented on 2009-03-28 06:05

I think this was the latest converter: http://www.rockbox.org/tracker/task/4755#comment13476 Building the converter should just be a matter of running the “compile” script to build the xml processor (xmlconv) and “Make.sh” to build the index builder (btcreate).
For converting, the first step is:
bunzip2 -c enwiki-XXX-pages-articles.xml.bz2 | ./xmlconv - wikipedia
assuming “enwiki-XXX-pages-articles.xml.bz2” is the dump, and that you want the resulting files to be called “wikipedia”. That will build the wikipediaN.wwa files (containing compressed article text), wikipedia.wwt (titles and indexes into the article data) and wikipedia.wwr (redirects).
The second step is:
./btcreate wikipedia.wwt wikipedia.wwr wikipedia.wwi
Which creates the index file wikipedia.wwi. Then just copy the .wwi and .wwa files to your device.

Sorry for editing this so many times.

Seth commented on 2009-03-29 23:46

ok thanks, but wouldnt it be more simple for everyone if we just put a pre converted wiki dump up on like megaupload or rapidshare? i mean, it would surely save a lot of processing power and quicker than converting it yourself (especialy for those of us with a slower computer)?

IDK, just my thought.

But thanks anyways for giving me those instructions on how to convert it.

Jerry Chapman commented on 2009-07-01 16:14

Needs a resync

Jonathan commented on 2009-07-05 20:35

In case anyone is interested, I’d be willing to upload my recent Wikipedia dump (7/02/09) in wwa and wwi format if someone can suggest a good filehost.

Cristián Quezada commented on 2009-07-06 02:43

I don’t know any filehost that allows files that big, but if you consider making a torrent I can seed it for some time.

Jonathan commented on 2009-07-06 02:48

Okay. I’ll try to get to it soon, and maybe send you a pm with the torrent link. Don’t forget though, the max size of each file is 1 gig, so maybe I could do it still.

Steve Bavin commented on 2009-07-06 07:09

Could you not post the link here? More people = better torrenting…

Jonathan commented on 2009-07-06 16:00

Sure, but I need a suggestion for a site. I don’t do much file hosting, much less torrents.

Adam Gashlin commented on 2009-07-06 18:57

I don’t know of any reason not to use a major site like The Pirate Bay, unless you have concerns over guilt by association. Maybe try the various fun stuff listed here:
http://azureuswiki.com/index.php/Legal_torrent_sites but I’ve used tpb for wiki stuff in the past.

Maurus Cuelenaere commented on 2009-07-07 12:17

This is a cleaned up version (removed TABs, make lots of functions static, added (basic) keymap for touchscreen targets and remove unneeded code) + some optimisations (html entities are now binary searched).

Jonathan commented on 2009-07-07 19:04

Tell me if this worked. I have never created a torrent before. I’ll probably be seeding all day. If someone else could download it and do it right, it would be much appreciated.

Adam Gashlin commented on 2009-07-07 23:04

It seems to be working fine, thanks, I’ll keep it seeded, too. It would help if in the future you include the dump date and language info in the torrent, preferably in the directory name or a comment.

Jonathan commented on 2009-07-07 23:18

Ok, now I know. Yeah, it would have been good to include that there. I did say it on this page, but not everyone will see that. I’m just glad to finally contribute something to Rockbox. :)

Frank M. commented on 2009-07-10 19:01

Hi Jonathan, the torrent is stuck at 84% - could you kindly seed a bit longer?
Thanks, Frank

Jonathan commented on 2009-07-11 16:48

Yeah, sorry. Whenever I shut it down at night, it stops, and I often forget to turn it back on the next day. It’s up again.

Cristián Quezada commented on 2009-07-11 23:10

The new Wikipedia dump works great with the updated patch. Thank you Jonathan and Maurus! I was out of town so I couldn’t seed, but now I’m back.

Dave Chapman commented on 2009-07-11 23:34

A few comments if the intention is to commit this patch:

1) tools/ is probably not the best place for the converter program (it’s intended for tools required to build Rockbox). Maybe utils/ would be better?

2) “converter” is too general a name - how about something like “wikiconv” ?

3) Lots of files are lacking names of authors/copyright holders. I don’t think they can be committed to Rockbox like that.

4) Documentation refers to “freqmod” - can that be changed to a real name?

5) Some files still contain the old version of the Rockbox license text (before the change to explicitly say “v2 of the GPL or later”). Other files don’t include that text at all.

Maurus Cuelenaere commented on 2009-07-12 02:51

I suppose in the end, the intention is to commit this (it still needs reworking).

1) I agree
2) sounds better
3) yes, I also wondered about that; there’s lots of anonymous/not credited code
5) should be doable

Gman commented on 2009-07-13 20:50

I have problems compiling this on my 5th Gen.

Gman commented on 2009-07-15 22:07

It seems it needs a resync?

Nick G commented on 2009-08-05 23:50

ok i’m trying compiling Rockbox for the first time since i’d like to try some of these patches out there. I’ve read through how to compile and successfully compiled the standard SVN from today without error. Since i’m new to all this, what is the base file from here that i need to start with regarding the wikipedia patch. Do i need to grab every file from the 1st posted and go in sequence?

Nick G commented on 2009-08-06 03:06

well actually i’m reading the diff and i see this:
diff –git a/apps/plugins/CATEGORIES b/apps/plugins/CATEGORIES
index 8e4bb7d..d8235c1 100644
— a/apps/plugins/CATEGORIES
+++ b/apps/plugins/CATEGORIES

i noticed there is no a or b directory. Could that be why? Please excuse my noobishness
.

Nick G commented on 2009-08-06 04:53

sorry about the double-post.

I decided to give it a shot and remove references to the directories a/ and b/ this is what i got at compiling the 8/5 SVN with the patch:

LD lua.rock
make: *** No rule to make target `/root/rockbox/ipod/liplayback_control.h’, needed by `/root/rockbox/ipod/apps/plugins/wikiviewer/wikiviewer.o’. Stop
obviously i’m missing something…

Thomas Martitz commented on 2009-08-06 11:19

a/ and b/ comes from git (a version control system) when creating patches with it. You need to run “patch -p1 < file.patch” to strip a/ and b/ automatically.

Nick G commented on 2009-08-06 16:50

ok did that but now i’m getting something different:

CC apps/plugins/wikiviewer/wikiviewer.c
/home/nicholas/Projects/rockbox/apps/plugins/wikiviewer/wikiviewer.c: In function ‘render’:
/home/nicholas/Projects/rockbox/apps/plugins/wikiviewer/wikiviewer.c:621: warning: unused variable ‘colorem’
/home/nicholas/Projects/rockbox/apps/plugins/wikiviewer/wikiviewer.c: In function ‘plugin_start’:
/home/nicholas/Projects/rockbox/apps/plugins/wikiviewer/wikiviewer.c:1765: error: ‘struct plugin_api’ has no member named ‘strncpy’
make: *** [/home/nicholas/Projects/rockbox/ipod/apps/plugins/wikiviewer/wikiviewer.o] Error 1

btw i’m building the ipod 5g (30gb) variation

Maurus Cuelenaere commented on 2009-08-06 17:25

The rb→strncpy() error is due to the replacement of strncpy() with strlcpy() in core, you can either change it to use the pluginlib one or make it use rb→strlcpy().

Vergil Weatherford commented on 2009-08-09 18:34

Maurus and Nick G,

I’m compiling the latest wikiviewer patch (posted by Maurus on July 7) for the first time, with today’s rockbox build. I also ran into the ‘strncpy’ error, saw the comments here, and changed rb→strncpy() to rb→strlcpy() in wikiviewer.c and recompiled.

upon recompiling, I ran into another error:

In function `render’:
wikiviewer.c:(.text+0x19a0): undefined reference to `LCD_RGBPACK’

any ideas what I might do to fix this error?

Maurus Cuelenaere commented on 2009-08-09 23:33

LCD_RGBPACK is a macro which is defined in lcd.h I think, so adding #include “lcd.h” to wikiviewer.c should do the trick.

I wonder though how I didn’t get that error, what target are you compiling for?

Vergil Weatherford commented on 2009-08-10 00:25

ipod mini 2nd generation, with an upgraded (extended life) battery and a 16gb flash card instead of the micro-drive.

6gb=wikipedia
10gb=music

sweet!

Vergil Weatherford commented on 2009-08-10 00:44

Maurus,

I added #include “lcd.h” to wikiviewer.c and then recompiled, but I still get the same error. Thanks for taking a look.

Jeff Wood commented on 2009-08-10 02:24

I am trying to compile the latest version of this viewer for an ipod 5g and keep running into troubles. Make keeps failing with the error:
make: ***No rule to make target ‘/home/Jeff/rockbox/apps/plugins/lib/oldmenuapi.h’ needed by ‘/home/jeff/rockbox/wiki/apps/plugins/wikiviewer/wikiviewer.o’.
Stop.
Any idea’s why this is happening something simple and stupid i’m missing that’s causing it? I’ll admit to having pretty much no idea what i’m doing. I ran enough linux software in the past to have no trouble getting the enivronment setup and making a basic build for my ipod but as soon as the compiler starts throwing errors I get lost.

Nick G commented on 2009-08-10 03:37

Jeff - Look at Thomas’s post just below mine on august 6th ^_^. Changed rb→strncpy() to rb→strlcpy() in wikiviewer.c, like Maurus says, and patch it with the command Thomas mentioned and you’ll be good to go to compile. That worked for me and it compiled fine :)

Jeff Wood commented on 2009-08-10 16:25

Nick - I tried changing rb→strncpy() this morning before heading to work but still got the same error. I think when I get home i’m gonna grab a fresh copy of the source code, patch as instructed above, and make the rb→strncpy() change. Then I’ll try a fresh compile and see where I get.

Jeff Wood commented on 2009-08-10 22:08

Started from scratch and it worked this time. Thanks for the help. In case others who don’t know what they’re doing have this same problem and can’t manage to read like me. I did:

1. Got a fresh copy of the source from svn (the one i screwed up patching originally I couldn’t seem to get to work)
2. patched using the wikiviewer.diff above using patch -p1 > wikiviewer.diff
3. edited apps/plugins/wikiviewer/wikiviewer.c and changed rb→strncpy() to rb→strlcpy()
3. compile rockbox as usual

Played with it and it’s definitely very cool to have on the ipod. Now I just gotta figure out a better keymap or someway to adjust that so it’s not impossible to type in what I’m searching for. Great plugin thanks all.

Rosso Maltese commented on 2009-08-10 22:21

The right command is: patch -p1 < wikiviewer.diff (”less than” instead of “greater than”).

As general note: since I don’t make a normal development activity, I wrote a script that first update my local source tree, then applies my selected patches, then build, and finally remove all the patches (if you apply a patch with “patch < file”, you can remove it with “patch -R < file”).

Vergil Weatherford commented on 2009-08-10 22:25

Still having troubles with the LCD_RGBPACK error on compile. Does anyone have any suggestions? On my ipod mini I don’t have a color screen, so is there anything wrong with commenting out that section in wikiviewer.c? I’m not much of a programmer, so I am not sure if that would have a ripple-effect the rest of the compile.

Would love to get this up and running! Thanks in advance for the help.

TruthTaco commented on 2009-08-10 23:31

I’ve tried to install this patch several times, done the things suggested above, and i still get these errors http://pastebin.com/m46ec1a5f Any suggestions?

G.P. commented on 2009-08-11 08:09

synced (changed strncpy to strlcpy), and added sansa fuze support. Record doesn’t work on the Fuze, because all the Buttons were already used. Any Suggestions on how to record on the Fuze? (I’m new at Rockbox)

TruthTaco, it looks like you are building for an unsupported target (or for the sansa fuze, which is now supported)

TruthTaco commented on 2009-08-11 17:47

Thanks fuze support was what i needed. Record function has not been added to the fuze yet.
I still get a few errors http://pastebin.com/m1baf2f70 but i think im just going to get a friend to compile it for me.

Vergil Weatherford commented on 2009-08-14 01:13

So trying again today, I started from scratch with the latest patch from TruthTaco. I added “#include “lcd.h” to wikiviewer.c, and compiled. Unfortunately, I ran into the same error as before: (undefined reference to `LCD_RGBPACK’). Is this because I’m trying to build for an unsupported target (ipod mini 2nd gen)? Is there any way to work around this?

Gman commented on 2009-08-17 23:48

I get this error when compiling it on my 5G, it is something about a, “wikiviewer.make” It stops it from compiling. Why is this?

Jerry Chapman commented on 2009-08-18 00:09

It looks like the last part of wikiviewer.c got clipped from the patch made on Tuesday, 07 July 2009, 07:17 GMT-5 by Maurus Cuelenaere (mcuelenaere). If you scroll back and get a copy of that patch, you will see what I mean. I was getting the same error as TruthTaco until I copied and pasted the last few lines of Maurus’ old patch.

It is in the last function, starts with:
enum plugin_status plugin_start(const void* file)

Just copy that function over from the old patch to wikiviewer.c Don’t forget to change the one instance of rb→strncpy() to rb→strlcpy()

Jerry Chapman commented on 2009-08-18 00:52

Ok, I prepared a patch so you don’t have to copy and paste. This compiles for me cleanly now on my Ipod Video.

Gman commented on 2009-08-18 01:16

Thanks man!

Marianne Arnold commented on 2009-09-01 00:16
Comment by Dave Chapman (linuxstb) - Saturday, 11 July 2009, 23:34 GMT
> 4) Documentation refers to “freqmod” - can that be changed to a real name?

I don’t know if the problem is getting the real name or about actually doing the work to change it. If it’s the former: freqmod’s real name is Frederik Vestre and he’s already in the credits for a Norwegian update, see http://svn.rockbox.org/viewvc.cgi?view=rev&revision=9930 - and later contributed a few more small patches. Just when I wrote this, I could still see him around in IRC - #rockbox and a some other channels, I’m quite sure it’s him so if you really want to contact him, there would be a way…

Seth commented on 2009-09-09 21:30

has anybody here tried converting a copy of simple Wikipedia or Wiktionary with the converters. I would, but I dont know how to use the converter program (havent found any instructions on how to use it if someone could give exact instructions I wouldnt have a problem converting it, and uploading it somewhere).

The reason I say this is because both of those would be smaller, and the simple wikipedia can be useful in a situation where you have a small player/ don’t have a lot of space you want to give up and for times when you want to just get a more generalized feel of the article/topic.

The converter should work, as they both use the same wiki type and are both in the XML format which the standard Wikipedia uses.

Gman commented on 2009-09-10 05:13

I would too like to know how to work the converter :(

Robin Bertram commented on 2009-09-11 12:31

in the converter folder, there is a README file that helped me compiling and converting

I had to install zlib and pcre first to be able to compile the converter succesfully, but perhaps that’s not neccesary… you’ll see
I also compiled the stardict converter that can be found here: http://www.rockbox.org/tracker/task/4755#comment27464

now the new dict plugin is redundant for me… although I like it’s search method better

PS: I can upload dewiki and some converted dictionaries, if needed

MKL commented on 2009-09-16 16:18

I can’t patch the dict file, do I need to have the source code or does the dict.rock file work with cygwin?

Might it be possible to upload a finished file (dict.rock) and the dewiki or frwiki dict files

(I would prefer some ready-to-copy-on-device-files, because i can’t patch

Robin Bertram commented on 2009-10-18 22:57

I uploaded the converted wikipedia dump (2009-08-10) (about 2.1GB)

here the netload and the rapidshare links
http://netfolder.in/folder.php?folder_id=X5og6rk http://rapidshare.com/files/294818716/dewiki-20090810.txt

Robin Bertram commented on 2009-10-18 23:14
Robin Bertram commented on 2009-10-18 23:15

oh sry… forgot to mention

it’s only dewiki

Seth commented on 2009-11-08 15:20

i know earlier on in this FS, it was mentioned that it is not case sensitive when you search for things.

If you type in ‘GUI’ on rockbox’s wikipedia it brings you to a food, because the case is ignored. Whereas you would have wanted it to go to “Graphical User Interface”. Thats just one example, I am sure there are more examples.

One thing I thought about how to solve this is to first search for the article in the case that it was entered. So it would search for GUI first, and then if nothing appears remove the case sensitivity and search for the article.

Adam Gashlin commented on 2009-11-08 18:16

Well, it is supposed to do that. On 07 November 2006 I said: “Text sensitive matching is now attempted before insensitive matching, if there is an exact match we should use that”. It may be that the search is messed up, but what you suggest was the intention.

Seth commented on 2009-11-08 22:23

hmmm….

I just tried it again to verify what I had previously reported. It did the same thing when I searched for “GUI“, it brought me to some korean dish/food. Then one i went to “find similar article” in the menu, it brought me to the Virtual Keyboard, displaying “Gui”, even though GUI was entered. On Wikipedia, GUI goes to: http://en.wikipedia.org/wiki/GUI and “Gui” goes to http://en.wikipedia.org/wiki/Gui .
Can you or someone else verify this?

Any ideas why that might be happening then?

Maurus Cuelenaere commented on 2010-01-04 01:03

I’ve made some changes to this in my git tree ( http://gitorious.org/mcuelenaere-rockbox/mcuelenaere-rockbox/commits/wiki_viewer ).

I’m planning to clean the patch up, add touchscreen support and perhaps add LZMA support.

Robin Bertram commented on 2010-02-19 15:59

here is a new version of dewiki

I only seed as long as my Notebook is on

Alex commented on 2010-02-27 23:59

Does anyone have a pre-compiled version of the Stardict → Wiki converter?

David Lorch commented on 2010-03-29 23:15

This plugin is amazing. I hope that it will eventually make it into SVN!

To everyone who’s interested: I have uploaded a conversion (.wwi and .wwa files, 7.4GB) of a March 2010 English Wikipedia dump to RapidShare.
The archive consists of 37 parts, 200MB each, the links to which are given in the attached file. The easiest way to download all of them is to copy&paste the links into jDownloader (www.jdownloader.org).

The source XML file was: http://download.wikimedia.org/enwiki/20100312/enwiki-20100312-pages-articles.xml.bz2

Robin Bertram commented on 2010-06-19 14:23

thanks mcuelenaere for your work in your git tree
I synced your changes and removed an ifdef because ONDA_VX747_PAD is not defined in svn

Robin Bertram commented on 2010-06-19 16:41

ONDA_VX747_PAD is ONDAVX747_PAD in svn
I think I should change that instead of removing it

Rosso Maltese commented on 2010-07-08 10:50

Sync against r27345.

Tom commented on 2010-07-12 18:21

Thanks Rosso for the updated patch. I creaderd a new rockbox github project and in one branch lives that patch:
http://github.com/tomka/rockbox/tree/wikiviewer

Matteo Italia commented on 2010-08-28 15:45

The last patch gives problems with the current build, since PLUGINLIB_EXIT_INIT does no longer exist, replaced with some new machinery that involves the new plugin_crt0.c. If I understood correctly what it did, I think it could be removed without much fear, but since I’m quite new to RockBox development I’d like to have confirmation from someone who knows before posting a new patch.

Tom commented on 2010-10-08 16:08

No rockbox guru here as well, but besides the suggestions you made Matteo I think some more USB-Event and exit related things have to be changed. With git it is commit 950ac28c3ff7e339bd388314ddb0a7f12bc0c74 which introduced plugin.crt0.c. With viewing at the changes of that commit I saw a couple of things that were changed in other plugins with similar structure. Well, I’ve added them and my changes are available on github repo from two posts ago. I try to veryfy that these changes are correct.

Rosso Maltese commented on 2010-11-12 15:38

Synced against r28563.

Haim commented on 2011-03-12 22:16

Hi, It will be possible to add support for RTL languages​​? (Like Hebrew and Arabic ) so the text will be aligned to the right instead of left
If so, it will help me a lot,
thanks, Haim.

sideral commented on 2011-06-06 23:27

Looks like we actually do know the original contributor’s real name – to be confirmed.

Justin Hannigan commented on 2011-06-15 19:57

Haim, RTL support is in the core but isn’t available to plugins at the moment.

Haim commented on 2011-06-16 21:02

Justin, Yes, I know, I tried to change the code of this plugin to add RTL support for this plugin, but I couldn’t do it, so I asked if anyone knows the code of this plugin can do it for me (if it’s not too hard…)
thanks again.

Christoph commented on 2012-04-01 12:39

I just added keymappings for the Sansa Fuze+. Find them in the attached patch (if anybody is interested). It is based upon 4755-plugins-Wikiviewer.0.patch.

Thanks for providing this plugin. I have been using it on my Sansa e250 and am now using it on my Sansa Fuze+ and it is working perfectly fine on both. It might be worthwile to consider adding this to official rockbox releases.

Haim commented on 2012-05-06 02:02

Not working on 88aeef9 build

Avi commented on 2013-03-19 17:47

If anyone is interested in the converted dumps , I have them and am trying to uploading them now .
The wwi file for February 4 ,2013 (466.8MB) is at http://www.freefilehosting.net/wikipedia .
When I upload the wwa files I will try to post links then.
The full dump is 10 files that are 1GB each and 1 that is 209MB.
It took about an hour and a half to convert (not including downloading and extracting) on a dell core-i7 with 8GB RAM.
You’ll need about 55-60 GB free to do it yourself

Avi commented on 2013-03-19 17:53

By the way the patch program didn’t patch the last few lines of wikiviewer.c or viewers.config .
I did it manually and have included them.
After using the patch, replace /apps/plugins/wikiviewer/wikiviewer.c and /apps/plugins/viewers.config with my file.

Avi commented on 2013-03-19 18:44

This should be the folder link (right now has the first 2 of 11 files) :
https://mega.co.nz/#F!pg431CDB!aeAo_HT3YngdsxBzYuoksg Please let me know if it doesn’t work.
It will get updated automatically when the rest are finished.

Avi commented on 2013-03-20 17:41

All the files should be up now.

Avi commented on 2013-05-27 03:58

This is a patch that should apply clean. Also I’m uploading the may 3 2013 dumps of wikipedia now, it should be on the mega link above but it’s not all there yet.

G.P. commented on 2013-05-31 06:25
As you can see at http://www.rockbox.org/irc/log-20130527#04:20:33 I am
> trying to put the wikiviewer into the rockbox core and they need official
> permission to use your code. I got 8 your emails addresses from
> http://www.rockbox.org/tracker/task/4755 . If any of you have objections
> to
> this please email me. If you are fine with it and release your code under
> the GPL, then the rockbox project needs your full names. Also please post
> any response to the tracker page . Thank you.

I hardly did anything on this patch, but for what it’s worth:

I have no problems with the release of this code under the GPL license. My name is Gregor Pohl.

Thomas Martitz commented on 2013-05-31 08:24

The mallocer.c should be removed before inclusion. Rockbox has a real malloc() for plugins, so if it this plugin relies on malloc() it should use this instead of creating a dumb malloc as the patch currently does. (I haven’t examined why it relies on malloc but it would be best if it didn’t need it at all)

Loading...

Available keyboard shortcuts

Tasklist

Task Details

Task Editing