This is the bug/patch tracker for Rockbox. Click here for more information.
Quick links: Bugs · Patches · Rockbox frontpage
FS#6697 - New dict plugin
Attached to Project:
Rockbox
Opened by Timo Horstschäfer (x1jmp) - Sunday, 25 February 2007, 20:43 GMT+2
Opened by Timo Horstschäfer (x1jmp) - Sunday, 25 February 2007, 20:43 GMT+2
|
DetailsThis is an almost complete rewrite of the current dict plugin.
It is based on the stardict dictionary file format and acts as a viewer for .idx files. Most important features are: - fuzzy search by showing all articles that start with a certain word - quite fast search (binary search, same as the current dict plugin) - scrolling for long descriptions - handles even huge dictionaries (e.g. Wikipedia) When first opening a dictionary, an offset table is created to provide random access to the index file. Stardict dictionaries can be used almost directly but all files have to be uncompressed. I created another patch, which adds a "Dicionaries" entry to the root menu (FS# 6630). It shows all dictionaries placed in "/dicts". Some more information may be found in a thread about an older version on MisticRiver http://www.misticriver.net/showthread.php?p=546738 Some dictionary ressources: - Stardict Website: http://stardict.sourceforge.net/Dictionaries.php - Wikipedia dumps: - english: http://srv2.filepoint.de/download/1457/enwiki-20061130.7z - german: http://srv2.filepoint.de/download/407/dewiki.7z 
dict.patch
(28.3 KiB)
apps/plugins/viewers.config | 1 apps/plugins/dict.c | 951 ++++++++++++++++++++++++++++++++++---------- 2 files changed, 753 insertions(+), 199 deletions(-)
root_menu_dictionaries.patch
(3.7 KiB)
apps/lang/english.lang | 14 ++++++++++++++ apps/recorder/icons.c | 1 + apps/recorder/icons.h | 1 + apps/root_menu.c | 9 ++++++++- apps/root_menu.h | 1 + apps/settings.h | 1 + 6 files changed, 26 insertions(+), 1 deletion(-) |
This task depends upon
I patched the dict.patch but the other one doesn't work
I extracted a couple of star dics and it lets me select them and search but when I select an entry it says that msg above.
Great work though I'm prob doing something wrong!
So uncompress the description file e.g. with
gunzip -S .dz *.dict.dz
It's just that there must be a .dict file with the same name.
The second patch requires the root menu patch (FS# 6630) to be applied as well.
I hope it'll work for you.
Also I had the problem that after scrolling down to the bottom of the text the next press on up cleared the screen apart from a number of squares in the top row. Pressing the menu button got me out of the mess.
Unfortunately I haven't found a way to reproduce the problems reliably. :(
But I don't know how much different the memory management on the player and the sim is.
Furthermore, there was another bug. After exiting the menu, the first page of the article was drawn instead of the one you were viewing before going into the menu.
I'm going to take a good hard look at your plugin to see if I can come away with anything interesting...
Where do you get the wikipedia dump in the format that you have for download? A lot of work we've been doing has been on a converter from the XML dumps to an archive with each article gzipped and with control codes for things like links and formatting.
However, at the beginning I didn't like it, because it was iPod only, though this has changed now and that it's limited to Wikipedia ATM.
My intention was to write a new search-only dict plugin and create a general document viewer for the articles.
I think working on a viewer is more attracting than working on a Wikipedia-only plugin.
The wikipedia dump was converted using the converter from stardict-tools. It doesn't support any kind of formatting.
Also the plugin doesn't support compression. I looked at the mww dumps and the size of the English Wikipedia with each article gzipped was around 800 MB less then uncompressed.
The Stardict (not the one for Rockbox) itself as well as the DICT project use special, but still standard-compliant, gzip files to allow random access to them. Compressing the whole dump should decrease the size much better, but I'm not sure, whether compression is needed at all.
I would very like to join the two wikimedia programs, on concentrate working on one. It's definitely a great feature for Rockbox and everyone who may want to switch to Rockbox.
In the last days I wrote such a viewer and added a basic formatting engine to Rockbox. It's now possible to pass formatting arguments to the lcd_puts_style function or to let it interpret some of the ANSI escape codes (the ones, that also control the style on your xterm, see http://en.wikipedia.org/wiki/ANSI_escape_code).
At least in the sim it works on all displays, even thar Player and the remotes. Some of them don't support all styles, of course. They just ignore styles and escape sequences, so that the API is the same on every device.
For example, the output, that finally comes out of the man command, should be direclty readable with the viewer function, as it's the same 'markup language'.
Patch Rockbox with dict_all.patch (it changes a lot, so I recommend to use a clean build) and have a look at the plugin test_styles (also for example code).
Then download the formatting_demo dictionary and search it for the article "Rockbox". It's a hand-formatted article, as I'm lacking a converter.
Please drop some feedback, because I still can only test with the sim.
make[2]: *** No rule to make target `rockboxlogo.160x128x16.bmp', needed by `/mnt/other/rockbox/build/h20/apps/bitmaps/native/rockboxlogo.160x128x16.c'. Stop.
make[1]: *** No rule to make target `/mnt/other/rockbox/build/h20/apps/dep-apps', needed by `/mnt/other/rockbox/build/h20/apps/rockbox.elf'. Stop.
make: *** [all] Error 2
Those come in that order. I then removed the patch and it built fine. then again patched it and it didn't work! sry for the news!
I forgot to take that out. This patch should be fine.
1 out of 2 hunks FAILED -- saving rejects to file apps/plugins/dict.c.rej
LOL. some1 must be trying to put your patch out! lol
The changes only made the patch program unhappy. Keeping your old dict.c would result in the same.
Synced.
Maybe you have other patches applied (e.g. a former dict patch) which conflict with it or your repository isn't up to date.
oh well I don't need a thesaurus or an encyclopedia for a while :(
Is it a bug, or is it just me :)
i use the mad cow build for my toshiba gigabeat. he included your dict patch. now i saw on installing that all headlines are with a shadow font and i can hardly read these headlines.
my question could you offer 2 version of your dict patch with your font style and with the old font style?
thank you
the pi
- id doesn't seem to be rebuilding the indexes in .rockbox anymore but instead says: "failed to open index" when you try and search
- the scroll bug seems to be back (where the status line is not taken into account when scrolling.)
Also would it be possible to split the different bits of functionality into different patches? The all in one currently causes some confusion: http://forums.rockbox.org/index.php?topic=9628.0
The headline style's purpose was actually just to demonstrate the style feature. I changed this to bold only, but you may disable it in apps/gui/list.c.
The plugin does not create the index files, just an offset table (.oft files in /.rockbox/). Not to find the index therefor sounds quite strange.
A tried to split up the patch into its funtionalities, but IMHO it's not worth the effort, because the dict plugin depends on most of the other patches.
styles.patch and viewer.patch are necessary for the dict plugin.
Additionaly dict_all.patch consists of all other patch, just as usual.
A problem I discovered is that the current english Wikipedia (4.9GiB) is too big to be used as a stardict dictionary at all. It uses uses uint32_t for the article offsets in the description file.
The old dumps are working because the converter seems to have missed some articles :-)
Some ideas to address articles over 4 GiB? Using a 64 bit type would break compatibility with stardict.
The converter also doesn't format the articles, it just copies the raw source. Have a look at mediawiki_format.c and feel free to enhance it!
Use `svn add` to add these:
apps/plugins/test_styles.c
apps/plugins/lib/viewer.c
apps/plugins/lib/viewer.h
firmware/export/ansi.h
firmware/ansi.c
Synced and added support for .lidx (index file with 64 bit offsets) to be able to use huge dictionaries (>4 GiB), though this is more like a temporary solution.
Have a look at the README in stardict-mediawiki.tar.bz2 to learn how to make your own Wikipedia dumps.
You might run svn-clean (http://svn.collab.net/repos/svn/trunk/contrib/client-side/svn-clean)before patching to get rid of _all_ files that don't belong to the repository.
It's now fully usable with the H1x0 remote (at least).
An other thing. With your new styles ther is an bug with the scrolling lines in the lists (Scrolling line one line above the line to scroll). Today I reseted my Database and after the clearing of the DB and befor I reinitialised it, it worked. So I don'T know is it because of the initialised database? mysic mysic, tomorow I try to reproduce.
I have no clue how the database could cause it, it's working fine on my player with the database initialized.
Btw, synced.
old_xmargin,old_ymargin
I would sync it if I knew how. :(
anyone know where I can learn?
The external bitmap icons are less friendly to patches than the old system. I think I'll include them inleave them out in the next patches, as it requires to replace around 10 bitmaps to make it work on all targets.
It doesn't include its own icons, but that's barely remarkable.
I'm not sure why. When I manually make the changes to dict.c (adding in #include "oldmenuapi.h" and removing several rb-->) everything compiles correctly. However, a patch, the patch doesn't seem to want to compile. Boo.
The first patch will do no changes to Rockbox itself, only the plugins.
what happened in under one day... They must really want your patch gone, which is crazy because I think it is wonderful
probably unsynced... or it may work fine but I have spent most of the day with the nary problem and I ended up just re-downloading the source to Rockbox and I do not want to have to do it again.
After some time I added some features to the viewer and rewrote some code.
Some new features:
- Search function to search within articles (case insensitive)
- Enhancement of the search algorithm (offset files will be recreated, but should be slightly faster)
- custom backlight setting
- REC button (iRiver, Sansa) as a configurable shortcut
Works with the current svn build.
@Enmar
If the patch doesn't work with dict.c, try to replace it with a previously patched dict.c.
Entering and searching for a word works fine on my iPod Nano, I get a list as a result.
But as soon as I try to browse to an article in this list, I get the following errors:
"illegal redirect (string)" followed immediately by a "Data abort at 01F218C (or another address)"
I'm using r13681M-070621 and I try to open wikipedia (both enwiki-20061130 and dewiki).
I have this problem for several months already and hoped that it would go away with a newer version of rockbox or dict_all, but it doesn't.
Am I doing something wrong or do others also have this problem?
The "Illegal redirect" should only occur when there's a redirect (#redirect [[somarticle]]) to a non-existing article.
If your a bit familiar with C, then please try to find out the line in the code, where it happens by putting some
rb->splash(HZ, "sometext");
into the plugin code.
@Matthew
As the scrolling margins patch does some changes to Rockbox, the plugin can only be compatible to either the original RB or RB with scrolling margins patch.
I don't use this patch, but this should be the only change to make it work with scrolling margins:
apps/plugins/lib/viewer.c
- rb->lcd_setmargins(VIEWER_SCROLLBAR_WIDTH, 0);
+ rb->lcd_setmargins(VIEWER_SCROLLBAR_WIDTH, 0, 0);
I get the same error as I am also using rockbox on an ipod nano
It is not just you...
Unfortunately I don't know C well so I'm not sure where to best insert the splash command, but I tried a few other things and noticed the folloing:
- I get the same "data abort" errors on an iPod Video G5 (60GB disk)
- When I compiled the patch for the Nano UISimulator, then the wikipedia files worked fine and I didn't get any "data abort" errors using the plugin.
- the problem does not seem to be with the .oft files as they look identical when built with the physical iPod or the UISimulator
debian:~/rockbox# patch --binary -p0 < Dict.patch
(Stripping trailing CRs from patch.)
patching file apps/lang/english.lang
Hunk #1 succeeded at 4939 (offset -6040 lines).
[b]Hunk #2 FAILED at 4980.[/b]
1 out of 2 hunks FAILED -- saving rejects to file apps/lang/english.lang.rej
(Stripping trailing CRs from patch.)
patching file apps/plugins/viewers.config
(Stripping trailing CRs from patch.)
patching file apps/plugins/dict.c
Hunk #1 succeeded at 7 with fuzz 2.
[b]Hunk #2 FAILED at 18.[/b]
[b]1 out of 2 hunks FAILED -- saving rejects to file apps/plugins/dict.c.rej[/b]
(Stripping trailing CRs from patch.)
patching file apps/plugins/lib/pluginlib_actions.h
Hunk #1 succeeded at 88 (offset 41 lines).
[b]Hunk #2 FAILED at 235.[/b]
[B]1 out of 2 hunks FAILED -- saving rejects to file apps/plugins/lib/pluginlib_actions.h.rej[/b]
(Stripping trailing CRs from patch.)
patching file apps/plugins/lib/viewer.c
(Stripping trailing CRs from patch.)
patching file apps/plugins/lib/SOURCES
Hunk #1 succeeded at 30 (offset 1 line).
(Stripping trailing CRs from patch.)
patching file apps/plugins/lib/viewer.h
(Stripping trailing CRs from patch.)
patching file apps/settings.h
(Stripping trailing CRs from patch.)
patching file apps/filetree.c
Hunk #1 succeeded at 265 (offset -1 lines).
Hunk #2 succeeded at 572 (offset -1 lines).
(Stripping trailing CRs from patch.)
patching file apps/root_menu.c
Hunk #1 succeeded at 179 with fuzz 1 (offset 7 lines).
Hunk #2 succeeded at 294 (offset 8 lines).
Hunk #3 succeeded at 308 (offset 8 lines).
Hunk #4 succeeded at 361 (offset 8 lines).
(Stripping trailing CRs from patch.)
patching file apps/root_menu.h
If you apply the ata_wakeup patch from
FS#7588you can comment out the rb->ata_wakeup() on line 853 in dict2.c. This will save some seconds when doing a new search while viewing an article (disk already spins up while you type the new search).But it compiles for me, did you patch the clean svn source?
Data abort
at 01XXXXXX
ipod nano simulator - works fine
ipod nano - data abort (I tried several versions over the last 3 months)
ipod video 5.5 30GB - data abort
sansa 280 - data abort
While the fuzzy search seems to works well on all these models, the data abort error appears when trying to open a topic.
60 gig 5th generation ipod
source : synced from svn today (10 minutes ago)
dict2_all.patch applied, compiles fine and :
"data abort" at (hex address)
is all I get. This looks really really nice, and I am gagging for it. I just wish the input methods contended with the Ipod Linux stuff, which looked pretty funky
The data abort (due to an unaligned memory access) is fixed. Thanks to parafin :)
Compilation is fine and fuzzy search too, the errors appear when trying to browse to an article.
It happens only for 1 out of 2 searches on my Nano, whereas it seems to happen on every search on my Sansa.
Also, scrolling is reversed on my ipod video - scrolling forward scrolls up instead of down.
Listing less than 80 articles causes no errors and this occurs on the ipod nano as far as I know.
If I change the topic limit to 15 and do it again -> No Data Abort
Also, I would like to have Undo | Redo function. For example: I read one article from my search, and want to go back to search results and read next article from the list. I simply hit one button and go there. :)
Is that what you are looking for? lol...
it brings up failed hunk while patching
Feel free to try it, and shout out if anything is not ok with it.
and thx to Kugel who had to have a lot of nerve with me^^
Feel free to try it, and shout out if anything is not ok with it.
and thx to Kugel who had to have a lot of nerve with me^^
The DEBUGF's mostly caused warnings, that's why I removed them.
I'm not sure how that could happen, but I seem to have missed some struct plugin_api -> const struct plugin_api conversions.
This should work, it compiles cleanly.
(look at 'white' and 'live' in the screendump)
Have a look at the last "stardict-mediawiki.tar.bz2" I uploaded, it allows you - once installed properly – to download convert wikipedia dumps with just one command, a URL to a pages-articles.xml.bz2 and enough disk space.
The archive has a README included that shows an example of how to use wikimedia.sh script (though the example URL is outdated and therefor doesn't work…)
http://rapidshare.com/files/118094472/dewikiREUP.7z.001
http://rapidshare.com/files/118112693/dewikiREUP.7z.002
http://rapidshare.com/files/118132773/dewikiREUP.7z.003
http://rapidshare.com/files/118297714/dewikiREUP.7z.004
http://rapidshare.com/files/118303247/dewikiREUP.7z.005
Have fun with it...
http://rapidshare.com/files/118094472/dewikiREUP.7z.001
http://rapidshare.com/files/118112693/dewikiREUP.7z.002
http://rapidshare.com/files/118132773/dewikiREUP.7z.003
http://rapidshare.com/files/118297714/dewikiREUP.7z.004
http://rapidshare.com/files/118303247/dewikiREUP.7z.005
Have fun with it...
I've removed the undefined function from the viewer.c and introduced a left_margin variable instead to use with putsxy().
Now it works fine on my recorder.
Somebody please check if my soulution is correct because this is my first patch ever. I hope it's ok.
This plugin is very useful! Thanks for it!
Hg
In fact, any change will work just fine. I couldn't get the function in lib/viewer.c to be actually used! (Note that there's also a viewer.c in plugins/, a rename would probably nice).
On the other hand, I'm not too into this patch.
I agree with that the renaming of the viewer.c would be nice.
Give it a try!
LD dict.rock
make: *** No rule to make target `pluginlib_actions.h', needed by `/home/ehyer/rockbox_ipod55/rockbox_source/build_20081210/apps/plugins/dict2.o'. Stop.
Also, <NOOB>, unclear to me how to revert to daily build after I screw things up patching. Does 'svn co' give me a fresh source tree?</NOOB>
Thank you for this patch!
Removed memcpy from dict2.c (use memcpy in apps/plugins/lib/gcc-support.c now).
because what i really want is a almost full wikipedia on my iPod
fixed warning in apps/lang/english.lang
See my forumthread to get your copy of a precompiled 3.3 build containing lots of GPL dicts from satrdict.org
Please let me know which player you have, so i can make a version for you and don't have to compile for every possible target.
http://forums.rockbox.org/index.php?topic=20584.0
Second: i'll sync to latest revision soon.
i forgot to update the revision numbers in the last patch, it doesn't change the fact that it's working, though.
Fixed missing last character at end of line.
still some bugs with moving between screens.
GRaTT
But I want to know that the plugin is kind of obsolete at least to me, as there's the wikipedia plugin (FS#4755) which has quite the same purpose and works quite well, or better.
Though it is primarily written for viewing wikipedia, you can find a converter for stardict dictionaries I posted on its flyspray task (http://www.rockbox.org/tracker/task/4755#comment27464).
in the error message the last character of the index file before the extension is missing
and there's a double slash before the filename
e.g. /dicts//Oxfor.idx instead of /dicts/Oxford.idx
unfortunately I can't code... It might be an easy fix (hopefully)
I think the "Dictionaries" root menu shouldn't be part of this patch anyway, because
(1) it works for the "new dict plugin" and the "wikipedia plugin".
(2) I think, that a patch should be split up into its funtionalities.
perhaps the root menu item should even have it's own task
It is beyond my ability to resolve this issue.
If someone would be kind enough to sync this to current SVN I would be very appreciative.
I have been trying to push this for a commit on IRC with recent discussion, it is vastly superior to "dict.rock" which in my opinion shouldn't even be in SVN with the limitation is has currently (no search, no scroll).
Again, if this could be synced with current SVN I would be very appreciative.
[St.]
make sure following files are removed and apply the patch again.
- apps/plugins/dict2.c
- apps/plugins/lib/viewer.c
- apps/plugins/lib/viewer.h
by the way, current dict plugin should scroll the text now.
[St.]