FS#9546 - UTF-8 BOM "visible" in viewer

Attached to Project: Rockbox
Opened by Robert (inrobert) - Thursday, 13 November 2008, 18:47 GMT
Last edited by Yoshihisa Uchida (Uchida) - Wednesday, 17 March 2010, 12:20 GMT
Task Type Bugs
Category Plugins
Status Closed
Assigned To No-one
Operating System All players
Severity Low
Priority Normal
Reported Version Daily build (which?)
Due in Version Version 3.1
Due Date Undecided
Percent Complete 100%
Votes 0
Private No


I have created some .txt files in UTF-8 format with Byte Order Marks (BOMs), using Windows XP SP3 Notepad and OpenOffice 3.0 Writer. In the viewer options I have set UTF-8 encoding. All the characters are correctly displayed, but at the beginning of each file there is strange rectangle that represents BOM. I have done a research and found out that BOM handling in Rockbox core was corrected by this patch –, but described issue is present in viewer plugin.

I know one can save without BOM, using Notepad2, but BOM allows Windows to recognize the UTF-8 file automatically, which is useful - for example it allows Wordpad and Lister to correctly open UTF-8 .txt files. So, as somebody is using .txt files both with Rockbox and Windows, the proper BOM handling by Rockbox is useful.
This task depends upon

Closed by  Yoshihisa Uchida (Uchida)
Wednesday, 17 March 2010, 12:20 GMT
Reason for closing:  Accepted
Additional comments about closing:  patch commits.
Comment by Yoshihisa Uchida (Uchida) - Wednesday, 21 January 2009, 09:20 GMT
I create the patch file.

Please confirm it.
Comment by Jonathan Gordon (jdgordon) - Wednesday, 21 January 2009, 09:39 GMT
I'm not sure that is the best way to go about fixing this....
I cant think of many places in the core where the text file position is as important for seeking as it is in viewer so I tihnk its better to fix viewer to be smarter than to waste code in the core for something which will only be used in a plugin
Comment by Dominik Riebeling (bluebrother) - Wednesday, 21 January 2009, 16:51 GMT
I put together a patch for this as well a bit ago (but forgot to post it here). The main problem I have with this is the fact that the viewer tries to align reads with the disk sectors. Both the above patch and my version (which is slightly more compact as it doesn't introduce a specific filesize function -- you can easily do this by using open_utf8() and then substracting the current byte position) break this functionality.
Comment by Yoshihisa Uchida (Uchida) - Thursday, 22 January 2009, 07:35 GMT
Gordon, Riebeling comment thanks.

My patch file found some problems.
When the problem is solved, the patch file will upload.

When I correct the patch file, *_utf8() functions will not use. (Then, viewer.c changes only)

Riebeling, because it might take time to correct the patch file, I think that you may upload your patch file where you are made without waiting for my correction.

Comment by Yoshihisa Uchida (Uchida) - Thursday, 22 January 2009, 10:25 GMT
Because the correction of problems was completed, a new patch file upload.
Comment by Yoshihisa Uchida (Uchida) - Friday, 13 February 2009, 03:32 GMT
When the patch of  FS#9855 ,  FS#9892 ,  FS#9893 , or  FS#9898  is applied, the patch of this task cannot be applied.

Even if you forcibly apply the patch file, it doesn't operate correctly.
Oppositely, even if you apply these patches after you apply the patch of this task, it doesn't operate correctly.

If you want to apply the patch of  FS#9855  (or  FS#9892 ,  FS#9893 ,  FS#9898 ), you must use the patch file of  FS#9899  (this task is closed).
Comment by Yoshihisa Uchida (Uchida) - Saturday, 21 February 2009, 03:50 GMT
Because the patch file of  FS#9899  is old, I send a new patch.
Please apply the patch in order of  FS#9855 ,  FS#9892 ,  FS#9893 ,  FS#9898 ,  FS#9902 ,  FS#9853  and this patch.

If you do not apply these patch files, this patch need not be applied.
Comment by Yoshihisa Uchida (Uchida) - Thursday, 05 March 2009, 13:36 GMT
sync the patch file.
Comment by Yoshihisa Uchida (Uchida) - Wednesday, 17 June 2009, 09:18 GMT
sync r21316
Comment by Yoshihisa Uchida (Uchida) - Saturday, 20 February 2010, 05:57 GMT
sync 24782