This page is outdated. See WpsTokenizerDEPRECATED instead.

(Outdated) Idea for a token based WPS handling

Info

This is a design paper of an idea to make the wps-handling token-based.
Not all wps-tags are handled in this paper.
This is only an idea, maybe this is the wrong way to do it if the wps-code should be rewritten.

Paper

this file describes a possible new methode for the wps-handling (token-based)
currently this only describes the used data-struct and how the wps gets updated
the biggest part will be to generate the tokens from the wps-format-file.

All non-"static"-tags and text (static-tags = which needs only be parsed on wps-file-loading)
will be converted to tokens.
the text-token holds all non-tag text-string until the line ends or a tag was found

each found tag (also "normal" text) in the wps-format will be represented by an token
each token points to the next token. When an tag is used in an conditional then the token 
(1)points to the next token after the conditional via "next",
(2) points to the first token in the conditional itself via "next_conditional"

known problems in this design:
- how to handle the creation of the tokens and how to manage then needed memory-space to hold all tokens
  (because we can't know how many tokens will be needed upon loading)
- how to handle when between two tags are only a few spaces(<5) to avoid to use an extra token(if possible).

pseudo-code:    

var strings                 // buffer which holds all "normal" text-string each string is null-terminated

define wps_token:
    var type                // which tag it is e.g. image-loading-tag either the tag-string(without the %/%?) or a number representation
    union
        pvar value          // the value(as an pointer to the string) of the tag if type = text otherwise null
        var id              // image id if type = image otherwise -1
    var line                // line-number
    var conditional         // if found as an conditional   
    pvar next               // points to the next token if conditional = 1 it points 
                            // to the token after the conditional
    pvar next_conditional   // points to first token in the conditional if conditional = 1

struct-size:
        4 Bytes (type)
      + 4 Bytes (value/id)
      + 4 Bytes (line)
      + 1 Byte  (conditional)
      + 4 Bytes (next)
      + 4 Bytes (next_conditional)
    ------------------------------
        19 Bytes

function update_wps:
    var token = start_token
    var oldcharpos = 0
    var after_con = null
    var oldline = 0    
    do
        if token.conditional     // is token an conditional
            value = get_value(token)    // get conditional value from the tag represented by the token
            if value != -1              // check if the tag was valid to be used in an conditional
                if(!after_con)          // is this token the first conditional (when an conditional is in an conditional)
                    after_con = token.next  // save the token after the conditional
                // go to the result-token of the conditional
                i = 0
                while i < value
                    token = token.next_conditional
                    i++
                // check if the token is an empty one (the result isn't used)
                // when empty then jump to the token after the (first) conditional
                if(token.type == empty)
                    token = after_con
                // check if the new token is not conditional-one
                else if(!token.conditional)
                    handle_token(token) // handle the token
                    token = after_con   // jump after the (first) conditional
                    after_con = null    
            else
                token = token.next      // go to the next token
        else
            handle_token(token)         // handle the token
            token = token.next          // go to the next token
                    
        if token.line != oldline
            oldcharpos = 0
    while token.next != null            // loop until all tokens are handled
    
    update_lcd()                        // update the lcd with the new frame-buffer

function handle_token(token):           // handles non conditional tokens
    if token.type != text and token.type != image   // if the token is not a simple-text-string or an image
        value = get_value(token)                    // get the value of the token as an string
        print(oldcharpos,token.line,value)          // draw the value-string on the given position
        oldcharpos = string_length(value)           // save the new string position for new to display values/text
    else if token.type == text                      // if token is an simple-text-string
        print(oldcharpos,token.line,token.value)    // draw the text
        oldcharpos = string_length(token.value)     // save the new string position for new to display values
    else if token.type == image                     // if token is an image
        draw_image(token.id)                        // draw the image

function get_value(token) //returns the value for the wps-tag
                          // if token.conditional is set then the "normal" value is mapped 
                          // to a conditional-value if the tag can be used in an conditional
                          // otherwise -1 (when token.conditional = 1)

Comments

-- MatthiasM - 25 Dec 2005: nice concept. But why don't make everything a token, why only dynamic elements? If you also make "normal" text parts a token it would be much easier to add "attributes" to screen elements. e.g. to give elements (even text parts) foreground and background color, to set fonts, to add icons, absolute or relative screen positions, ... The hardest and biggest part would still be parsing the WPS file and generate appropriate tokens

-- NicolasPennequin - 15 Jan 2007
There are two flyspray entries about this : FS#2898 and FS#4826.
I've started work on trying to implement this, though not exactly in the way described above (I posted some preliminary code on FS#4826). IMHO it can be much simpler than what's suggested, i.e. no need to keep track of the next conditional and the next token.

-- NicolasPennequin - 20 Mar 2007
I have made good progress on this. I have created a new tracker entry for my work : FS#6862. I will try to find time to edit this page to explain the concepts of the patch, and maybe add information and ideas about a possible parser plugin and bytecode WPS format.