Hello,

I am working on a parser for a small language and was playing around with the fslex tool. I am trying to create a list of tokens by only using LEX not YACC.

e.g. the lex file looks like this, where the only thing that is happening is that I create tokens for NEWLINE and for ID's

1
2
3
4
5
6
7
8
9
10
11
12
13

{
type test = NEWLINE | ID of string
}

let newline = ('\n' | '\r' '\n')
let id = ['a'-'z' 'A'-'Z' '_']

rule testing = parse
| newline        { NEWLINE }
| ' '            { testing lexbuf } 
| id+            { ID(lexeme lexbuf) }
| _                { testing lexbuf }
| eof    { () }

My main program loops through a string and produces the tokens and i want to put them into a list - but I cannot seem to find where to get the "value" that the lexer generates when it creates a token...

I would like this program to put all the NEWLINE and ID("...") int the list_of_tokens list.

1
2
3
4
5
6
7
8
9

let lexbuf = Lexing.from_string "\n\n myID" in
let list_of_tokens = ref [] in
    try 
        while true do
            testing lexbuf;
            printf "\n%s\n" (Lexing.lexeme lexbuf);
        done
    with e -> 
        printf "done"

Where can I reference the NEWLINE and ID("...") objects ???