[:D]
I've revised the code in-line with how fslex works. Fine.
[:$]However the program only works if we take input from a file specifed on the program command-line. If we use stdin,we get an infinite loop as the code calls through to the default case for unrecognised input.
Do we have some form of bytes/text confusion going on in the input stream [8-)]?

New code :-

{
open Printf
let create_hashtable size init =
let tbl = Hashtbl.create size in
List.iter (fun (key, data) -> Hashtbl.add tbl key data) init;
tbl

type token =
| IF
| THEN
| ELSE
| BEGIN
| END
| FUNCTION
| ID of string
| OP of char
| INT of int
| FLOAT of float
| CHAR of char

let keyword_table =
create_hashtable 8 [
("if", IF);
("then", THEN);
("else", ELSE);
("begin", BEGIN);
("end", END);
("function", FUNCTION)
]
}

let digit = ['0'-'9']
let inum = digit+
let fnum = digit+ '.' digit*
let id = ['a'-'z' 'A'-'Z']['a'-'z' '0'-'9']*
let word = id
let op = '+' | '-' | '*' | '/'

rule toy_lang = parse
| inum
{ let num = int_of_string ( Lexing.lexeme lexbuf ) in
printf "integer: %s (%d)\n" ( Lexing.lexeme lexbuf ) num;
INT num
}
| fnum
{ let num = float_of_string ( Lexing.lexeme lexbuf ) in
printf "float: %s (%f)\n" ( Lexing.lexeme lexbuf ) num;
FLOAT num
}
| id
{ try
let token = Hashtbl.find keyword_table ( Lexing.lexeme lexbuf ) in
printf "keyword: %s\n" ( Lexing.lexeme lexbuf );
token
with Not_found ->
printf "identifier: %s\n" ( Lexing.lexeme lexbuf );
ID ( Lexing.lexeme lexbuf )
}
| op
{ printf "operator: %c\n" ( Lexing.lexeme_char lexbuf 1 );
OP ( Lexing.lexeme_char lexbuf 1 )
}
| '{' [^ '\n']* '}' (* eat up one-line comments *)
| [' ' '\t' '\n'] (* eat up whitespace *)
{ toy_lang lexbuf }
| _
{ printf "Unrecognized character: %c\n" ( Lexing.lexeme_char lexbuf 0 ) ;
CHAR ( Lexing.lexeme_char lexbuf 0 )
}
| eof
{ raise End_of_file }
{
let rec parse lexbuf =
let token = toy_lang lexbuf in
(* do nothing in this example *)

parse lexbuf
let main () =
let cin =
if Array.length Sys.argv > 1
then open_in Sys.argv.(1)
else stdin
in
let lexbuf = Lexing.from_channel cin in
try parse lexbuf
with End_of_file -> ()
let _ = Printexc.print main ()
}

By on 8/29/2007 2:25 PM ()

Hi Mark,

You may well want Lexing.from_text_reader for stdin, though it's hard to be sure - it depends what encoding the text is coming in as and what encoding you want to use to present the bytes to the lexer. If it's straight-up ASCII input it shouldn't make much difference.

What characters are being printed?

don

By on 9/4/2007 6:21 PM ()

Don,

The lexer works fine until it runs out of input. It then just seems to get stuck in a loop, and spits out the null character for ever. I know that this is a trivial problem, given the right level of knowledge, but right now, my knowledge of .net and f# is around zero percent.

Regards, Mark

eg :-

12 + 12
12
12
3/2/2/2/2/1

yields :-

integer: 12 (12)
operator: +
integer: 12 (12)
integer: 12 (12)
integer: 12 (12)
integer: 3 (3)
operator: /
integer: 2 (2)
operator: /
integer: 2 (2)
operator: /
integer: 2 (2)
operator: /
integer: 2 (2)
operator: /
integer: 1 (1)
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:
Unrecognized character:

By on 9/5/2007 8:05 AM ()

On a quick look, in the F# lexer we use

| _ { unexpected_char lexbuf }

| eof { EOF }

i.e. return an explicit token in the eof condition. That may be worth trying here rather than raising an exception.

Kind regards

don

By on 9/7/2007 9:56 AM ()

OK, I've sorted this one, just needed to replace the aliases with separate let definitions.

A bit of a bind though..........

By on 8/29/2007 7:18 AM ()
IntelliFactory Offices Copyright (c) 2011-2012 IntelliFactory. All rights reserved.
Home | Products | Consulting | Trainings | Blogs | Jobs | Contact Us | Terms of Use | Privacy Policy | Cookie Policy
Built with WebSharper