Compiler/lexical analyzer: Difference between revisions

For example, the following two program fragments are equivalent, and should produce the same token stream except for the line and column positions:

 

print ( n , " " ) ;

count = count + 1 ; /* number of primes found so far */

 

;Complete list of token names

| style="vertical-align:top" |

Test Case 1:

Hello world

*/

 

| style="vertical-align:top" |

| style="vertical-align:top" |

Test Case 2:

Show Ident and Integers

*/

phoenix_number = 142857;

 

| style="vertical-align:top" |

| style="vertical-align:top" |

Test Case 3:

All lexical tokens - not syntactically correct, but that will

have to wait until syntax analysis

/* character literal */ '\n'

/* character literal */ '\\'

 

| style="vertical-align:top" |

| style="vertical-align:top" |

Test Case 4:

print(42);

print("\nHello World\nGood Bye\nok\n");

 

| style="vertical-align:top" |

 

=={{header|Ada}}==

Ada.Exceptions;

use Ada.Strings, Ada.Strings.Unbounded, Ada.Streams, Ada.Exceptions;

when error : others => IO.Put_Line("Error: " & Exception_Message(error));

end Main;

{{out}} Test case 3:

<pre>

23 1 END_OF_INPUT

</pre>

 

=={{header|ALGOL W}}==

%lexical analyser %

% Algol W strings are limited to 256 characters in length so we limit source lines %

while nextToken not = tEnd_of_input do writeToken;

writeToken

{{out}} Test case 3:

<pre>

(One point of note: the C "EOF" pseudo-character is detected in the following code by looking for a negative number. That EOF has to be negative and the other characters non-negative is implied by the ISO C standard.)

 

(* Usage: lex [INPUTFILE [OUTPUTFILE]]

If INPUTFILE or OUTPUTFILE is "-" or missing, then standard input

end

 

 

{{out}}

=={{header|AWK}}==

Tested with gawk 4.1.1 and mawk 1.3.4.

BEGIN {

all_syms["tk_EOI" ] = "End_of_input"

}

}

{{out|case=count}}

<b>

=={{header|C}}==

Tested with gcc 4.81 and later, compiles warning free with -Wpedantic -pedantic -Wall -Wextra

#include <stdio.h>

#include <stdarg.h>

run();

return 0;

 

{{out|case=test case 3}}

=={{header|C sharp|C#}}==

Requires C#6.0 because of the use of null coalescing operators.

using System;

using System.IO;

}

}

 

{{out|case=test case 3}}

=={{header|C++}}==

Tested with GCC 9.3.0 (g++ -std=c++17)

#include <fstream> // file_to_string, string_to_file

#include <functional> // std::invoke

});

}

 

{{out|case=test case 3}}

Using GnuCOBOL 2. By Steve Williams (with one change to get around a Rosetta Code code highlighter problem).

 

*> this code is dedicated to the public domain

*> (GnuCOBOL) 2.3-dev.0

end-if

.

 

{{out|case=test case 3}}

Lisp has a built-in reader and you can customize the reader by modifying its readtable. I'm also using the Gray stream, which is an almost standard feature of Common Lisp, for counting lines and columns.

 

(:use #:cl #:sb-gray)

(:export #:main))

 

(defun main ()

{{out|case=test case 3}}

<pre> 5 16 KEYWORD-PRINT

{{trans|ATS}}

 

# -*- elixir -*-

 

end ## module Lex

 

 

{{out}}

=={{header|Emacs Lisp}}==

{{works with|Emacs|GNU 27.2}}

 

 

;;

;; The Rosetta Code lexical analyzer in GNU Emacs Lisp.

(scan-text t))

 

 

 

 

 

%%%-------------------------------------------------------------------

 

%%% erlang-indent-level: 3

%%% end:

 

 

=={{header|Euphoria}}==

Tested with Euphoria 4.05.

include std/map.e

include std/types.e

end procedure

 

 

{{out|case=test case 3}}

=={{header|Flex}}==

Tested with Flex 2.5.4.

#include <stdio.h>

#include <stdlib.h>

} while (tok != tk_EOI);

return 0;

 

{{out|case=test case 3}}

=={{header|Forth}}==

Tested with Gforth 0.7.3.

CREATE COLUMN# 0 ,

CREATE LINE# 1 ,

THEN THEN ;

: TOKENIZE BEGIN CONSUME AGAIN ;

 

{{out}}

 

The author has placed this Fortran code in the public domain.

!!! An implementation of the Rosetta Code lexical analyzer task:

!!! https://rosettacode.org/wiki/Compiler/lexical_analyzer

end subroutine print_usage

 

{{out}}

=={{header|FreeBASIC}}==

Tested with FreeBASIC 1.05

tk_EOI

tk_Mul

print : print "Hit any to end program"

sleep

{{out|case=test case 3}}

<b>

=={{header|Go}}==

{{trans|FreeBASIC}}

 

import (

initLex()

process()

 

{{out}}

=={{header|Haskell}}==

Tested with GHC 8.0.2

import Control.Monad.State.Lazy

import Control.Monad.Trans.Maybe (MaybeT, runMaybeT)

where (Just t, s') = runState (runMaybeT lexer) s

(txt, _, _) = s'

 

{{out|case=test case 3}}

Global variables are avoided except for some constants that require initialization.

 

# The Rosetta Code lexical analyzer in Icon with co-expressions. Based

# upon the ATS implementation.

toktup := scan_character_literal_without_checking_end(inp)

ch1 := @inp

}

}

if ch[1] == "*" then {

ch1 := @inp

}

}

procedure max(x, y)

return (if x < y then y else x)

 

 

Implementation:

 

ch=: {{1 0+x[symbols=: x (a.i.y)} symbols}}

'T0 token' =: 0 ch '%+-!(){};,<>=!|&'

keep=. (tokens~:<,'''')*-.comments+.whitespace+.unknown*a:=values

keep&#each ((1+lines),.columns);<names,.values

 

Test case 3:

 

flex=: {{

'A B'=.y

21 28 Integer 92

22 27 Integer 32

 

Here, it seems expedient to retain a structured representation of the lexical result. As shown, it's straightforward to produce a "pure" textual result for a hypothetical alternative implementation of the syntax analyzer, but the structured representation will be easier to deal with.

 

=={{header|Java}}==

// Translated from python source

 

if (text.equals("")) {

}

if (Character.isDigit(text.charAt(0))) {

if (!is_number) {

}

return new Token(TokenType.Integer, text, line, pos);

}

}

 

=={{header|JavaScript}}==

{{incorrect|Javascript|Please show output. Code is identical to [[Compiler/syntax_analyzer]] task}}

/*

Token: type, value, line, pos

l.printTokens()

})

 

=={{header|Julia}}==

startline::Int

startcol::Int

println(lpad(tok.startline, 3), lpad(tok.startcol, 5), lpad(tok.name, 18), " ", tok.value != nothing ? tok.value : "")

end

Line Col Name Value

5 16 Keyword_print

23 1 End_of_input

</pre>

 

=={{header|Lua}}==

 

The first module is simply a table defining the names of tokens which don't have an associated value.

local token_name = {

['*'] = 'Op_multiply',

['putc'] = 'Keyword_putc',

}

 

This module exports a function <i>find_token</i>, which attempts to find the next valid token from a specified position in a source line.

local M = {} -- only items added to M will be public (via 'return M' at end)

local table, concat = table, table.concat

end

 

The <i>lexer</i> module uses <i>finder.find_token</i> to produce an iterator over the tokens in a source.

local M = {} -- only items added to M will publicly available (via 'return M' at end)

local string, io, coroutine, yield = string, io, coroutine, coroutine.yield

-- M._INTERNALS = _ENV

return M

 

This script uses <i>lexer.tokenize_text</i> to show the token sequence produced from a source text.

 

format, gsub = string.format, string.gsub

 

-- etc.

end

 

===Using only standard libraries===

This version replaces the <i>lpeg_token_finder</i> module of the LPeg version with this <i>basic_token_finder</i> module, altering the <i>require</i> expression near the top of the <i>lexer</i> module accordingly. Tested with Lua 5.3.5. (Note that <i>select</i> is a standard function as of Lua 5.2.)

 

local M = {} -- only items added to M will be public (via 'return M' at end)

local table, string = table, string

 

-- M._ENV = _ENV

 

=={{header|M2000 Interpreter}}==

Module lexical_analyzer {

a$={/*

}

lexical_analyzer

 

{{out}}

 

 

%

% Compile with maybe something like:

 

:- func eof = int is det.

 

{{out}}

Tested with Nim v0.19.4. Both examples are tested against all programs in [[Compiler/Sample programs]].

===Using string with regular expressions===

import re, strformat, strutils

 

 

echo input.tokenize.output

===Using stream with lexer library===

import lexbase, streams

from strutils import Whitespace

echo &"({l.lineNumber},{l.getColNumber l.bufpos + 1}) {l.error}"

main()

 

===Using nothing but system and strutils===

 

type

stdout.write('\n')

if token.kind == tokEnd:

 

=={{header|ObjectIcon}}==

 

 

#

# The Rosetta Code lexical analyzer in Object Icon. Based upon the ATS

toktup := scan_character_literal_without_checking_end(inp)

ch1 := @inp

}

}

write!([FileStream.stderr] ||| args)

exit(1)

 

 

(Much of the extra complication in the ATS comes from arrays being a linear type (whose "views" need tending), and from values of linear type having to be local to any function using them. This limitation could have been worked around, and arrays more similar to OCaml arrays could have been used, but at a cost in safety and efficiency.)

 

(* The Rosetta Code lexical analyzer, in OCaml. Based on the ATS. *)

 

main ()

 

 

{{out}}

Note: we do not print the line and token source code position for the simplicity.

 

(import (owl parse))

 

(if (null? (cdr stream))

(print 'End_of_input))))

 

==== Testing ====

 

Testing function:

(define (translate source)

(let ((stream (try-parse token-parser (str-iter source) #t)))

(if (null? (force (cdr stream)))

(print 'End_of_input))))

 

====== Testcase 1 ======

 

(translate "

/*

*/

print(\"Hello, World!\\\\n\");

{{Out}}

<pre>

====== Testcase 2 ======

 

(translate "

/*

phoenix_number = 142857;

print(phoenix_number, \"\\\\n\");

{{Out}}

<pre>

====== Testcase 3 ======

 

(translate "

/*

/* character literal */ '\\\\'

/* character literal */ ' '

{{Out}}

<pre>

====== Testcase 4 ======

 

(translate "

/*** test printing, embedded \\\\n and comments with lots of '*' ***/

print(\"Print a slash n - \\\\\\\\n.\\\\n\");

")

{{Out}}

<pre>

=={{header|Perl}}==

 

 

use strict;

($line, $col)

}

 

{{out|case=test case 3}}

===Alternate Perl Solution===

Tested on perl v5.26.1

 

use strict; # lex.pl - source to tokens

1 + $` =~ tr/\n//, 1 + length $` =~ s/.*\n//sr, $^R;

}

 

=={{header|Phix}}==

form. If required, demo\rosetta\Compiler\extra.e (below) contains some code that achieves the latter.

Code to print the human readable forms is likewise kept separate from any re-usable parts.

<span style="color: #000080;font-style:italic;">--

-- demo\rosetta\Compiler\core.e

<span style="color: #008080;">return</span> <span style="color: #000000;">s</span>

<span style="color: #008080;">end</span> <span style="color: #008080;">function</span>

For running under pwa/p2js, we also have a "fake file/io" component:

<span style="color: #000080;font-style:italic;">--

-- demo\rosetta\Compiler\js_io.e

--</span>

<span style="color: #008080;">with</span> <span style="color: #008080;">javascript_semantics</span>

<span style="color: #0000FF;">{</span><span style="color: #008000;">"test3.c"</span><span style="color: #0000FF;">,</span><span style="color: #7060A8;">split</span><span style="color: #0000FF;">(</span><span style="color: #008000;">"""

/*

}

print(a);

"""</span><span style="color: #0000FF;">,</span><span style="color: #008000;">"\n"</span><span style="color: #0000FF;">)}})</span>

<span style="color: #008080;">global</span> <span style="color: #008080;">function</span> <span style="color: #000000;">js_open</span><span style="color: #0000FF;">(</span><span style="color: #004080;">string</span> <span style="color: #000000;">filename</span><span style="color: #0000FF;">)</span>

<span style="color: #7060A8;">assert</span><span style="color: #0000FF;">(</span><span style="color: #000000;">fn</span><span style="color: #0000FF;">!=</span><span style="color: #000000;">0</span><span style="color: #0000FF;">)</span>

<span style="color: #008080;">return</span> <span style="color: #000000;">fn</span>

<span style="color: #008080;">end</span> <span style="color: #008080;">function</span>

<span style="color: #008080;">end</span> <span style="color: #008080;">function</span>

The main lexer is also written to be reusable by later stages.

<span style="color: #000080;font-style:italic;">--

-- demo\\rosetta\\Compiler\\lex.e

<span style="color: #000000;">line</span> <span style="color: #0000FF;">+=</span> <span style="color: #000000;">1</span>

<span style="color: #000000;">col</span> <span style="color: #0000FF;">=</span> <span style="color: #000000;">0</span>

<span style="color: #0000FF;">:</span><span style="color: #7060A8;">gets</span><span style="color: #0000FF;">(</span><span style="color: #000000;">input_file</span><span style="color: #0000FF;">))</span>

<span style="color: #008080;">else</span>

<span style="color: #008080;">end</span> <span style="color: #008080;">function</span>

<span style="color: #008080;">constant</span> <span style="color: #000000;">whitespace</span> <span style="color: #0000FF;">=</span> <span style="color: #0000FF;">{</span><span style="color: #008000;">' '</span><span style="color: #0000FF;">,</span><span style="color: #008000;">'\t'</span><span style="color: #0000FF;">,</span><span style="color: #008000;">'\r'</span><span style="color: #0000FF;">,</span><span style="color: #008000;">'\n'</span><span style="color: #0000FF;">,</span><span style="color: #000000;">#0B</span><span style="color: #0000FF;">,</span><span style="color: #000000;">#A0</span><span style="color: #0000FF;">}</span>

<span style="color: #000080;font-style:italic;">-- (0x0B is Vertical Tab, 0xA0 is Non-breaking space)</span>

<span style="color: #008080;">function</span> <span style="color: #000000;">get_op</span><span style="color: #0000FF;">()</span>

<span style="color: #004080;">string</span> <span style="color: #000000;">operator</span> <span style="color: #0000FF;">=</span> <span style="color: #008000;">""</span><span style="color: #0000FF;">&</span><span style="color: #000000;">ch</span>

<span style="color: #000000;">ch</span> <span style="color: #0000FF;">=</span> <span style="color: #000000;">next_ch</span><span style="color: #0000FF;">()</span>

<span style="color: #008080;">return</span> <span style="color: #000000;">toks</span>

<span style="color: #008080;">end</span> <span style="color: #008080;">function</span>

Optional: if you need human-readable output/input at each (later) stage, so you can use pipes

<span style="color: #000080;font-style:italic;">--

-- demo\rosetta\Compiler\extra.e

<span style="color: #008080;">return</span> <span style="color: #0000FF;">{</span><span style="color: #000000;">n_type</span><span style="color: #0000FF;">,</span> <span style="color: #000000;">left</span><span style="color: #0000FF;">,</span> <span style="color: #000000;">right</span><span style="color: #0000FF;">}</span>

<span style="color: #008080;">end</span> <span style="color: #008080;">function</span>

Finally, a simple test driver for the specific task:

<span style="color: #000080;font-style:italic;">--

-- demo\rosetta\Compiler\lex.exw

<span style="color: #000080;font-style:italic;">--main(command_line())</span>

<span style="color: #000000;">main</span><span style="color: #0000FF;">({</span><span style="color: #000000;">0</span><span style="color: #0000FF;">,</span><span style="color: #000000;">0</span><span style="color: #0000FF;">,</span><span style="color: #008000;">"test4.c"</span><span style="color: #0000FF;">})</span>

{{out}}

<pre>

=={{header|Prolog}}==

 

Test harness for the analyzer, not needed if we are actually using the output.

*/

 

% anything else is an error

{{out}}

<pre>

=={{header|Python}}==

Tested with Python 2.7 and 3.x

import sys

 

#*** "string"

def string_lit(start, err_line, err_col):

text = ""

 

if the_ch == '\n':

error(err_line, err_col, "EOL while scanning string literal")

text += the_ch

 

 

if tok == tk_EOI:

 

{{out|case=test case 3}}

=={{header|QB64}}==

Tested with QB64 1.5

dim shared line_n as integer, col_n as integer, text_p as integer, err_line as integer, err_col as integer, errors as integer

 

end

end sub

{{out|case=test case 3}}

<b>

 

=={{header|Racket}}==

#lang racket

(require parser-tools/lex)

"TEST 5"

(display-tokens (string->tokens test5))

 

=={{header|Raku}}==

{{works with|Rakudo|2016.08}}

 

rule TOP { ^ <.whitespace>? <tokens> + % <.whitespace> <.whitespace> <eoi> }

 

 

my $tokenizer = tiny_C.parse(@*ARGS[0].IO.slurp);

 

{{out|case=test case 3}}

23 1 End_of_input

</pre>

 

=={{header|Scala}}==

The following code implements a configurable (from a symbol map and keyword map provided as parameters) lexical analyzer.

 

package xyz.hyperreal.rosettacodeCompiler

 

 

}

 

=={{header|Scheme}}==

 

(import (scheme base)

(scheme char)

(display-tokens (lexer (cadr (command-line))))

(display "Error: provide program filename\n"))

 

{{out}}

 

 

(* The Rosetta Code lexical analyzer, in Standard ML. Based on the ATS

and the OCaml. The intended compiler is Mlton or Poly/ML; there is

(* sml-indent-args: 2 *)

(* end: *)

 

 

{{libheader|Wren-fmt}}

{{libheader|Wren-ioutil}}

import "os" for Process

 

lineCount = lines.count

initLex.call()

 

{{out}}

 

=={{header|Zig}}==

const std = @import("std");

 

return result.items;

}