NAME¶

re2c - convert regular expressions to C/C++

SYNOPSIS¶

re2c [OPTIONS] FILE

DESCRIPTION¶

re2c is a lexer generator for C/C++. It finds regular expression specifications inside of C/C++ comments and replaces them with a hard-coded DFA. The user must supply some interface code in order to control and customize the generated DFA.

EXAMPLE¶

Given the following code:

unsigned int stou (const char * s)
{
#   define YYCTYPE char


    const YYCTYPE * YYCURSOR = s;


    unsigned int result = 0;


    for (;;)


    {


        /*!re2c


            re2c:yyfill:enable = 0;


            "\x00" { return result; }


            [0-9]  { result = result * 10 + c; continue; }


        */


    }
}

re2c -is will generate:

/* Generated by re2c 0.13.7.dev on Mon Jul 14 13:37:46 2014 */
unsigned int stou (const char * s)
{
#   define YYCTYPE char


    const YYCTYPE * YYCURSOR = s;


    unsigned int result = 0;


    for (;;)


    {
{


        YYCTYPE yych;


        yych = *YYCURSOR;


        if (yych <= 0x00) goto yy3;


        if (yych <= '/') goto yy2;


        if (yych <= '9') goto yy5;
yy2:
yy3:


        ++YYCURSOR;


        { return result; }
yy5:


        ++YYCURSOR;


        { result = result * 10 + c; continue; }
}


    }
}

OPTIONS¶

-?, -h

-b

-c

-d

-D

-e

-f

-F

-g

-i

-o OUTPUT

-r

-s

-t

-u

-v

-V

-w

-x

-1

-8

--case-insensitive

--case-inverted

--no-generation-date

--encoding-policy POLICY

INTERFACE CODE¶

The user must supply interface code either in the form of C/C++ code (macros, functions, variables, etc.) or in the form of inplace configurations. Which symbols must be defined and which are optional depends on a particular use case.

YYCONDTYPE

YYCTXMARKER

YYCTYPE

YYCURSOR

YYDEBUG (state, current)

YYFILL (n)

YYGETCONDITION ()

YYGETSTATE ()

YYLIMIT

YYMARKER

YYMAXFILL

YYSETCONDITION (c)

YYSETSTATE (s)

SYNTAX¶

Code for re2c consists of a set of rules, named definitions and inplace configurations.

rules consist of a regular-expressions along with a block of C/C++ code that is to be executed when the associated regular-expression is matched. You can either start the code with an opening curly brace or the sequence :=. When the code with a curly brace then re2c counts the brace depth and stops looking for code automatically. Otherwise curly braces are not allowed and re2c stops looking for code at the first line that does not begin with whitespace. If two or more rules overlap, the first rule is preferred.

regular-expression { C/C++ code }

regular-expression := C/C++ code

There is one special rule: default rule *:

* { C/C++ code }

* := C/C++ code

If -c is active then each regular-expression is preceeded by a list of comma separated condition names. Besides normal naming rules there are two special cases. A rule may contain the single condition name * and no contition name at all. In the latter case the rule cannot have a regular-expression. Non empty rules may further more specify the new condition. In that case re2c will generated the necessary code to change the condition automatically. Just as above code can be started with a curly brace of the sequence :=. Further more rules can use :=> as a shortcut to automatically generate code that not only sets the new condition state but also continues execution with the new state. A shortcut rule should not be used in a loop where there is code between the start of the loop and the re2c block unless re2c:cond:goto is changed to continue. If code is necessary before all rule (though not simple jumps) you can doso by using <! pseudo-rules.

<condition-list> regular-expression { C/C++ code }

<condition-list> regular-expression := C/C++ code

<condition-list> * { C/C++ code }

<condition-list> * := C/C++ code

<condition-list> regular-expression => condition { C/C++ code }

<condition-list> regular-expression => condition := C/C++ code

<condition-list> regular-expression :=> condition

<*> regular-expression { C/C++ code }

<*> regular-expression := C/C++ code

<*> * { C/C++ code }

<*> * := C/C++ code

<*> regular-expression => condition { C/C++ code }

<*> regular-expression => condition := C/C++ code

<*> regular-expression :=> condition

<> { C/C++ code }

<> := C/C++ code

<> => condition { C/C++ code }

<> => condition := C/C++ code

<> :=> condition

<!condition-list> { C/C++ code }

<!condition-list> := C/C++ code

<!*> { C/C++ code }

<!*> := C/C++ code

named definitions are of the form:

name = regular-expression;

If -F is active, then named definitions are also of the form:

name regular-expression

inplace configurations are of the form:

re2c:name = value;

re2c:name = “_value_”;

REGULAR EXPRESSIONS¶

“foo”

‘foo’

[xyz]

[abj-oZ]

[^class]

r \ s

r *

r +

r ?

name

( r )

r s

r | s

r / s

r { n }

r { n , }

r { n , m }

.

def

Character classes and string literals may contain octal or hexadecimal character definitions and the following set of escape sequences: \a, \b, \f, \n, \r, \t, \v, \\. An octal character is defined by a backslash followed by its three octal digits (e.g. \377). Hexadecimal characters from 0 to 0xFF are defined by backslash, a lower cased ‘x’ and two hexadecimal digits (e.g. \x12). Hexadecimal characters from 0x100 to 0xFFFF are defined by backslash, a lower cased ‘u’ (or an upper cased ‘X’) and four hexadecimal digits (e.g. \u1234). Hexadecimal characters from 0x10000 to 0xFFFFffff are defined by backslash, an upper cased ‘U’ and eight hexadecimal digits (e.g. \U12345678).

The only portable “any” rule is the default rule *.

INPLACE CONFIGURATIONS¶

It is possible to configure code generation inside re2c blocks. The following lists the available configurations:

re2c:condprefix = yyc_;

re2c:condenumprefix = yyc;

re2c:cond:divider = “/* *********************************** */”;

re2c:cond:divider@cond = @@;

re2c:cond:goto = “goto @@;”;

re2c:cond:goto@cond = @@;

re2c:indent:top = 0;

re2c:indent:string = “\t”;

re2c:yych:conversion = 0;

re2c:yych:emit = 1;

re2c:yybm:hex = 0;

re2c:yyfill:enable = 1;

re2c:yyfill:check = 1;

re2c:yyfill:parameter = 1;

re2c:startlabel = 0;

re2c:labelprefix = yy;

re2c:state:abort = 0;

re2c:state:nextlabel = 0;

re2c:cgoto:threshold = 9;

re2c:yych:conversion = 0;

re2c:define:define:YYCONDTYPE = YYCONDTYPE;

re2c:define:YYCTXMARKER = YYCTXMARKER;

re2c:define:YYCTYPE = YYCTYPE;

re2c:define:YYCURSOR = YYCURSOR;

re2c:define:YYDEBUG = YYDEBUG;

re2c:define:YYFILL = YYFILL;

re2c:define:YYFILL:naked = 0;

re2c:define:YYFILL@len = @@;

re2c:define:YYGETCONDITION = YYGETCONDITION;

re2c:define:YYGETCONDITION:naked = 0;

re2c:define:YYGETSTATE = YYGETSTATE;

re2c:define:YYGETSTATE:naked = 0;

re2c:define:YYLIMIT = YYLIMIT;

re2c:define:YYMARKER = YYMARKER;

re2c:define:YYSETCONDITION = YYSETCONDITION;

re2c:define:YYSETCONDITION@cond = @@;

re2c:define:YYSETSTATE = YYSETSTATE;

re2c:define:YYSETSTATE:naked = 0;

re2c:define:YYSETSTATE@state = @@;

re2c:label:yyFillLabel = yyFillLabel;

re2c:label:yyNext = yyNext;

re2c:variable:yyaccept = yyaccept;

re2c:variable:yybm = yybm;

re2c:variable:yych = yych;

re2c:variable:yyctable = yyctable;

re2c:variable:yystable = yystable;

re2c:variable:yytarget = yytarget;

SCANNER WITH STORABLE STATES¶

When the -f flag is specified, re2c generates a scanner that can store its current state, return to the caller, and later resume operations exactly where it left off.

The default operation of re2c is a “pull” model, where the scanner asks for extra input whenever it needs it. However, this mode of operation assumes that the scanner is the “owner” the parsing loop, and that may not always be convenient.

Typically, if there is a preprocessor ahead of the scanner in the stream, or for that matter any other procedural source of data, the scanner cannot “ask” for more data unless both scanner and source live in a separate threads.

The -f flag is useful for just this situation: it lets users design scanners that work in a “push” model, i.e. where data is fed to the scanner chunk by chunk. When the scanner runs out of data to consume, it just stores its state, and return to the caller. When more input data is fed to the scanner, it resumes operations exactly where it left off.

When using the -f option re2c does not accept stdin because it has to do the full generation process twice which means it has to read the input twice. That means re2c would fail in case it cannot open the input twice or reading the input for the first time influences the second read attempt.

Changes needed compared to the “pull” model:

Please see examples/push.re for push-model scanner. The generated code can be tweaked using inplace configurations state:abort and state:nextlabel.

SCANNER WITH CONDITION SUPPORT¶

You can preceed regular expressions with a list of condition names when using the -c switch. In this case re2c generates scanner blocks for each conditon. Where each of the generated blocks has its own precondition. The precondition is given by the interface define YYGETCONDITON() and must be of type YYCONDTYPE.

There are two special rule types. First, the rules of the condition * are merged to all conditions. And second the empty condition list allows to provide a code block that does not have a scanner part. Meaning it does not allow any regular expression. The condition value referring to this special block is always the one with the enumeration value 0. This way the code of this special rule can be used to initialize a scanner. It is in no way necessary to have these rules: but sometimes it is helpful to have a dedicated uninitialized condition state.

Non empty rules allow to specify the new condition, which makes them transition rules. Besides generating calls for the define YYSETCONDTITION no other special code is generated.

There is another kind of special rules that allow to prepend code to any code block of all rules of a certain set of conditions or to all code blocks to all rules. This can be helpful when some operation is common among rules. For instance this can be used to store the length of the scanned string. These special setup rules start with an exclamation mark followed by either a list of conditions <! condition, ... > or a star <!*>. When re2c generates the code for a rule whose state does not have a setup rule and a star’d setup rule is present, than that code will be used as setup code.

ENCODINGS¶

re2c supports the following encodings: ASCII (default), EBCDIC (-e), UCS-2 (-w), UTF-16 (-x), UTF-32 (-u) and UTF-8 (-8). ASCII is default. You can either pass cmd flag or use inplace configuration in the form re2c:flags.

The following concepts should be clarified when talking about encoding. Code point is an abstract number, which represents single encoding symbol. Code unit is the smallest unit of memory, which is used in the encoded text (it corresponds to one character in the input stream). One or more code units can be needed to represent a single code point, depending on the encoding. In fixed-length encoding, each code point is represented with equal number of code units. In variable-length encoding, different code points can be represented with different number of code units.

ASCII

EBCDIC

UCS-2

UTF-16

UTF-32

UTF-8

In Unicode, values from range 0xD800 to 0xDFFF (surrogates) are not valid Unicode code points, any encoded sequence of code units, that would map to Unicode code points in the range 0xD800-0xDFFF, is ill-formed. The user can control how re2c treats such ill-formed sequences with --encoding-policy policy flag (see OPTIONS section for full explanation).

For some encodings, there are code units, that never occur in valid encoded stream (e.g. 0xFF byte in UTF-8). If the generated scanner must check for invalid input, the only true way to do so is to use default rule *. Note, that full range rule [^] won’t catch invalid code units when variable-length encoding is used ([^] means “all valid code points”, while default rule * means “all possible code units”: see Note about default rule in SYNTAX section).

GENERIC INPUT API¶

re2c usually operates on input using pointer-like primitives YYCURSOR, YYMARKER, YYCTXMARKER and YYLIMIT.

Generic input API (enabled with --input custom switch) allows to customize input operations. In this mode, re2c will express all operations on input in terms of the following primitives:

This article (http://skvadrik.github.io/aleph_null/posts/re2c/2015-01-13-input_model.html) has more details, and you can find some usage examples: http://skvadrik.github.io/aleph_null/posts/re2c/2015-01-15-input_model_custom.html .

UNDERSTANDING RE2C¶

The subdirectory lessons of the re2c distribution contains a few step by step lessons to get you started with re2c. All examples in the lessons subdirectory can be compiled and actually work.

BUGS¶

SEE ALSO¶

flex(1), lex(1), quex (http://quex.sourceforge.net)

More information on re2c can be found here: http://re2c.org/.

AUTHORS¶

VERSION INFORMATION¶

This manpage describes re2c, version 0.14.3, package date 20 May 2015.