2.27. Macros

In daScript macros are the machinery which allows direct manipulation of a syntax tree.

Macros are exposed via daslib/ast module and daslib/ast_boost helper module.

Macros are evaluated at compilation time during different compilation passes. Macros assigned to specific module are evaluated as part of the module, every time module is included.

2.27.1. Compilation passes

daScript compiler performs compilation passes in the following order for each module (see Modules)

  1. parser transforms das program to AST

    1. if there are any parsing errors, compilation stops

  2. apply is called for every function or structure

    1. if there are any errors, compilation stops

  3. infer pass repeats itself until no new transformations are reported

    1. built-in infer pass happens

      1. transform macros are called for every function, or expression

    2. macro passes happen

  4. if there are still any errors left, compilation stops

  5. finish is called for all function and structure macros

  6. lint pass happens

    1. if there are any errors, compilation stops

  7. optimization pass repeats itself until no new transformations are reported

    1. built-in optimization pass happens

    2. macro optimization pass happens

  8. if there are any errors during optimization passes, compilation stops

  9. if module contains any macros, simulation happens

    1. if there are any simulation errors, compilation stops

    2. module macros functions (annotated with _macro) are invoked

      1. if there are any errors, compilation stops

Modules are compiled in require order.

2.27.2. Invoking macros

To specify function to be evaluated in compilation time [_macro] annotation is used. Consider the following example from daslib/ast_boost:

[_macro,private]
def setup
    if is_compiling_macros_in_module("ast_boost")
        add_new_function_annotation("macro", new MacroMacro())

setup function will be evaluated after compilation of each module, which includes ast_boost. is_compiling_macros_in_module function returns true if currently compiled module name matches the argument. In this particular example function annotation macro would only be added once, when the module ast_boost is compiled.

Macros are invoked in the following fashion:

  1. class is derived from the appropriate base macro class

  2. adapter is created

  3. adapter is registered with the module

For example this is how this lifetime cycle is implemented for the reader macro:

def add_new_reader_macro ( name:string; someClassPtr )
    var ann <- make_reader_macro(name, someClassPtr)
    this_module() |> add_reader_macro(ann)
    unsafe
        delete ann

2.27.3. AstFunctionAnnotation

AstFunctionAnnotation macro allows to manipulate call to specific function as well as function body. Annotation can be added to regular or generic function.

add_new_function_annotation adds function annotation to a module. There is additionally [function_macro] annotation which accomplishes the same thing.

AstFunctionAnnotation allows 3 different manipulations:

class AstFunctionAnnotation
    def abstract transform ( var call : smart_ptr<ExprCall>; var errors : das_string ) : ExpressionPtr
    def abstract apply ( var func:FunctionPtr; var group:ModuleGroup; args:AnnotationArgumentList; var errors : das_string ) : bool
    def abstract finish ( var func:FunctionPtr; var group:ModuleGroup; args,progArgs:AnnotationArgumentList; var errors : das_string ) : bool
    def abstract isCompatible ( var func:FunctionPtr; var types:VectorTypeDeclPtr; decl:AnnotationDeclaration; var errors : das_string ) : bool
    def abstract isSpecialized : bool

transform allows changing call to the function and is applied at infer pass. Transform is the best way to replace or modify function call with other semantics.

apply is applied to function itself before the infer pass. Apply is typically where global function body modifications or instancing occurs.

finish is applied to function itself after the infer pass. It’s only called to non-generic functions or instances of the generic functions. finish is typically used to register functions, notify C++ code, etc. Function is fully defined and inferred, and can no longer be modified.

isSpecialized must return true, if the particular function matching is governed by contracts. In that case isCompatible will be called, and result taken into account.

isCompatible returns true, if specialized function is compatible with arguments. If function is not compatible, errors field must be specified.

Lets review the following example from ast_boost of how macro annotation is implemented:

class MacroMacro : AstFunctionAnnotation
    def override apply ( var func:FunctionPtr; var group:ModuleGroup; args:AnnotationArgumentList; var errors : das_string ) : bool
        compiling_program().flags |= ProgramFlags needMacroModule
        func.flags |= FunctionFlags init
        var blk <- new [[ExprBlock() at=func.at]]
        var ifm <- new [[ExprCall() at=func.at, name:="is_compiling_macros"]]
        var ife <- new [[ExprIfThenElse() at=func.at, cond<-ifm, if_true<-func.body]]
        push(blk.list,ife)
        func.body <- blk
        return true

During the apply pass function body is appended with if is_compiling_macros() closure, additionally init flag is set, which is equivalent to _macro annotation. Function annotated with [macro] will be evaluated during module compilation.

2.27.4. AstStructureAnnotation

AstStructureAnnotation macro allows to manipulate structure or class definitions via annotation.

add_new_structure_annotation adds structure annotation to a module.

AstStructureAnnotation allows 2 different manipulations:

class AstStructureAnnotation
    def abstract apply ( var st:StructurePtr; var group:ModuleGroup; args:AnnotationArgumentList; var errors : das_string ) : bool
    def abstract finish ( var st:StructurePtr; var group:ModuleGroup; args:AnnotationArgumentList; var errors : das_string ) : bool

apply is invoked before the infer pass. It is the best time to modify structure, generated some code, etc.

finish is invoked after the successful infer pass. Its typically used to register structures, perform RTTI operations etc. Structure is fully inferred and defined and can no longer be modified afterwards.

Example of such annotation is SetupAnyAnnotation from daslib/ast_boost.

2.27.5. AstVariantMacro

AstVariantMacro is specialized in transforming is, as, and ?as expressions.

add_new_variant_macro adds variant macro to a module. There is additionally [variant_macro] annotation which accomplishes the same thing.

Each of the 3 transformations are covered in appropriate abstract function:

class AstVariantMacro
    def abstract visitExprIsVariant     ( prog:ProgramPtr; mod:Module?; expr:smart_ptr<ExprIsVariant> ) : ExpressionPtr
    def abstract visitExprAsVariant     ( prog:ProgramPtr; mod:Module?; expr:smart_ptr<ExprAsVariant> ) : ExpressionPtr
    def abstract visitExprSafeAsVariant ( prog:ProgramPtr; mod:Module?; expr:smart_ptr<ExprSafeAsVariant> ) : ExpressionPtr

Lets review the following example from daslib/ast_boost:

// replacing ExprIsVariant(value,name) => ExprOp2('==",value.__rtti,"name")
// if value is ast::Expr*
class BetterRttiVisitor : AstVariantMacro
    def override visitExprIsVariant(prog:ProgramPtr; mod:Module?;expr:smart_ptr<ExprIsVariant>) : ExpressionPtr
        if isExpression(expr.value._type)
            var vdr <- new [[ExprField() at=expr.at, name:="__rtti", value <- clone_expression(expr.value)]]
            var cna <- new [[ExprConstString() at=expr.at, value:=expr.name]]
            var veq <- new [[ExprOp2() at=expr.at, op:="==", left<-vdr, right<-cna]]
            return veq
        return [[ExpressionPtr]]

// note the following ussage
class GetHintFnMacro : AstFunctionAnnotation
    def override transform ( var call : smart_ptr<ExprCall>; var errors : das_string ) : ExpressionPtr
        if call.arguments[1] is ExprConstString     // HERE EXPRESSION WILL BE REPLACED
            ...

Here the macro takes advantage of the ExprIsVariant syntax. It replaces expr is TYPENAME expression with expr.__rtti = "TYPENAME" expression. isExpression function ensures that expr is from the ast::Expr* family, i.e. part of the daScript syntax tree.

2.27.6. AstReaderMacro

AstReaderMacro allows embedding completely different syntax inside daScript code.

add_new_reader_macro adds reader macro to a module. There is additionally reader_macro annotation, which essentially automates the same thing.

Reader macro accepts characters, collects them if necessary, and returns ast::Expression:

class AstReaderMacro
    def abstract accept ( prog:ProgramPtr; mod:Module?; expr:ExprReader?; ch:int; info:LineInfo ) : bool
    def abstract visit ( prog:ProgramPtr; mod:Module?; expr:smart_ptr<ExprReader> ) : ExpressionPtr

Reader macros are invoked via following syntax % READER_MACRO_NAME ~ character_sequence. accept function notifies the correct terminator of the character sequence:

var x = %arr~\{\}\w\x\y\n%% // invoking reader macro arr, %% is a terminator

Consider the implementation for the example above:

[reader_macro(name="arr")]
class ArrayReader : AstReaderMacro
    def override accept ( prog:ProgramPtr; mod:Module?; var expr:ExprReader?; ch:int; info:LineInfo ) : bool
        append(expr.sequence,ch)
        if ends_with(expr.sequence,"%%")
            let len = length(expr.sequence)
            resize(expr.sequence,len-2)
            return false
        else
            return true
    def override visit ( prog:ProgramPtr; mod:Module?; expr:smart_ptr<ExprReader> ) : ExpressionPtr
        let seqStr = string(expr.sequence)
        var arrT <- new [[TypeDecl() baseType=Type tInt]]
        push(arrT.dim,length(seqStr))
        var mkArr <- new [[ExprMakeArray() at = expr.at, makeType <- arrT]]
        for x in seqStr
            var mkC <- new [[ExprConstInt() at=expr.at, value=x]]
            push(mkArr.values,mkC)
        return mkArr

In accept function macro collects symbols in the sequence. Once the sequence ends with the terminator sequence %%, accept returns false to notify for the end of read.

In visit the collected sequence is converted into make array [[int ch1; ch2; ..]] expression.

More complex examples are JsonReader macro in daslib/json_boost or RegexReader in daslib/regex_boost.

2.27.7. AstCallMacro

AstCallMacro operates on expressions, which have similar to function call syntax. It occurs during the infer pass.

add_new_call_macro adds call macro to a module. [call_macro] annotation automates the same thing:

class AstCallMacro
    def abstract visit ( prog:ProgramPtr; mod:Module?; expr:smart_ptr<ExprCallMacro> ) : ExpressionPtr

apply from the daslib/apply is an example of such macro:

[call_macro(name="apply")]  // apply(value, block)
class ApplyMacro : AstCallMacro
    def override visit ( prog:ProgramPtr; mod:Module?; var expr:smart_ptr<ExprCallMacro> ) : ExpressionPtr
        ...

Note how name is provided in the [call_macro] annotation.

2.27.8. AstPassMacro

AstPassMacro is one macro to rule them all. It gets entire module as an input, and can be invoked at numerous passes:

class AstPassMacro
    def abstract apply ( prog:ProgramPtr; mod:Module? ) : bool

make_pass_macro registers class as a pass macro.

add_dirty_infer_macro adds pass macro to the infer pass.

Typically such macro creates an AstVisitor which performs necessary transformations.

2.27.9. AstTypeInfoMacro

AstTypeInfoMacro is designed to implement custom type information inside typeinfo expression:

class AstTypeInfoMacro
    def abstract getAstChange ( expr:smart_ptr<ExprTypeInfo>; var errors:das_string ) : ExpressionPtr
    def abstract getAstType ( var lib:ModuleLibrary; expr:smart_ptr<ExprTypeInfo>; var errors:das_string ) : TypeDeclPtr

getAstChange returns newly generated ast for the typeinfo expression. Alternatively it returns null if no changes are required, or if there is an error. In case of error errors string must be filled.

getAstType returns type of the new typeinfo expression.

2.27.10. AstVisitor

AstVisitor implements visitor pattern for the daScript expression tree. It contains callback for every single expression in prefix and postfix form, as well as some additional callbacks:

class AstVisitor
    ...
    // find
        def abstract preVisitExprFind(expr:smart_ptr<ExprFind>) : void          // prefix
        def abstract visitExprFind(expr:smart_ptr<ExprFind>) : ExpressionPtr    // postifx
    ...

Postfix callback can return expression to replace the one passed to the callback.

PrintVisitor form ast_print example implements printing of every single expression in daScript syntax.

make_visitor creates visitor adapter from the class, derived from the AstVisitor. Adapter then can be applied to a program via visit function:

var astVisitor = new PrintVisitor()
var astVisitorAdapter <- make_visitor(*astVisitor)
visit(this_program(), astVisitorAdapter)