macros as arguments to high-level functions

Hi Ittay,

Fun thing - I just watched a Factor video during this week-end. It was
the Google tech talk named "Factor, an extensible interactive
language". Nice language tbh, I'm really impressed by the fancyness of
combinators in concatenative languages:
http://elasticdog.com/2008/12/beginning-factor-shufflers-and-combinators/.
Though I digress.

Speaking of first-class macros. Long story short - that'd be very
challenging to implement such stuff. The main problem here is not the
necessity to compile/recompile the code during the runtime (this area
is quite explored, say, in Da Vinci Machine project or in DLR), but
rather the need to have a full-blown compiler environment at the macro
expansion point.

1) Say, we have the following code:
val macro = if (phaseOfMoon == fullMoon) printf else
fprintf("foo.bar")
macro("hello %s", "world")

How are we supposed to compile this? Okay, say, we can reason that
printf and partially applied fprintf actually have compatible
signatures (by "compatible" I mean that they have the same number of
arguments; we cannot reason about types when working with partially
applied macros, because subsequent application can generate whatever
code it likes). Then we have to generate some runtime stub of
signature "Any => Any* => Any" that calls the compiler during the
runtime and emit it for both branches of the conditional. All in all,
we had some problems, but it seems doable.

2) All right, it's more or less okay for printf, since it's self-
contained (in the sense that it only works with its arguments), but
other macros might wish to interact with the lexical scope (e.g. by
binding to variables in scope or by introducing new variables). To do
that we need to have full information about Scala scopes during the
runtime, which is quite problematic in itself, but there's even more
to it.

How exactly do we emit the code that manipulates locals of some
method? A common idiom to represent closures in a runtime that doesn't
support this concept (JVM, CLR) is to lift involved locals into
synthetic classes and rewrite all operations that operate upon those
locals (http://blogs.msdn.com/b/abhinaba/archive/
2005/10/18/482180.aspx). However, in general case this requires
rewriting the body of the method that hosts the method, which is our
case would lead to the necessity of regenerating and re-JITting the
code, which is a high-order acrobacy.

3) Finally, the macro might also like to operate on exact shapes of
the arguments passed into it. Take a look at the printf example at the
page 5 of my presentation (http://scalamacros.org/talks/2011-09-27-
ProjectKepler.pdf). The code produced by expanding the "printf("Value
= %d", 123 + 877)" macro call contains the "123 + 877" part verbatim,
since the printf macro accepts not values but ASTs (i.e. the parameter
it will receive will have the shape of Plus(Constant(123),
Constant(877)), not Constant(1000)). This is moot for printf, but
might be important for other macros (say, for LINQ). We could work
around this as well by implementing some sort pessimistic analysis and
lifting (i.e. turning into runtime representation of ASTs) all
expressions that might ever be parameters to macros, but this is quite
a challenge. Also we could ask the developer himself to manually lift
the parameters in question, but this also has its downsides.

Interestingly enough, Factor doesn't suffer from these problems, since
its computational model is much simpler (no notion of locals (though
locals can be emulated), no expressions). That's another example
(along with Lisp) of how syntactic minimalism can be beneficial for
metaprogramming.

All in all, truely first-class macros are not impossible, but they
present several significant implementation challenges. Depending on
the situation, there might exist certain workarounds (as for printf
which doesn't interact with lexical environment), but none of them
seem to be general.

Cheers,
Eugene

On Nov 29, 7:04 am, Ittay Dror wrote:
> Hi,
>
> I was watching a video about Factor. It demonstrated how printf could be written as a macro. The interesting bit was that printf could then be passed as an argument to a map function and mapped over a list of format string.  The language figures it out and the macro expansion is then done at runtime (when it is applied to each string) instead of compile time.
>
> I think it is an important feature. Will scala macros support this?
>
> Regads,
>
> Ittay