scala.util.parsing.ast and parser combinators

Hello,

the following should do the trick :

package mypackage

import scala.util.parsing.input.Positional

//declare your tree
class CSRational extends Positional {
 ....
}

Manohar

On Sat, Jan 31, 2009 at 8:30 PM, Robert Fischer <robert.fischer@smokejumperit.com> wrote:

I'd like to track the source position with the AST elements that are generated.  In order to do this, I need to know the position at the time of the element's construction.

Most of my transformations are like the following example:

 def rational_number : Parser[CSRational] = (
   (rawFloat ^^ { new CSRational(_) }) |
   fraction |
   (rawInteger ^^ { new CSRational(_) })
 )

So how can I attach the position information?  I'd like to leverage scala.util.parsing.ast.AbstractSyntax.Element if at all possible.

Any examples about how to do that?

The source code of the parser is at http://snipr.com/b37f5

(Unsurprisingly, a follow up question will be how to work with Binders.)

~~ Robert Fischer.
Grails Trainining      http://www.smokejumperit.com/grails_training.html
Smokejumper Consulting http://smokejumperit.com
Enfranchised Mind Blog http://enfranchisedmind.com/blog

Check out my book, "Grails Persistence with GORM and GSQL"!
http://www.smokejumperit.com/redirect.html

That gives me a place to put the position. How does that position get set?

~~ Robert.

Manohar Jonnalagedda wrote:
> Hello,
>
> the following should do the trick :
>
> package mypackage
>
> import scala.util.parsing.input.Positional
>
> //declare your tree
> class CSRational extends Positional {
> ....
> }
>
> Manohar
>
> On Sat, Jan 31, 2009 at 8:30 PM, Robert Fischer
> > wrote:
>
> I'd like to track the source position with the AST elements that are
> generated. In order to do this, I need to know the position at the
> time of the element's construction.
>
> Most of my transformations are like the following example:
>
> def rational_number : Parser[CSRational] = (
> (rawFloat ^^ { new CSRational(_) }) |
> fraction |
> (rawInteger ^^ { new CSRational(_) })
> )
>
> So how can I attach the position information? I'd like to leverage
> scala.util.parsing.ast.AbstractSyntax.Element if at all possible.
>
> Any examples about how to do that?
>
> The source code of the parser is at http://snipr.com/b37f5
>
> (Unsurprisingly, a follow up question will be how to work with Binders.)
>
> ~~ Robert Fischer.
> Grails Trainining http://www.smokejumperit.com/grails_training.html
> Smokejumper Consulting http://smokejumperit.com
> Enfranchised Mind Blog http://enfranchisedmind.com/blog
>
> Check out my book, "Grails Persistence with GORM and GSQL"!
> http://www.smokejumperit.com/redirect.html
>
>

I think you use the positioned combinator:

def positioned [T <: Positional](p : => Parser[T]) : Parser[T]
`positioned' decorates a parser's result with the start position of
the input it consumed.

regards,
Tony

On 01/02/2009, at 4:23 PM, Robert Fischer wrote:

> That gives me a place to put the position. How does that position
> get set?
>
> ~~ Robert.
>
> Manohar Jonnalagedda wrote:
>> Hello,
>> the following should do the trick :
>> package mypackage
>> import scala.util.parsing.input.Positional
>> //declare your tree
>> class CSRational extends Positional {
>> ....
>> }
>> Manohar
>> On Sat, Jan 31, 2009 at 8:30 PM, Robert Fischer > > wrote:
>> I'd like to track the source position with the AST elements that
>> are
>> generated. In order to do this, I need to know the position at
>> the
>> time of the element's construction.
>> Most of my transformations are like the following example:
>> def rational_number : Parser[CSRational] = (
>> (rawFloat ^^ { new CSRational(_) }) |
>> fraction |
>> (rawInteger ^^ { new CSRational(_) })
>> )
>> So how can I attach the position information? I'd like to
>> leverage
>> scala.util.parsing.ast.AbstractSyntax.Element if at all possible.
>> Any examples about how to do that?
>> The source code of the parser is at http://snipr.com/b37f5
>> (Unsurprisingly, a follow up question will be how to work with
>> Binders.)
>> ~~ Robert Fischer.
>> Grails Trainining http://www.smokejumperit.com/grails_training.html
>> Smokejumper Consulting http://smokejumperit.com
>> Enfranchised Mind Blog http://enfranchisedmind.com/blog
>> Check out my book, "Grails Persistence with GORM and GSQL"!
>> http://www.smokejumperit.com/redirect.html
>

Okay -- so I have to wrapping every AST-constructing call with "positioned" will take care of it?

~~ Robert.

Tony Sloane wrote:
> I think you use the positioned combinator:
>
> def positioned [T <: Positional](p : => Parser[T]) : Parser[T]
> `positioned' decorates a parser's result with the start position of the
> input it consumed.
>
> regards,
> Tony
>
> On 01/02/2009, at 4:23 PM, Robert Fischer wrote:
>
>> That gives me a place to put the position. How does that position get
>> set?
>>
>> ~~ Robert.
>>
>> Manohar Jonnalagedda wrote:
>>> Hello,
>>> the following should do the trick :
>>> package mypackage
>>> import scala.util.parsing.input.Positional
>>> //declare your tree
>>> class CSRational extends Positional {
>>> ....
>>> }
>>> Manohar
>>> On Sat, Jan 31, 2009 at 8:30 PM, Robert Fischer
>>> >
>>> wrote:
>>> I'd like to track the source position with the AST elements that are
>>> generated. In order to do this, I need to know the position at the
>>> time of the element's construction.
>>> Most of my transformations are like the following example:
>>> def rational_number : Parser[CSRational] = (
>>> (rawFloat ^^ { new CSRational(_) }) |
>>> fraction |
>>> (rawInteger ^^ { new CSRational(_) })
>>> )
>>> So how can I attach the position information? I'd like to leverage
>>> scala.util.parsing.ast.AbstractSyntax.Element if at all possible.
>>> Any examples about how to do that?
>>> The source code of the parser is at http://snipr.com/b37f5
>>> (Unsurprisingly, a follow up question will be how to work with
>>> Binders.)
>>> ~~ Robert Fischer.
>>> Grails Trainining
>>> http://www.smokejumperit.com/grails_training.html
>>> Smokejumper Consulting http://smokejumperit.com
>>> Enfranchised Mind Blog http://enfranchisedmind.com/blog
>>> Check out my book, "Grails Persistence with GORM and GSQL"!
>>> http://www.smokejumperit.com/redirect.html
>>

I think that is the intention, although you can presumably hide these
calls
in another combinator that combines your AST node construction with
position
annotation.

Tony

On 02/02/2009, at 1:59 AM, Robert Fischer wrote:

> Okay -- so I have to wrapping every AST-constructing call with
> "positioned" will take care of it?
>
> ~~ Robert.
>
> Tony Sloane wrote:
>> I think you use the positioned combinator:
>> def positioned [T <: Positional](p : => Parser[T]) : Parser[T]
>> `positioned' decorates a parser's result with the start position of
>> the input it consumed.
>> regards,
>> Tony
>> On 01/02/2009, at 4:23 PM, Robert Fischer wrote:
>>> That gives me a place to put the position. How does that position
>>> get set?
>>>
>>> ~~ Robert.
>>>
>>> Manohar Jonnalagedda wrote:
>>>> Hello,
>>>> the following should do the trick :
>>>> package mypackage
>>>> import scala.util.parsing.input.Positional
>>>> //declare your tree
>>>> class CSRational extends Positional {
>>>> ....
>>>> }
>>>> Manohar
>>>> On Sat, Jan 31, 2009 at 8:30 PM, Robert Fischer >>> > wrote:
>>>> I'd like to track the source position with the AST elements
>>>> that are
>>>> generated. In order to do this, I need to know the position at
>>>> the
>>>> time of the element's construction.
>>>> Most of my transformations are like the following example:
>>>> def rational_number : Parser[CSRational] = (
>>>> (rawFloat ^^ { new CSRational(_) }) |
>>>> fraction |
>>>> (rawInteger ^^ { new CSRational(_) })
>>>> )
>>>> So how can I attach the position information? I'd like to
>>>> leverage
>>>> scala.util.parsing.ast.AbstractSyntax.Element if at all possible.
>>>> Any examples about how to do that?
>>>> The source code of the parser is at http://snipr.com/b37f5
>>>> (Unsurprisingly, a follow up question will be how to work with
>>>> Binders.)
>>>> ~~ Robert Fischer.
>>>> Grails Trainining http://www.smokejumperit.com/grails_training.html
>>>> Smokejumper Consulting http://smokejumperit.com
>>>> Enfranchised Mind Blog http://enfranchisedmind.com/blog
>>>> Check out my book, "Grails Persistence with GORM and GSQL"!
>>>> http://www.smokejumperit.com/redirect.html
>>>
>>> --
>>> ~~ Robert Fischer.
>>> Grails Trainining http://www.smokejumperit.com/grails_training.html
>>> Smokejumper Consulting http://smokejumperit.com
>>> Enfranchised Mind Blog http://enfranchisedmind.com/blog
>>>
>>> Check out my book, "Grails Persistence with GORM and GSQL"!
>>> http://www.smokejumperit.com/redirect.html
>

Hi! I'm currently using the StandardTokenParsers class.
The problem is that it is choking on the letter underscore, '_',
and also on tabs '\t' with the message:

(failure: ``)'' expected but ErrorToken(illegal character) found).

Strangely enough it doesn't choke on japanese nor swedish
characters. Any suggestions?

danielswe schrieb:
> The problem is that it is choking on the letter underscore, '_',

This is not a letter at all, so it should choke if you give it
an underscore when it expects a letter. Note that scala accepts a
different set of characters in an identifier than StdLexical.
StdLexical only accepts a letter in Ll, Lu, Lt, Lm and Lo
followed by decimal digits or letters in said classes.

Scala thinks that $ and _ and charachters in Nl are letters, but
excludes Lm, and in addition accepts some operator characters,
math symbols and other symbols in identifiers, if you follow certain
rules.

Curiously, it doesn't allow other numbers, so

scala> val äↂ_⅀™∰ = 1
äↂ_⅀™∰: Int = 1

is valid, but this isn't:

scala> val ä⅚_⅀™∰ = 1
:1: error: illegal character: '⅚'
val ä⅚_⅀™∰ = 1

> and also on tabs '\t' with the message:
> (failure: ``)'' expected but ErrorToken(illegal character) found).

That is weird, as it should ignore whitespace.
Are you perchance using triple quoted strings to define the text you are
trying to parse?

scala> "\\t" == """\t"""
res1: Boolean = true

scala> "\t" == """\t"""
res2: Boolean = false

> Strangely enough it doesn't choke on japanese nor swedish
> characters.

Nothing strange here, those *are* indeed letters.

- Florian.