Two related proposals: lightweight extension methods, and extension classes

After some thought, I need to make a couple of amendments to this proposal:

1. Extension classes must have exactly one default constructor parameter.
2. The @ExtensionClass annotation does not need to have an extendedClass parameter. This is implied by the class's default constructor argument.
3. In my generated code section, I put the @ExtensionClass annotation on the ExtensionClass companion object - I meant to retain it on the class itself.
4. If the compiler would be made to allow the code to call a lightweight extension method after an implicit conversion, then the generated extension class need not define copies of each lightweight extension method. Rather, the compiler simply needs to generate an implicit conversion from the extension class back to the original type:

// compiler generated code:object MyExtensionClass {

// only one copy of extensionMethod is needed in the final binary@ExtensionMethod def extensionMethod(self: SealedType, arg: ArgType): Unit = ...
// implicitConversionMethod is not defined here
// convert SealedType to MyExtensionClass

implicit def sealedTypeToMyExtensionClass(self: SealedType) = {

new MyExtensionClass(self)

}

// compiler can convert back to SealedType, after which the code can
// call lightweight extension methods directly on the SealedType instance
// so there would be no need to redefine them in the wrapper class
implicit def myExtensionClassToSealedType(wrapper: ExtensionClass) = {

wrapper.self

}

}
@ExtensionClass class MyExtensionClass(val self: SealedType) {

private var additionalVar = 3

// no need to generate a redefinition of extensionMethod here,// unless the retainLightweightExtensionMethods parameter of  // the @ExtensionClass annotation is set to true (see note below)
// only the methods which reference other fields besides self// need to remain in the extension class

def implicitConversionMethod(arg: ArgType): Unit = ...

}

The drawback to this is that code which expects MyExtensionClass to define a method extensionMethod will have to be recompiled so that it uses an implicit conversion back to the SealedType instead. To alleviate this, we could add an optional parameter to the ExtensionClass annotation:
@ExtensionClass(retainLightweightExtensionMethods = true)
This would retain all methods in an extension class (they would simply be redirects to the extension methods, as in my original example), even if they are implemented using lightweight extension methods. Thus binary compatibility would be maintained at the expense of binary size: 

// generated code@ExtensionClass(retainLightweightExtensionMethods = true) class MyExtensionClass(val self: SealedType) {

private var additionalVar = 3
// extension method is retained in the wrapper class, for binary// compatibility. The method just calls the lightweight extension // method directly.

def extensionMethod(arg: ArgType) = MyExtensionClass.extensionMethod(self, arg)

def implicitConversionMethod(arg: ArgType): Unit = ...

}

On Tue, Jun 1, 2010 at 12:36 PM, Jeremy Bell <bell.jeremy@gmail.com> wrote:

First, some background: 
As you know, in Scala the current idiomatic way of adding a new method to an existing type is to:

1. Create a new class that defines the new method. Usually this class takes the existing type as a default constructor parameter.
2. Write an implicit conversion function to convert the existing type into the new "extension" type.
3. Call the new method on an instance of the existing type. Compiler generates an implicit conversion to an instance of this new type, the new method is called on this temporary instance, and then the temporary instance is discarded.

This works great for the most part, but there are drawbacks:

1. It is cumbersome to write both the new class and the implicit conversion, if just a small number of extension methods are needed. 
2. The implicit conversion and the "wrapper" class are highly related, but often separated in the code.
3. For many use cases, the "boxing" operation performed during an implicit conversion is needlessly inefficient. Every time one of the new methods are called, a new temporary object is constructed, then discarded after the method is called. It's possible that escape analysis can reduce the cost of this operation, but not eliminate it completely.

To address these issues, I propose two new features: lightweight extension methods, and a related proposal: extension classes.
Lightweight extension methods:
A lightweight extension method is simply a method defined in an object which takes at least one parameter: an instance of the original type. In addition, the method is marked with an annotation declaring it an extension method:

package somelibobject ExtensionMethods {

@ExtensionMethod def extensionMethod(self: SealedType, args: ArgType): ReturnType = ......

}

 If the ExtensonMethods object is imported (i.e. "import somelib.ExtensionMethods._"), and the extensionMethod method is called on an instance of SealedType, the following transformation occurs:

import somelib.ExtensionMethods._

var original = new SealedType
// original code original.extensionMethod(new ArgType)
// compiler generated code: extensionMethod(original, new ArgType)// in other words: somelib.ExtensonMethods.extensionMethod(original, new ArgType)

As you can see, no temporary object needs to be generated, as we are simply calling an object method on an instance of the original type.
The benefits are:

1. It is a lot simpler to define a single extension method in an object (possibly along with other extension methods of different types) than to do an implicit conversion to a new type. This is especially helpful for defining internal DSLs.
2. It is more efficient, because no boxing of the original instance is necessary to call the extension method.

The drawback is:

1. If in the future you decide to create a wrapper class after all, with an implicit conversion, and you want to include the extension methods you've already defined, then you need to redefine all those methods in the wrapper class and call them on the original instance.

To address this drawback, I propose the concept of an extension class. An extension class is a short-hand for defining both a wrapper class and the implicit conversion needed. However, in addition the compiler can potentially optimize by generating extension methods when possible:

// Note: MyExtensionClass could potentially mixin traits or inherit another class @ExtensionClass(extendedClass = classOf[SealedType])// default constructor must have at least one argument: the wrapped instance. // if more than one argument, then the first argument is used as the wrapped instance// we call it self here as a convention only class MyExtensionClass(val self: SealedType) { 

private var additionalVar = 3 // can define vars

def extensionMethod(arg: ArgType): Unit = println(arg.toString)def implicitConversionMethod(arg: ArgType): Unit = println(additionalVar.toString + arg.toString)

}

The compiler should at a minimum generate an implicit conversion to this extension class with all the methods defined as usual.
However, in addition, the compiler may be able to determine whether a particular method can be implemented using a lightweight extension method. For example, in the above MyExtensionClass, the extensionMethod method can be defined as both a lightweight extension method and a method of MyExtensionClass (which simply calls the lightweight extension method), because the only field of MyExtensionClass it references is the wrapped SealedType. However, the implicitConversionMethod cannot be made into a lightweight extension method, because it references the extension class's additionalVar field, so an implicit conversion will always be done when the implicitConversionMethod is called"

// compiler generated code: @ExtensionClass(extendedClass = classOf[SealedType]) // annotation is retainedobject MyExtensionClass {

@ExtensionMethod def extensionMethod(self: SealedType, arg: ArgType): Unit = ...// implicitConversionMethod is not defined here

implicit def sealedTypeToMyExtensionClass(self: SealedType) = new MyExtensionClass(self)

}
class MyExtensionClass(val self: SealedType) {

private var additionalVar = 3

def extensionMethod(arg: ArgType): Unit = MyExtensionClass.extensionMethod(self, arg)

def implicitConversionMethod(arg: ArgType): Unit = ...

}

The great thing about this is that the code need not care whether the compiler generates lightweight extension methods or not - it will work the same either way. This means it may be possible to convert existing implicit conversions into an extension class without breaking client code:

import somelib._ import somelib.MyExtensionClass._
// code can use references to MyExtensionClass, and they work like you would expect:def foo1(arg: MyExtensionClass) {

arg.implicitConversionMethod(new ArgType)
// code can still call methods that are converted to lightweight extension methods// but in this case it calls the extensionMethod defined in MyExtensionClass, because // we're working on an instance of MyExtensionClass class. That method just redirects to// the lightweight extension method defined in the MyExtensionClass companion object. arg.extensionMethod(new ArgType) 

}
def foo(arg: SealedType): Unit = {

// calling code doesn't care how the compiler implements the extension methods:

arg.extensionMethod(new ArgType) // calls lightweight extension method, no implicit conversion needed

arg.implicitConversionMethod(new ArgType) // implicit conversion to MyExtensionClass required foo2(arg) // implicit conversion to MyExtensionClass required

}

Ouch, I can feel your pain here!  We all go through a phase of thinking pimp-my-library / extension methods are wastefully inefficient.  "what, you mean if I chain together 6 such extension methods then it'll be implicitly converted 6 times, but that's crazy!"
This feeling usually passes after a few sharp whacks round the head with a good clue stick, plus a bit of knowledge what's going on under the covers:
1. The efficiency argument is a non-starter.  It's a common misconception that this pattern is inefficient (yup, I fell prey to it as well).  The truth is that such conversions are optimised out (so no temp object created).  Even if this wasn't the case, temp objects are really really cheap, far more so than common sense would imply.

2. As for the separation issue; if you're "pimping" a 3rd party library then this is unavoidable with *any* approach.  If it's all your own code then any separation is purely self-imposed.  Scala's good to us like that - we can even put multiple classes in the one file. (hey, if it's good enough for the compiler...)
3. Boilerplate reduction doesn't help much either, you're substituting a 1-line method for a 1-line annotation, and introducing new syntax so that you can do so.  Scala really doesn't need new syntax, we're still coming to terms with the alternate uses of underscores!


On 1 June 2010 17:36, Jeremy Bell <bell.jeremy@gmail.com> wrote:

First, some background: 
As you know, in Scala the current idiomatic way of adding a new method to an existing type is to:

1. Create a new class that defines the new method. Usually this class takes the existing type as a default constructor parameter.
2. Write an implicit conversion function to convert the existing type into the new "extension" type.
3. Call the new method on an instance of the existing type. Compiler generates an implicit conversion to an instance of this new type, the new method is called on this temporary instance, and then the temporary instance is discarded.

This works great for the most part, but there are drawbacks:

1. It is cumbersome to write both the new class and the implicit conversion, if just a small number of extension methods are needed. 
2. The implicit conversion and the "wrapper" class are highly related, but often separated in the code.
3. For many use cases, the "boxing" operation performed during an implicit conversion is needlessly inefficient. Every time one of the new methods are called, a new temporary object is constructed, then discarded after the method is called. It's possible that escape analysis can reduce the cost of this operation, but not eliminate it completely.

To address these issues, I propose two new features: lightweight extension methods, and a related proposal: extension classes.
Lightweight extension methods:
A lightweight extension method is simply a method defined in an object which takes at least one parameter: an instance of the original type. In addition, the method is marked with an annotation declaring it an extension method:

package somelibobject ExtensionMethods {

@ExtensionMethod def extensionMethod(self: SealedType, args: ArgType): ReturnType = ......

}

 If the ExtensonMethods object is imported (i.e. "import somelib.ExtensionMethods._"), and the extensionMethod method is called on an instance of SealedType, the following transformation occurs:

import somelib.ExtensionMethods._

var original = new SealedType
// original code original.extensionMethod(new ArgType)
// compiler generated code: extensionMethod(original, new ArgType)// in other words: somelib.ExtensonMethods.extensionMethod(original, new ArgType)

As you can see, no temporary object needs to be generated, as we are simply calling an object method on an instance of the original type.
The benefits are:

1. It is a lot simpler to define a single extension method in an object (possibly along with other extension methods of different types) than to do an implicit conversion to a new type. This is especially helpful for defining internal DSLs.
2. It is more efficient, because no boxing of the original instance is necessary to call the extension method.

The drawback is:

1. If in the future you decide to create a wrapper class after all, with an implicit conversion, and you want to include the extension methods you've already defined, then you need to redefine all those methods in the wrapper class and call them on the original instance.

To address this drawback, I propose the concept of an extension class. An extension class is a short-hand for defining both a wrapper class and the implicit conversion needed. However, in addition the compiler can potentially optimize by generating extension methods when possible:

// Note: MyExtensionClass could potentially mixin traits or inherit another class @ExtensionClass(extendedClass = classOf[SealedType])// default constructor must have at least one argument: the wrapped instance. // if more than one argument, then the first argument is used as the wrapped instance// we call it self here as a convention only class MyExtensionClass(val self: SealedType) { 

private var additionalVar = 3 // can define vars

def extensionMethod(arg: ArgType): Unit = println(arg.toString)def implicitConversionMethod(arg: ArgType): Unit = println(additionalVar.toString + arg.toString)

}

The compiler should at a minimum generate an implicit conversion to this extension class with all the methods defined as usual.
However, in addition, the compiler may be able to determine whether a particular method can be implemented using a lightweight extension method. For example, in the above MyExtensionClass, the extensionMethod method can be defined as both a lightweight extension method and a method of MyExtensionClass (which simply calls the lightweight extension method), because the only field of MyExtensionClass it references is the wrapped SealedType. However, the implicitConversionMethod cannot be made into a lightweight extension method, because it references the extension class's additionalVar field, so an implicit conversion will always be done when the implicitConversionMethod is called"

// compiler generated code: @ExtensionClass(extendedClass = classOf[SealedType]) // annotation is retainedobject MyExtensionClass {

@ExtensionMethod def extensionMethod(self: SealedType, arg: ArgType): Unit = ...// implicitConversionMethod is not defined here

implicit def sealedTypeToMyExtensionClass(self: SealedType) = new MyExtensionClass(self)

}
class MyExtensionClass(val self: SealedType) {

private var additionalVar = 3

def extensionMethod(arg: ArgType): Unit = MyExtensionClass.extensionMethod(self, arg)

def implicitConversionMethod(arg: ArgType): Unit = ...

}

The great thing about this is that the code need not care whether the compiler generates lightweight extension methods or not - it will work the same either way. This means it may be possible to convert existing implicit conversions into an extension class without breaking client code:

import somelib._ import somelib.MyExtensionClass._
// code can use references to MyExtensionClass, and they work like you would expect:def foo1(arg: MyExtensionClass) {

arg.implicitConversionMethod(new ArgType)
// code can still call methods that are converted to lightweight extension methods// but in this case it calls the extensionMethod defined in MyExtensionClass, because // we're working on an instance of MyExtensionClass class. That method just redirects to// the lightweight extension method defined in the MyExtensionClass companion object. arg.extensionMethod(new ArgType) 

}
def foo(arg: SealedType): Unit = {

// calling code doesn't care how the compiler implements the extension methods:

arg.extensionMethod(new ArgType) // calls lightweight extension method, no implicit conversion needed

arg.implicitConversionMethod(new ArgType) // implicit conversion to MyExtensionClass required foo2(arg) // implicit conversion to MyExtensionClass required

}

--
Kevin Wright

mail/google talk: kev.lee.wright@gmail.com
wave: kev.lee.wright@googlewave.com
skype: kev.lee.wright
twitter: @thecoda

On Tue, Jun 1, 2010 at 8:52 PM, Kevin Wright <kev.lee.wright@gmail.com> wrote:

Ouch, I can feel your pain here!  We all go through a phase of thinking pimp-my-library / extension methods are wastefully inefficient.  "what, you mean if I chain together 6 such extension methods then it'll be implicitly converted 6 times, but that's crazy!"
This feeling usually passes after a few sharp whacks round the head with a good clue stick, plus a bit of knowledge what's going on under the covers:
1. The efficiency argument is a non-starter.  It's a common misconception that this pattern is inefficient (yup, I fell prey to it as well).

I fell prey to that as well. :/
 

 The truth is that such conversions are optimised out (so no temp object created).  Even if this wasn't the case, temp objects are really really cheap, far more so than common sense would imply.

2. As for the separation issue; if you're "pimping" a 3rd party library then this is unavoidable with *any* approach.  If it's all your own code then any separation is purely self-imposed.  Scala's good to us like that - we can even put multiple classes in the one file. (hey, if it's good enough for the compiler...)
3. Boilerplate reduction doesn't help much either, you're substituting a 1-line method for a 1-line annotation, and introducing new syntax so that you can do so.  Scala really doesn't need new syntax, we're still coming to terms with the alternate uses of underscores!


On 1 June 2010 17:36, Jeremy Bell <bell.jeremy@gmail.com> wrote:

First, some background: 
As you know, in Scala the current idiomatic way of adding a new method to an existing type is to:

1. Create a new class that defines the new method. Usually this class takes the existing type as a default constructor parameter.
2. Write an implicit conversion function to convert the existing type into the new "extension" type.
3. Call the new method on an instance of the existing type. Compiler generates an implicit conversion to an instance of this new type, the new method is called on this temporary instance, and then the temporary instance is discarded.

This works great for the most part, but there are drawbacks:

1. It is cumbersome to write both the new class and the implicit conversion, if just a small number of extension methods are needed. 
2. The implicit conversion and the "wrapper" class are highly related, but often separated in the code.
3. For many use cases, the "boxing" operation performed during an implicit conversion is needlessly inefficient. Every time one of the new methods are called, a new temporary object is constructed, then discarded after the method is called. It's possible that escape analysis can reduce the cost of this operation, but not eliminate it completely.

To address these issues, I propose two new features: lightweight extension methods, and a related proposal: extension classes.
Lightweight extension methods:
A lightweight extension method is simply a method defined in an object which takes at least one parameter: an instance of the original type. In addition, the method is marked with an annotation declaring it an extension method:

package somelibobject ExtensionMethods {

@ExtensionMethod def extensionMethod(self: SealedType, args: ArgType): ReturnType = ......

}

 If the ExtensonMethods object is imported (i.e. "import somelib.ExtensionMethods._"), and the extensionMethod method is called on an instance of SealedType, the following transformation occurs:

import somelib.ExtensionMethods._

var original = new SealedType
// original code original.extensionMethod(new ArgType)
// compiler generated code: extensionMethod(original, new ArgType)// in other words: somelib.ExtensonMethods.extensionMethod(original, new ArgType)

As you can see, no temporary object needs to be generated, as we are simply calling an object method on an instance of the original type.
The benefits are:

1. It is a lot simpler to define a single extension method in an object (possibly along with other extension methods of different types) than to do an implicit conversion to a new type. This is especially helpful for defining internal DSLs.
2. It is more efficient, because no boxing of the original instance is necessary to call the extension method.

The drawback is:

1. If in the future you decide to create a wrapper class after all, with an implicit conversion, and you want to include the extension methods you've already defined, then you need to redefine all those methods in the wrapper class and call them on the original instance.

To address this drawback, I propose the concept of an extension class. An extension class is a short-hand for defining both a wrapper class and the implicit conversion needed. However, in addition the compiler can potentially optimize by generating extension methods when possible:

// Note: MyExtensionClass could potentially mixin traits or inherit another class @ExtensionClass(extendedClass = classOf[SealedType])// default constructor must have at least one argument: the wrapped instance. // if more than one argument, then the first argument is used as the wrapped instance// we call it self here as a convention only class MyExtensionClass(val self: SealedType) { 

private var additionalVar = 3 // can define vars

def extensionMethod(arg: ArgType): Unit = println(arg.toString)def implicitConversionMethod(arg: ArgType): Unit = println(additionalVar.toString + arg.toString)

}

The compiler should at a minimum generate an implicit conversion to this extension class with all the methods defined as usual.
However, in addition, the compiler may be able to determine whether a particular method can be implemented using a lightweight extension method. For example, in the above MyExtensionClass, the extensionMethod method can be defined as both a lightweight extension method and a method of MyExtensionClass (which simply calls the lightweight extension method), because the only field of MyExtensionClass it references is the wrapped SealedType. However, the implicitConversionMethod cannot be made into a lightweight extension method, because it references the extension class's additionalVar field, so an implicit conversion will always be done when the implicitConversionMethod is called"

// compiler generated code: @ExtensionClass(extendedClass = classOf[SealedType]) // annotation is retainedobject MyExtensionClass {

@ExtensionMethod def extensionMethod(self: SealedType, arg: ArgType): Unit = ...// implicitConversionMethod is not defined here

implicit def sealedTypeToMyExtensionClass(self: SealedType) = new MyExtensionClass(self)

}
class MyExtensionClass(val self: SealedType) {

private var additionalVar = 3

def extensionMethod(arg: ArgType): Unit = MyExtensionClass.extensionMethod(self, arg)

def implicitConversionMethod(arg: ArgType): Unit = ...

}

The great thing about this is that the code need not care whether the compiler generates lightweight extension methods or not - it will work the same either way. This means it may be possible to convert existing implicit conversions into an extension class without breaking client code:

import somelib._ import somelib.MyExtensionClass._
// code can use references to MyExtensionClass, and they work like you would expect:def foo1(arg: MyExtensionClass) {

arg.implicitConversionMethod(new ArgType)
// code can still call methods that are converted to lightweight extension methods// but in this case it calls the extensionMethod defined in MyExtensionClass, because // we're working on an instance of MyExtensionClass class. That method just redirects to// the lightweight extension method defined in the MyExtensionClass companion object. arg.extensionMethod(new ArgType) 

}
def foo(arg: SealedType): Unit = {

// calling code doesn't care how the compiler implements the extension methods:

arg.extensionMethod(new ArgType) // calls lightweight extension method, no implicit conversion needed

arg.implicitConversionMethod(new ArgType) // implicit conversion to MyExtensionClass required foo2(arg) // implicit conversion to MyExtensionClass required

}

--
Kevin Wright

mail/google talk: kev.lee.wright@gmail.com
wave: kev.lee.wright@googlewave.com
skype: kev.lee.wright
twitter: @thecoda

--
Viktor Klang
| "A complex system that works is invariably
| found to have evolved from a simple system
| that worked." - John Gall

Akka - the Actor Kernel: Akkasource.org
Twttr: twitter.com/viktorklang

Told you it was common :P

On 1 June 2010 19:55, Viktor Klang <viktor.klang@gmail.com> wrote:

On Tue, Jun 1, 2010 at 8:52 PM, Kevin Wright <kev.lee.wright@gmail.com> wrote:

Ouch, I can feel your pain here!  We all go through a phase of thinking pimp-my-library / extension methods are wastefully inefficient.  "what, you mean if I chain together 6 such extension methods then it'll be implicitly converted 6 times, but that's crazy!"
This feeling usually passes after a few sharp whacks round the head with a good clue stick, plus a bit of knowledge what's going on under the covers:
1. The efficiency argument is a non-starter.  It's a common misconception that this pattern is inefficient (yup, I fell prey to it as well).

I fell prey to that as well. :/
 

 The truth is that such conversions are optimised out (so no temp object created).  Even if this wasn't the case, temp objects are really really cheap, far more so than common sense would imply.

2. As for the separation issue; if you're "pimping" a 3rd party library then this is unavoidable with *any* approach.  If it's all your own code then any separation is purely self-imposed.  Scala's good to us like that - we can even put multiple classes in the one file. (hey, if it's good enough for the compiler...)
3. Boilerplate reduction doesn't help much either, you're substituting a 1-line method for a 1-line annotation, and introducing new syntax so that you can do so.  Scala really doesn't need new syntax, we're still coming to terms with the alternate uses of underscores!


On 1 June 2010 17:36, Jeremy Bell <bell.jeremy@gmail.com> wrote:

First, some background: 
As you know, in Scala the current idiomatic way of adding a new method to an existing type is to:

1. Create a new class that defines the new method. Usually this class takes the existing type as a default constructor parameter.
2. Write an implicit conversion function to convert the existing type into the new "extension" type.
3. Call the new method on an instance of the existing type. Compiler generates an implicit conversion to an instance of this new type, the new method is called on this temporary instance, and then the temporary instance is discarded.

This works great for the most part, but there are drawbacks:

1. It is cumbersome to write both the new class and the implicit conversion, if just a small number of extension methods are needed. 
2. The implicit conversion and the "wrapper" class are highly related, but often separated in the code.
3. For many use cases, the "boxing" operation performed during an implicit conversion is needlessly inefficient. Every time one of the new methods are called, a new temporary object is constructed, then discarded after the method is called. It's possible that escape analysis can reduce the cost of this operation, but not eliminate it completely.

To address these issues, I propose two new features: lightweight extension methods, and a related proposal: extension classes.
Lightweight extension methods:
A lightweight extension method is simply a method defined in an object which takes at least one parameter: an instance of the original type. In addition, the method is marked with an annotation declaring it an extension method:

package somelibobject ExtensionMethods {

@ExtensionMethod def extensionMethod(self: SealedType, args: ArgType): ReturnType = ......

}

 If the ExtensonMethods object is imported (i.e. "import somelib.ExtensionMethods._"), and the extensionMethod method is called on an instance of SealedType, the following transformation occurs:

import somelib.ExtensionMethods._

var original = new SealedType
// original code original.extensionMethod(new ArgType)
// compiler generated code: extensionMethod(original, new ArgType)// in other words: somelib.ExtensonMethods.extensionMethod(original, new ArgType)

As you can see, no temporary object needs to be generated, as we are simply calling an object method on an instance of the original type.
The benefits are:

1. It is a lot simpler to define a single extension method in an object (possibly along with other extension methods of different types) than to do an implicit conversion to a new type. This is especially helpful for defining internal DSLs.
2. It is more efficient, because no boxing of the original instance is necessary to call the extension method.

The drawback is:

1. If in the future you decide to create a wrapper class after all, with an implicit conversion, and you want to include the extension methods you've already defined, then you need to redefine all those methods in the wrapper class and call them on the original instance.

To address this drawback, I propose the concept of an extension class. An extension class is a short-hand for defining both a wrapper class and the implicit conversion needed. However, in addition the compiler can potentially optimize by generating extension methods when possible:

// Note: MyExtensionClass could potentially mixin traits or inherit another class @ExtensionClass(extendedClass = classOf[SealedType])// default constructor must have at least one argument: the wrapped instance. // if more than one argument, then the first argument is used as the wrapped instance// we call it self here as a convention only class MyExtensionClass(val self: SealedType) { 

private var additionalVar = 3 // can define vars

def extensionMethod(arg: ArgType): Unit = println(arg.toString)def implicitConversionMethod(arg: ArgType): Unit = println(additionalVar.toString + arg.toString)

}

The compiler should at a minimum generate an implicit conversion to this extension class with all the methods defined as usual.
However, in addition, the compiler may be able to determine whether a particular method can be implemented using a lightweight extension method. For example, in the above MyExtensionClass, the extensionMethod method can be defined as both a lightweight extension method and a method of MyExtensionClass (which simply calls the lightweight extension method), because the only field of MyExtensionClass it references is the wrapped SealedType. However, the implicitConversionMethod cannot be made into a lightweight extension method, because it references the extension class's additionalVar field, so an implicit conversion will always be done when the implicitConversionMethod is called"

// compiler generated code: @ExtensionClass(extendedClass = classOf[SealedType]) // annotation is retainedobject MyExtensionClass {

@ExtensionMethod def extensionMethod(self: SealedType, arg: ArgType): Unit = ...// implicitConversionMethod is not defined here

implicit def sealedTypeToMyExtensionClass(self: SealedType) = new MyExtensionClass(self)

}
class MyExtensionClass(val self: SealedType) {

private var additionalVar = 3

def extensionMethod(arg: ArgType): Unit = MyExtensionClass.extensionMethod(self, arg)

def implicitConversionMethod(arg: ArgType): Unit = ...

}

The great thing about this is that the code need not care whether the compiler generates lightweight extension methods or not - it will work the same either way. This means it may be possible to convert existing implicit conversions into an extension class without breaking client code:

import somelib._ import somelib.MyExtensionClass._
// code can use references to MyExtensionClass, and they work like you would expect:def foo1(arg: MyExtensionClass) {

arg.implicitConversionMethod(new ArgType)
// code can still call methods that are converted to lightweight extension methods// but in this case it calls the extensionMethod defined in MyExtensionClass, because // we're working on an instance of MyExtensionClass class. That method just redirects to// the lightweight extension method defined in the MyExtensionClass companion object. arg.extensionMethod(new ArgType) 

}
def foo(arg: SealedType): Unit = {

// calling code doesn't care how the compiler implements the extension methods:

arg.extensionMethod(new ArgType) // calls lightweight extension method, no implicit conversion needed

arg.implicitConversionMethod(new ArgType) // implicit conversion to MyExtensionClass required foo2(arg) // implicit conversion to MyExtensionClass required

}

--
Kevin Wright

mail/google talk: kev.lee.wright@gmail.com
wave: kev.lee.wright@googlewave.com
skype: kev.lee.wright
twitter: @thecoda

--
Viktor Klang
| "A complex system that works is invariably
| found to have evolved from a simple system
| that worked." - John Gall

Akka - the Actor Kernel: Akkasource.org
Twttr: twitter.com/viktorklang

--
Kevin Wright

mail/google talk: kev.lee.wright@gmail.com
wave: kev.lee.wright@googlewave.com
skype: kev.lee.wright
twitter: @thecoda

Can someone please post some objective evidence that the allocation is consistently eliminated for common patterns?   I know I'm being lazy here because I haven't bothered benchmarking this particular JVM-based optimization, but I've found that more advanced ones tend to have enough caveats to make ones head spin, and JVM tuning is not what I would consider a fun use of my time.   People keep on claiming they've tested it and they've benchmarked it.  I'd like to see the benchmarks and results so I can run them myself.

On Tue, Jun 1, 2010 at 2:55 PM, Viktor Klang <viktor.klang@gmail.com> wrote:

On Tue, Jun 1, 2010 at 8:52 PM, Kevin Wright <kev.lee.wright@gmail.com> wrote:

Ouch, I can feel your pain here!  We all go through a phase of thinking pimp-my-library / extension methods are wastefully inefficient.  "what, you mean if I chain together 6 such extension methods then it'll be implicitly converted 6 times, but that's crazy!"
This feeling usually passes after a few sharp whacks round the head with a good clue stick, plus a bit of knowledge what's going on under the covers:
1. The efficiency argument is a non-starter.  It's a common misconception that this pattern is inefficient (yup, I fell prey to it as well).

I fell prey to that as well. :/
 

 The truth is that such conversions are optimised out (so no temp object created).  Even if this wasn't the case, temp objects are really really cheap, far more so than common sense would imply.

2. As for the separation issue; if you're "pimping" a 3rd party library then this is unavoidable with *any* approach.  If it's all your own code then any separation is purely self-imposed.  Scala's good to us like that - we can even put multiple classes in the one file. (hey, if it's good enough for the compiler...)
3. Boilerplate reduction doesn't help much either, you're substituting a 1-line method for a 1-line annotation, and introducing new syntax so that you can do so.  Scala really doesn't need new syntax, we're still coming to terms with the alternate uses of underscores!


On 1 June 2010 17:36, Jeremy Bell <bell.jeremy@gmail.com> wrote:

First, some background: 
As you know, in Scala the current idiomatic way of adding a new method to an existing type is to:

1. Create a new class that defines the new method. Usually this class takes the existing type as a default constructor parameter.
2. Write an implicit conversion function to convert the existing type into the new "extension" type.
3. Call the new method on an instance of the existing type. Compiler generates an implicit conversion to an instance of this new type, the new method is called on this temporary instance, and then the temporary instance is discarded.

This works great for the most part, but there are drawbacks:

1. It is cumbersome to write both the new class and the implicit conversion, if just a small number of extension methods are needed. 
2. The implicit conversion and the "wrapper" class are highly related, but often separated in the code.
3. For many use cases, the "boxing" operation performed during an implicit conversion is needlessly inefficient. Every time one of the new methods are called, a new temporary object is constructed, then discarded after the method is called. It's possible that escape analysis can reduce the cost of this operation, but not eliminate it completely.

To address these issues, I propose two new features: lightweight extension methods, and a related proposal: extension classes.
Lightweight extension methods:
A lightweight extension method is simply a method defined in an object which takes at least one parameter: an instance of the original type. In addition, the method is marked with an annotation declaring it an extension method:

package somelib object ExtensionMethods {

@ExtensionMethod def extensionMethod(self: SealedType, args: ArgType): ReturnType = ... ...

}

 If the ExtensonMethods object is imported (i.e. "import somelib.ExtensionMethods._"), and the extensionMethod method is called on an instance of SealedType, the following transformation occurs:

import somelib.ExtensionMethods._

var original = new SealedType
// original code original.extensionMethod(new ArgType)
// compiler generated code: extensionMethod(original, new ArgType) // in other words: somelib.ExtensonMethods.extensionMethod(original, new ArgType)

As you can see, no temporary object needs to be generated, as we are simply calling an object method on an instance of the original type.
The benefits are:

1. It is a lot simpler to define a single extension method in an object (possibly along with other extension methods of different types) than to do an implicit conversion to a new type. This is especially helpful for defining internal DSLs.
2. It is more efficient, because no boxing of the original instance is necessary to call the extension method.

The drawback is:

1. If in the future you decide to create a wrapper class after all, with an implicit conversion, and you want to include the extension methods you've already defined, then you need to redefine all those methods in the wrapper class and call them on the original instance.

To address this drawback, I propose the concept of an extension class. An extension class is a short-hand for defining both a wrapper class and the implicit conversion needed. However, in addition the compiler can potentially optimize by generating extension methods when possible:

// Note: MyExtensionClass could potentially mixin traits or inherit another class @ExtensionClass(extendedClass = classOf[SealedType]) // default constructor must have at least one argument: the wrapped instance. // if more than one argument, then the first argument is used as the wrapped instance // we call it self here as a convention only class MyExtensionClass(val self: SealedType) { 

private var additionalVar = 3 // can define vars

def extensionMethod(arg: ArgType): Unit = println(arg.toString) def implicitConversionMethod(arg: ArgType): Unit = println(additionalVar.toString + arg.toString)

}

The compiler should at a minimum generate an implicit conversion to this extension class with all the methods defined as usual.
However, in addition, the compiler may be able to determine whether a particular method can be implemented using a lightweight extension method. For example, in the above MyExtensionClass, the extensionMethod method can be defined as both a lightweight extension method and a method of MyExtensionClass (which simply calls the lightweight extension method), because the only field of MyExtensionClass it references is the wrapped SealedType. However, the implicitConversionMethod cannot be made into a lightweight extension method, because it references the extension class's additionalVar field, so an implicit conversion will always be done when the implicitConversionMethod is called"

// compiler generated code: @ExtensionClass(extendedClass = classOf[SealedType]) // annotation is retained object MyExtensionClass {

@ExtensionMethod def extensionMethod(self: SealedType, arg: ArgType): Unit = ... // implicitConversionMethod is not defined here

implicit def sealedTypeToMyExtensionClass(self: SealedType) = new MyExtensionClass(self)

}
class MyExtensionClass(val self: SealedType) {

private var additionalVar = 3

def extensionMethod(arg: ArgType): Unit = MyExtensionClass.extensionMethod(self, arg)

def implicitConversionMethod(arg: ArgType): Unit = ...

}

The great thing about this is that the code need not care whether the compiler generates lightweight extension methods or not - it will work the same either way. This means it may be possible to convert existing implicit conversions into an extension class without breaking client code:

import somelib._ import somelib.MyExtensionClass._
// code can use references to MyExtensionClass, and they work like you would expect: def foo1(arg: MyExtensionClass) {

arg.implicitConversionMethod(new ArgType)
// code can still call methods that are converted to lightweight extension methods // but in this case it calls the extensionMethod defined in MyExtensionClass, because // we're working on an instance of MyExtensionClass class. That method just redirects to // the lightweight extension method defined in the MyExtensionClass companion object. arg.extensionMethod(new ArgType) 

}
def foo(arg: SealedType): Unit = {

// calling code doesn't care how the compiler implements the extension methods:

arg.extensionMethod(new ArgType) // calls lightweight extension method, no implicit conversion needed

arg.implicitConversionMethod(new ArgType) // implicit conversion to MyExtensionClass required foo2(arg) // implicit conversion to MyExtensionClass required

}

--
Kevin Wright

mail/google talk: kev.lee.wright@gmail.com
wave: kev.lee.wright@googlewave.com
skype: kev.lee.wright
twitter: @thecoda

--
Viktor Klang
| "A complex system that works is invariably
| found to have evolved from a simple system
| that worked." - John Gall

Akka - the Actor Kernel: Akkasource.org
Twttr: twitter.com/viktorklang

--
http://erikengbrecht.blogspot.com/

What we DO need though is a way to add state to an existing object.
The natural way to do this is via traits, but they're only applicable when an object is created.
Rest assured, I'm continuing to work on the problem.  I think I'm onto a new approach that could fix the outstanding problems in my autoproxy plugin :)

On 1 June 2010 20:09, Kevin Wright <kev.lee.wright@gmail.com> wrote:

Told you it was common :P

On 1 June 2010 19:55, Viktor Klang <viktor.klang@gmail.com> wrote:

On Tue, Jun 1, 2010 at 8:52 PM, Kevin Wright <kev.lee.wright@gmail.com> wrote:

Ouch, I can feel your pain here!  We all go through a phase of thinking pimp-my-library / extension methods are wastefully inefficient.  "what, you mean if I chain together 6 such extension methods then it'll be implicitly converted 6 times, but that's crazy!"
This feeling usually passes after a few sharp whacks round the head with a good clue stick, plus a bit of knowledge what's going on under the covers:
1. The efficiency argument is a non-starter.  It's a common misconception that this pattern is inefficient (yup, I fell prey to it as well).

I fell prey to that as well. :/
 

 The truth is that such conversions are optimised out (so no temp object created).  Even if this wasn't the case, temp objects are really really cheap, far more so than common sense would imply.

2. As for the separation issue; if you're "pimping" a 3rd party library then this is unavoidable with *any* approach.  If it's all your own code then any separation is purely self-imposed.  Scala's good to us like that - we can even put multiple classes in the one file. (hey, if it's good enough for the compiler...)
3. Boilerplate reduction doesn't help much either, you're substituting a 1-line method for a 1-line annotation, and introducing new syntax so that you can do so.  Scala really doesn't need new syntax, we're still coming to terms with the alternate uses of underscores!


On 1 June 2010 17:36, Jeremy Bell <bell.jeremy@gmail.com> wrote:

First, some background: 
As you know, in Scala the current idiomatic way of adding a new method to an existing type is to:

1. Create a new class that defines the new method. Usually this class takes the existing type as a default constructor parameter.
2. Write an implicit conversion function to convert the existing type into the new "extension" type.
3. Call the new method on an instance of the existing type. Compiler generates an implicit conversion to an instance of this new type, the new method is called on this temporary instance, and then the temporary instance is discarded.

This works great for the most part, but there are drawbacks:

1. It is cumbersome to write both the new class and the implicit conversion, if just a small number of extension methods are needed. 
2. The implicit conversion and the "wrapper" class are highly related, but often separated in the code.
3. For many use cases, the "boxing" operation performed during an implicit conversion is needlessly inefficient. Every time one of the new methods are called, a new temporary object is constructed, then discarded after the method is called. It's possible that escape analysis can reduce the cost of this operation, but not eliminate it completely.

To address these issues, I propose two new features: lightweight extension methods, and a related proposal: extension classes.
Lightweight extension methods:
A lightweight extension method is simply a method defined in an object which takes at least one parameter: an instance of the original type. In addition, the method is marked with an annotation declaring it an extension method:

package somelibobject ExtensionMethods {

@ExtensionMethod def extensionMethod(self: SealedType, args: ArgType): ReturnType = ......

}

 If the ExtensonMethods object is imported (i.e. "import somelib.ExtensionMethods._"), and the extensionMethod method is called on an instance of SealedType, the following transformation occurs:

import somelib.ExtensionMethods._

var original = new SealedType
// original code original.extensionMethod(new ArgType)
// compiler generated code: extensionMethod(original, new ArgType)// in other words: somelib.ExtensonMethods.extensionMethod(original, new ArgType)

As you can see, no temporary object needs to be generated, as we are simply calling an object method on an instance of the original type.
The benefits are:

1. It is a lot simpler to define a single extension method in an object (possibly along with other extension methods of different types) than to do an implicit conversion to a new type. This is especially helpful for defining internal DSLs.
2. It is more efficient, because no boxing of the original instance is necessary to call the extension method.

The drawback is:

1. If in the future you decide to create a wrapper class after all, with an implicit conversion, and you want to include the extension methods you've already defined, then you need to redefine all those methods in the wrapper class and call them on the original instance.

To address this drawback, I propose the concept of an extension class. An extension class is a short-hand for defining both a wrapper class and the implicit conversion needed. However, in addition the compiler can potentially optimize by generating extension methods when possible:

// Note: MyExtensionClass could potentially mixin traits or inherit another class @ExtensionClass(extendedClass = classOf[SealedType])// default constructor must have at least one argument: the wrapped instance. // if more than one argument, then the first argument is used as the wrapped instance// we call it self here as a convention only class MyExtensionClass(val self: SealedType) { 

private var additionalVar = 3 // can define vars

def extensionMethod(arg: ArgType): Unit = println(arg.toString)def implicitConversionMethod(arg: ArgType): Unit = println(additionalVar.toString + arg.toString)

}

The compiler should at a minimum generate an implicit conversion to this extension class with all the methods defined as usual.
However, in addition, the compiler may be able to determine whether a particular method can be implemented using a lightweight extension method. For example, in the above MyExtensionClass, the extensionMethod method can be defined as both a lightweight extension method and a method of MyExtensionClass (which simply calls the lightweight extension method), because the only field of MyExtensionClass it references is the wrapped SealedType. However, the implicitConversionMethod cannot be made into a lightweight extension method, because it references the extension class's additionalVar field, so an implicit conversion will always be done when the implicitConversionMethod is called"

// compiler generated code: @ExtensionClass(extendedClass = classOf[SealedType]) // annotation is retainedobject MyExtensionClass {

@ExtensionMethod def extensionMethod(self: SealedType, arg: ArgType): Unit = ...// implicitConversionMethod is not defined here

implicit def sealedTypeToMyExtensionClass(self: SealedType) = new MyExtensionClass(self)

}
class MyExtensionClass(val self: SealedType) {

private var additionalVar = 3

def extensionMethod(arg: ArgType): Unit = MyExtensionClass.extensionMethod(self, arg)

def implicitConversionMethod(arg: ArgType): Unit = ...

}

The great thing about this is that the code need not care whether the compiler generates lightweight extension methods or not - it will work the same either way. This means it may be possible to convert existing implicit conversions into an extension class without breaking client code:

import somelib._ import somelib.MyExtensionClass._
// code can use references to MyExtensionClass, and they work like you would expect:def foo1(arg: MyExtensionClass) {

arg.implicitConversionMethod(new ArgType)
// code can still call methods that are converted to lightweight extension methods// but in this case it calls the extensionMethod defined in MyExtensionClass, because // we're working on an instance of MyExtensionClass class. That method just redirects to// the lightweight extension method defined in the MyExtensionClass companion object. arg.extensionMethod(new ArgType) 

}
def foo(arg: SealedType): Unit = {

// calling code doesn't care how the compiler implements the extension methods:

arg.extensionMethod(new ArgType) // calls lightweight extension method, no implicit conversion needed

arg.implicitConversionMethod(new ArgType) // implicit conversion to MyExtensionClass required foo2(arg) // implicit conversion to MyExtensionClass required

}

--
Kevin Wright

mail/google talk: kev.lee.wright@gmail.com
wave: kev.lee.wright@googlewave.com
skype: kev.lee.wright
twitter: @thecoda

--
Viktor Klang
| "A complex system that works is invariably
| found to have evolved from a simple system
| that worked." - John Gall

Akka - the Actor Kernel: Akkasource.org
Twttr: twitter.com/viktorklang

--
Kevin Wright

mail/google talk: kev.lee.wright@gmail.com
wave: kev.lee.wright@googlewave.com
skype: kev.lee.wright
twitter: @thecoda

--
Kevin Wright

mail/google talk: kev.lee.wright@gmail.com
wave: kev.lee.wright@googlewave.com
skype: kev.lee.wright
twitter: @thecoda

Regarding the reduction in boilerplate: I meant to say that the reduction is most prevalent when no wrapper class is needed. Instead of writing a class, with a method or two, and an object, with an implicit conversion, you only need to write the methods with the annotation:
instead of:

object ExtensionClass {

implicit def sealedClassToExtensionClass(self: SealedType) : ExtensionClass = {

new ExtensionClass(self)

}

}
class ExtensionClass(private val self: SealedType) {

def extensionMethod(arg: ArgType)

}

You would instead only need the following:

object ExtensionClass {

@ExtensionMethod def extensionMethod(self: SealedType, arg: ArgType)

}

The separation I'm referring to is the separation of the wrapper class (ExtensionClass in this case) from the implicit conversion function, which here is defined in the ExtensionClass companion object, but in a lot of scala code I've seen so far, the common practice is to define all implicit conversions in one object, rather than in separate companion objects, so the implicit conversion functions may be in a completely different file from the wrapper class.
Also, assuming that extension methods are implemented, the extension classes proposal is a way of refactoring existing code to use lightweight extension methods while retaining binary compatibility, and without a lot of boilerplate wrapper methods. Plus, there is no need to define the implicit conversions to and from the wrapper type.
As for new syntax, I'm really just modifying the lookup behavior when calling a function on an object. Sure, the compiler is transforming a method call on an instance to a "static" method call that takes the instance as a parameter, but implicit conversions are already doing a similar transformation. In fact one could argue that is conceptually a bigger transformation. 
Regarding performance: by "optimized out" do you mean by the compiler or by the JIT? If you mean that the compiler already does the kind of transformations I'm describing, where possible (essentially generating "static" methods and passing the original instance to these methods instead of actually instantiating a new wrapper class), I would believe you. Cool beans! You can ignore the rest of this email!
However, if you mean that the JIT is supposed to do it (i.e. with escape analysis and runtime-inlining of methods), then I can't help but be skeptical, and also I need to point out that sometimes scala code will be running on mobile or special purpose JVM variants (robotics, embedded hardware, cell phones, etc..) where the hardware may not even be capable of JIT, and where even short term allocations are more expensive than you would think (non-generational garbage collectors and so forth). 
Also, even though the allocation would be optimized out by escape analysis, the JIT compiler would also need to inline the constructor, and then optimize the constructor completely out to be equivalent to calling an extension method and passing the original object. In other words, given this code:
// original codeobject ImplicitConversions {  implicit def myTypeToOriginalType(arg: OriginalType) = new MyType(arg)}
class MyType(val self: OriginalType) {   var x: Int = 0  def extensionMethod(arg: ArgType) // extensionMethod references only self, not x  def implicitMethod(arg: ArgType) // references self and x}
def foo = {  var orig = new OriginalType    // original code  // var orig.extensionMethod(new ArgType)
  // what the JIT sees (if I'm correct about the compiler not optimizing out the allocation ahead of time):   var $temporary = new MyType(orig)  $temporary.extensionMethod(new ArgType)}

The JIT would have to convert the above code to something along the lines of this (if extensionMethod is not inlined as well):
object $generated_object {  def extensionMethod(self: OriginalType, arg: ArgType)}
def foo = {    var orig = new OriginalType    $generated_object.extensionMethod(orig, new ArgType)
}
So, in other words, the JIT has to first allocate the MyType on the stack (i.e. MyType's self field becomes a local variable), inlining the MyType constructor in the process. Then, it needs to either generate a static version of extensionMethod that takes an OriginalType as an argument and pass the self local variable to it, or else inline extensionMethod as well, using the local variable self. Then it needs to determine that self is never anything but a copy of orig, so orig can be used instead of self (either passed into a static extensionMethod or used in the inlined extensionMethod. Then it needs to determine that the x field of MyType is never used, and can be eliminated from the inlined constructor. Is the JIT going to be this aggressive in optimizing code?
And, keep in mind that any time you rely on the JIT to inline code for you, that after a threshold the JIT will stop inlining methods in the interests of reducing code size so that it all fits in the instruction cache. And, because implicit conversions are so prevalent in scala code, I imagine that threshold would be reached rather quickly.

On Tue, Jun 1, 2010 at 2:55 PM, Viktor Klang <viktor.klang@gmail.com> wrote:

On Tue, Jun 1, 2010 at 8:52 PM, Kevin Wright <kev.lee.wright@gmail.com> wrote:

Ouch, I can feel your pain here!  We all go through a phase of thinking pimp-my-library / extension methods are wastefully inefficient.  "what, you mean if I chain together 6 such extension methods then it'll be implicitly converted 6 times, but that's crazy!"
This feeling usually passes after a few sharp whacks round the head with a good clue stick, plus a bit of knowledge what's going on under the covers:
1. The efficiency argument is a non-starter.  It's a common misconception that this pattern is inefficient (yup, I fell prey to it as well).

I fell prey to that as well. :/
 

 The truth is that such conversions are optimised out (so no temp object created).  Even if this wasn't the case, temp objects are really really cheap, far more so than common sense would imply.

2. As for the separation issue; if you're "pimping" a 3rd party library then this is unavoidable with *any* approach.  If it's all your own code then any separation is purely self-imposed.  Scala's good to us like that - we can even put multiple classes in the one file. (hey, if it's good enough for the compiler...)
3. Boilerplate reduction doesn't help much either, you're substituting a 1-line method for a 1-line annotation, and introducing new syntax so that you can do so.  Scala really doesn't need new syntax, we're still coming to terms with the alternate uses of underscores!


On 1 June 2010 17:36, Jeremy Bell <bell.jeremy@gmail.com> wrote:

First, some background: 
As you know, in Scala the current idiomatic way of adding a new method to an existing type is to:

1. Create a new class that defines the new method. Usually this class takes the existing type as a default constructor parameter.
2. Write an implicit conversion function to convert the existing type into the new "extension" type.
3. Call the new method on an instance of the existing type. Compiler generates an implicit conversion to an instance of this new type, the new method is called on this temporary instance, and then the temporary instance is discarded.

This works great for the most part, but there are drawbacks:

1. It is cumbersome to write both the new class and the implicit conversion, if just a small number of extension methods are needed. 
2. The implicit conversion and the "wrapper" class are highly related, but often separated in the code.
3. For many use cases, the "boxing" operation performed during an implicit conversion is needlessly inefficient. Every time one of the new methods are called, a new temporary object is constructed, then discarded after the method is called. It's possible that escape analysis can reduce the cost of this operation, but not eliminate it completely.

To address these issues, I propose two new features: lightweight extension methods, and a related proposal: extension classes.
Lightweight extension methods:
A lightweight extension method is simply a method defined in an object which takes at least one parameter: an instance of the original type. In addition, the method is marked with an annotation declaring it an extension method:

package somelibobject ExtensionMethods {

@ExtensionMethod def extensionMethod(self: SealedType, args: ArgType): ReturnType = ......

}

 If the ExtensonMethods object is imported (i.e. "import somelib.ExtensionMethods._"), and the extensionMethod method is called on an instance of SealedType, the following transformation occurs:

import somelib.ExtensionMethods._

var original = new SealedType
// original code original.extensionMethod(new ArgType)
// compiler generated code: extensionMethod(original, new ArgType)// in other words: somelib.ExtensonMethods.extensionMethod(original, new ArgType)

As you can see, no temporary object needs to be generated, as we are simply calling an object method on an instance of the original type.
The benefits are:

1. It is a lot simpler to define a single extension method in an object (possibly along with other extension methods of different types) than to do an implicit conversion to a new type. This is especially helpful for defining internal DSLs.
2. It is more efficient, because no boxing of the original instance is necessary to call the extension method.

The drawback is:

1. If in the future you decide to create a wrapper class after all, with an implicit conversion, and you want to include the extension methods you've already defined, then you need to redefine all those methods in the wrapper class and call them on the original instance.

To address this drawback, I propose the concept of an extension class. An extension class is a short-hand for defining both a wrapper class and the implicit conversion needed. However, in addition the compiler can potentially optimize by generating extension methods when possible:

// Note: MyExtensionClass could potentially mixin traits or inherit another class @ExtensionClass(extendedClass = classOf[SealedType])// default constructor must have at least one argument: the wrapped instance. // if more than one argument, then the first argument is used as the wrapped instance// we call it self here as a convention only class MyExtensionClass(val self: SealedType) { 

private var additionalVar = 3 // can define vars

def extensionMethod(arg: ArgType): Unit = println(arg.toString)def implicitConversionMethod(arg: ArgType): Unit = println(additionalVar.toString + arg.toString)

}

The compiler should at a minimum generate an implicit conversion to this extension class with all the methods defined as usual.
However, in addition, the compiler may be able to determine whether a particular method can be implemented using a lightweight extension method. For example, in the above MyExtensionClass, the extensionMethod method can be defined as both a lightweight extension method and a method of MyExtensionClass (which simply calls the lightweight extension method), because the only field of MyExtensionClass it references is the wrapped SealedType. However, the implicitConversionMethod cannot be made into a lightweight extension method, because it references the extension class's additionalVar field, so an implicit conversion will always be done when the implicitConversionMethod is called"

// compiler generated code: @ExtensionClass(extendedClass = classOf[SealedType]) // annotation is retainedobject MyExtensionClass {

@ExtensionMethod def extensionMethod(self: SealedType, arg: ArgType): Unit = ...// implicitConversionMethod is not defined here

implicit def sealedTypeToMyExtensionClass(self: SealedType) = new MyExtensionClass(self)

}
class MyExtensionClass(val self: SealedType) {

private var additionalVar = 3

def extensionMethod(arg: ArgType): Unit = MyExtensionClass.extensionMethod(self, arg)

def implicitConversionMethod(arg: ArgType): Unit = ...

}

The great thing about this is that the code need not care whether the compiler generates lightweight extension methods or not - it will work the same either way. This means it may be possible to convert existing implicit conversions into an extension class without breaking client code:

import somelib._ import somelib.MyExtensionClass._
// code can use references to MyExtensionClass, and they work like you would expect:def foo1(arg: MyExtensionClass) {

arg.implicitConversionMethod(new ArgType)
// code can still call methods that are converted to lightweight extension methods// but in this case it calls the extensionMethod defined in MyExtensionClass, because // we're working on an instance of MyExtensionClass class. That method just redirects to// the lightweight extension method defined in the MyExtensionClass companion object. arg.extensionMethod(new ArgType) 

}
def foo(arg: SealedType): Unit = {

// calling code doesn't care how the compiler implements the extension methods:

arg.extensionMethod(new ArgType) // calls lightweight extension method, no implicit conversion needed

arg.implicitConversionMethod(new ArgType) // implicit conversion to MyExtensionClass required foo2(arg) // implicit conversion to MyExtensionClass required

}

--
Kevin Wright

mail/google talk: kev.lee.wright@gmail.com
wave: kev.lee.wright@googlewave.com
skype: kev.lee.wright
twitter: @thecoda

--
Viktor Klang
| "A complex system that works is invariably
| found to have evolved from a simple system
| that worked." - John Gall

Akka - the Actor Kernel: Akkasource.org
Twttr: twitter.com/viktorklang

I think I can confirm that the compiler does not actually eliminate the object allocations, so we are in fact relying on the VM to do this for us. For example, here is what the compiler generates for foo and foo2 methods (this is the bytecode decompiled to java):
  public void foo1(WrapperClass arg)  {    Predef..MODULE$.println(arg.extensionMethod());    Predef..MODULE$.println(BoxesRunTime.boxToInteger(arg.implicitMethod()).toString());   }
  public void foo2(SealedClass arg) {    ExtensionClass..MODULE$.sealedClassToWrapperClass(arg).extensionMethod();    ExtensionClass..MODULE$.sealedClassToWrapperClass(arg).implicitMethod();     foo1(ExtensionClass..MODULE$.sealedClassToWrapperClass(arg));  }
Here is ExtensionClass$ decompiled (note, the implicit conversion method is not even compiled down to a static method, despite not referencing any variables in the object):
public final class ExtensionClass$   implements ScalaObject{  public static final  MODULE$;
  static  {    new ();  }
  public WrapperClass sealedClassToWrapperClass(SealedClass self)   {    return new WrapperClass(self);  }
  private ExtensionClass$()  {    MODULE$ = this;  }}
I take that back, it is compiled down to a static method in a different class, but that method just calls the non-static method on the singleton object (shouldn't this be the other way around, if the method doesn't access any data in the singleton?):
public final class ExtensionClass{  public static final WrapperClass sealedClassToWrapperClass(SealedClass paramSealedClass)  {    return ExtensionClass..MODULE$.sealedClassToWrapperClass(paramSealedClass);   }}
Here is WrapperClass decompiled. Note that even though the local field is private, the compiler still generates and uses a private local() method to retrieve the value of local in implicitMethod, rather than simply referencing the field directly. 
public class WrapperClass  implements ScalaObject{  private final SealedClass self;  private int local;
  public SealedClass self()   {    return this.self; }   private int local() { return this.local; }   private void local_$eq(int paramInt) { this.local = paramInt; }   public String extensionMethod() { return self().toString(); }    public int implicitMethod() { return local(); }

  public WrapperClass(SealedClass self)  {    this.local = 2;  }}

Here is the original scala code:sealed class SealedClass {}
object ExtensionClass { implicit def sealedClassToWrapperClass(self: SealedClass) : WrapperClass = { new WrapperClass(self) }}
class WrapperClass(val self: SealedClass) { private var local = 2 def extensionMethod = self.toString def implicitMethod = local}
import ExtensionClass._
class SandBox { def foo1(arg: WrapperClass): Unit = { println(arg.extensionMethod) println(arg.implicitMethod.toString) } def foo2(arg: SealedClass): Unit = { arg.extensionMethod arg.implicitMethod foo1(arg) }}
So, the next step is to profile this code on different VMs (with/without escape analysis, etc..) to see if the JIT steps in to save the day, though already I can tell you that on VMs like android's dalvik, you will be getting all of those temporary allocations without any optimization at all (at least for another year or so until android 2.2+ (which added method-level JIT) gains a significant installed base). 

I did a quick microbenchmark on JRE 6 (update 18) and JRE 7 (milestone 5, build b76) on the following code:
// SandBox.scalapackage org.jeb
sealed class SealedClass { }
object ExtensionClass { implicit def sealedClassToWrapperClass(self: SealedClass) : WrapperClass = { new WrapperClass(self) } def extensionMethod(self: SealedClass) = 1 }
class WrapperClass(val self: SealedClass) { def extensionMethod = 1}
import ExtensionClass._
object SandBox { def test1(arg: SealedClass): Unit = { arg.extensionMethod } def test2(arg: SealedClass): Unit = { ExtensionClass.extensionMethod(arg) }}
// Main.javapublic class Main {
static int extensionMethod(SealedClass arg) { return 1; } static void test3(SealedClass arg) { extensionMethod(arg); }
public static void main(String[] args) { SealedClass arg = new SealedClass(); // JIT/hotspot warmup for(int i = 0; i < 3000; ++i) System.currentTimeMillis(); //for(int i = 0; i < 3000; ++i) SandBox.test1(arg); for(int i = 0; i < 3000; ++i) SandBox.test2(arg); //for(int i = 0; i < 3000; ++i) test3(arg); long startTime = System.currentTimeMillis(); long delta = 0; long iters = 0; while(delta < 5000) { //SandBox.test1(arg); SandBox.test2(arg); //test3(arg); iters++; delta = System.currentTimeMillis() - startTime; } System.out.println("delta: " + delta); System.out.println("iters: " + iters); System.out.println("iters per second: " + ((double)iters / ((double)delta / 1000.0))); }
}
Testing method:

• One test was run at a time. The others were commented out.
• The first run after a compile was thrown out
• Five runs were performed for each test after the first run post-compile. The average iters per second over the 5 runs was used for the comparison.

Here I am comparing SandBox.test1, SandBox.test2, and test3, a static method defined in Main, which calls another static method. Here are my findings:

1. SandBox.test2 performed an average of 24.67% faster than SandBox.test1
2. Main.test3 performed roughly the same as SandBox.test2, plus or minus about 2%.
3. JRE 7 (Milestone 6, build b76) did not significantly affect the performance of any of these tests. In fact the tests performed roughly 1-2% slower on average when run in JRE 7 vs JRE 6 (update 18).

I thought perhaps the fact that extensionMethod always returned 1 was having an affect on the tests, so as a variation of the test, I added a long field to SealedClass, and made extensionMethod increment the field by 1. With this change, the difference between test2 and test1 was reduced to only a 18% improvement, vs. a 24.67% improvement. JRE 7 caused a less than 1% performance decrease over the same tests in jre 6.
I'm not sure how conclusive these tests are, being a microbenchmark, but it appears that there is reasonably strong evidence that even with the latest and greatest VM, lightweight extension methods could potentially provide a significant performance boost over the equivalent implicit conversion.

have you tried enabling EscapeAnalysis (JRE6u14+)? it used to be off by default and you had to pass
-XX:+DoEscapeAnalysis
to enable it
theoretically, with EA and these scenarios, the overhead would be a lot smaller (if not zero)

On Wed, Jun 2, 2010 at 12:50 PM, Jeremy Bell <bell.jeremy@gmail.com> wrote:

I did a quick microbenchmark on JRE 6 (update 18) and JRE 7 (milestone 5, build b76) on the following code:
// SandBox.scalapackage org.jeb
sealed class SealedClass { }
object ExtensionClass { implicit def sealedClassToWrapperClass(self: SealedClass) : WrapperClass = { new WrapperClass(self) } def extensionMethod(self: SealedClass) = 1 }
class WrapperClass(val self: SealedClass) { def extensionMethod = 1}
import ExtensionClass._
object SandBox { def test1(arg: SealedClass): Unit = { arg.extensionMethod } def test2(arg: SealedClass): Unit = { ExtensionClass.extensionMethod(arg) }}
// Main.javapublic class Main {
static int extensionMethod(SealedClass arg) { return 1; } static void test3(SealedClass arg) { extensionMethod(arg); }
public static void main(String[] args) { SealedClass arg = new SealedClass(); // JIT/hotspot warmup for(int i = 0; i < 3000; ++i) System.currentTimeMillis(); //for(int i = 0; i < 3000; ++i) SandBox.test1(arg); for(int i = 0; i < 3000; ++i) SandBox.test2(arg); //for(int i = 0; i < 3000; ++i) test3(arg); long startTime = System.currentTimeMillis(); long delta = 0; long iters = 0; while(delta < 5000) { //SandBox.test1(arg); SandBox.test2(arg); //test3(arg); iters++; delta = System.currentTimeMillis() - startTime; } System.out.println("delta: " + delta); System.out.println("iters: " + iters); System.out.println("iters per second: " + ((double)iters / ((double)delta / 1000.0))); }
}
Testing method:

• One test was run at a time. The others were commented out.
• The first run after a compile was thrown out
• Five runs were performed for each test after the first run post-compile. The average iters per second over the 5 runs was used for the comparison.

Here I am comparing SandBox.test1, SandBox.test2, and test3, a static method defined in Main, which calls another static method. Here are my findings:

1. SandBox.test2 performed an average of 24.67% faster than SandBox.test1
2. Main.test3 performed roughly the same as SandBox.test2, plus or minus about 2%.
3. JRE 7 (Milestone 6, build b76) did not significantly affect the performance of any of these tests. In fact the tests performed roughly 1-2% slower on average when run in JRE 7 vs JRE 6 (update 18).

I thought perhaps the fact that extensionMethod always returned 1 was having an affect on the tests, so as a variation of the test, I added a long field to SealedClass, and made extensionMethod increment the field by 1. With this change, the difference between test2 and test1 was reduced to only a 18% improvement, vs. a 24.67% improvement. JRE 7 caused a less than 1% performance decrease over the same tests in jre 6.
I'm not sure how conclusive these tests are, being a microbenchmark, but it appears that there is reasonably strong evidence that even with the latest and greatest VM, lightweight extension methods could potentially provide a significant performance boost over the equivalent implicit conversion.

--
http://mapsdev.blogspot.com/
Marcelo Takeshi Fukushima

Escape analysis is on by default in JRE 7 (since I think build b73 or b72. I'm using b76), so it would have shown up there if that was the real bottleneck. 
Also, I tried enabling it for JRE 6 explicitly, but it no longer recognizes the option: Unrecognized VM option '+DoEscapeAnalysis'
On Wed, Jun 2, 2010 at 1:34 PM, Marcelo Fukushima <takeshi10@gmail.com> wrote:

have you tried enabling EscapeAnalysis (JRE6u14+)? it used to be off by default and you had to pass
-XX:+DoEscapeAnalysis
to enable it
theoretically, with EA and these scenarios, the overhead would be a lot smaller (if not zero)

On Wed, Jun 2, 2010 at 12:50 PM, Jeremy Bell <bell.jeremy@gmail.com> wrote:

I did a quick microbenchmark on JRE 6 (update 18) and JRE 7 (milestone 5, build b76) on the following code:
// SandBox.scalapackage org.jeb
sealed class SealedClass { }
object ExtensionClass { implicit def sealedClassToWrapperClass(self: SealedClass) : WrapperClass = { new WrapperClass(self) } def extensionMethod(self: SealedClass) = 1 }
class WrapperClass(val self: SealedClass) { def extensionMethod = 1}
import ExtensionClass._
object SandBox { def test1(arg: SealedClass): Unit = { arg.extensionMethod } def test2(arg: SealedClass): Unit = { ExtensionClass.extensionMethod(arg) }}
// Main.javapublic class Main {
static int extensionMethod(SealedClass arg) { return 1; } static void test3(SealedClass arg) { extensionMethod(arg); }
public static void main(String[] args) { SealedClass arg = new SealedClass(); // JIT/hotspot warmup for(int i = 0; i < 3000; ++i) System.currentTimeMillis(); //for(int i = 0; i < 3000; ++i) SandBox.test1(arg); for(int i = 0; i < 3000; ++i) SandBox.test2(arg); //for(int i = 0; i < 3000; ++i) test3(arg); long startTime = System.currentTimeMillis(); long delta = 0; long iters = 0; while(delta < 5000) { //SandBox.test1(arg); SandBox.test2(arg); //test3(arg); iters++; delta = System.currentTimeMillis() - startTime; } System.out.println("delta: " + delta); System.out.println("iters: " + iters); System.out.println("iters per second: " + ((double)iters / ((double)delta / 1000.0))); }
}
Testing method:

• One test was run at a time. The others were commented out.
• The first run after a compile was thrown out
• Five runs were performed for each test after the first run post-compile. The average iters per second over the 5 runs was used for the comparison.

Here I am comparing SandBox.test1, SandBox.test2, and test3, a static method defined in Main, which calls another static method. Here are my findings:

1. SandBox.test2 performed an average of 24.67% faster than SandBox.test1
2. Main.test3 performed roughly the same as SandBox.test2, plus or minus about 2%.
3. JRE 7 (Milestone 6, build b76) did not significantly affect the performance of any of these tests. In fact the tests performed roughly 1-2% slower on average when run in JRE 7 vs JRE 6 (update 18).

I thought perhaps the fact that extensionMethod always returned 1 was having an affect on the tests, so as a variation of the test, I added a long field to SealedClass, and made extensionMethod increment the field by 1. With this change, the difference between test2 and test1 was reduced to only a 18% improvement, vs. a 24.67% improvement. JRE 7 caused a less than 1% performance decrease over the same tests in jre 6.
I'm not sure how conclusive these tests are, being a microbenchmark, but it appears that there is reasonably strong evidence that even with the latest and greatest VM, lightweight extension methods could potentially provide a significant performance boost over the equivalent implicit conversion.

--
http://mapsdev.blogspot.com/
Marcelo Takeshi Fukushima

On Wed, Jun 2, 2010 at 6:34 PM, Marcelo Fukushima wrote:
> have you tried enabling EscapeAnalysis (JRE6u14+)? it used to be off by
> default and you had to pass
> -XX:+DoEscapeAnalysis
> to enable it
> theoretically, with EA and these scenarios, the overhead would be a lot
> smaller (if not zero)

EA is disabled altogether in u18 (the option doesn't work). It should
be back in u21.

Best,
Ismael

> From: bell.jeremy@gmail.com
> To: scala@listes.epfl.ch

> Escape analysis is on by default in JRE 7 (since I think build b73 or b72. I'm using b76), so it would have shown up there if that was the real bottleneck.

Is escape analysis on by default in both server and client VMs? I thought I had seen that it was enabled only in the server VM, but that info might be out of date.

A


The New Busy is not the old busy. Search, chat and e-mail from your inbox. Get started.

If you look at the actual difference between test1 and test2 in the compiled code (this is the bytecode of the test variant I described, decompiled to java):
  public long test1(SealedClass arg)   {    return ExtensionClass..MODULE$.sealedClassToWrapperClass(arg).extensionMethod();  }  public long test2(SealedClass arg) {    return ExtensionClass..MODULE$.extensionMethod(arg);   }
What you are expecting is that with escape analysis, the JIT should be able to optimize out the extra allocation and the extra call to sealedClassToWrapperClass. So in other words, the JIT has to:

1. Inline the sealedClassToWrapperClass function, so that the WrapperClass it creates is now a local variable in the test1 function. 
2. Then, it has to recognize that this local WrapperClass reference doesn't escape, so it can replace all of its fields with local variables, and inline its constructor in the process (in other words, create a copy of arg). 
3. Then it has to either inline WrapperClass.extensionMethod, using this new temporary copy of arg, or it has to generate a new version of extensionMethod that takes a SealedClass as an argument (and I don't think the JIT would do this). 
4. Then, it has to recognize that the temporary copy of arg is never anything but arg, so it can optimize out the inlined constructor, remove the temporary copy of arg, and change the inlined version of extensionMethod so that it uses arg instead of this temporary copy. 

At this point, the JIT has converted test1 to test2, but only after multiple passes of optimization. It may be that the JIT does some of this (inlining sealedClassToWrapperClass, for example), but if it doesn't do all of the passes, then test1 is always going to be slower than test2. From the tests results, its seems clear that the JIT is not able to completely transform test1 into test2, even with escape analysis enabled, and that the overhead that remains is severe enough to be significant.
On Wed, Jun 2, 2010 at 1:47 PM, Ismael Juma <mlists@juma.me.uk> wrote:

On Wed, Jun 2, 2010 at 6:34 PM, Marcelo Fukushima <takeshi10@gmail.com> wrote:
> have you tried enabling EscapeAnalysis (JRE6u14+)? it used to be off by
> default and you had to pass
> -XX:+DoEscapeAnalysis
> to enable it
> theoretically, with EA and these scenarios, the overhead would be a lot
> smaller (if not zero)

EA is disabled altogether in u18 (the option doesn't work). It should
be back in u21.

Best,
Ismael

Interesting, because the JRE 7 client VM doesn't support the -XX:+DoEscapeAnalysis option either. I assumed it was because it was just enabled all the time. 

On Wed, Jun 2, 2010 at 2:08 PM, adriaN kinG <ceroxylon@hotmail.com> wrote:

> From: bell.jeremy@gmail.com
> To: scala@listes.epfl.ch

> Escape analysis is on by default in JRE 7 (since I think build b73 or b72. I'm using b76), so it would have shown up there if that was the real bottleneck.

Is escape analysis on by default in both server and client VMs? I thought I had seen that it was enabled only in the server VM, but that info might be out of date.

A


The New Busy is not the old busy. Search, chat and e-mail from your inbox. Get started.

another thing to be aware of is that in the path without the allocation, JIT may be able to inline all the methods and eventually eliminate the loop entirely (since it basically does nothing with the number returned from the method)... ive seen it happen on micro benchmarks that would become a noop after a while
On Wed, Jun 2, 2010 at 3:30 PM, Jeremy Bell <bell.jeremy@gmail.com> wrote:

If you look at the actual difference between test1 and test2 in the compiled code (this is the bytecode of the test variant I described, decompiled to java):
  public long test1(SealedClass arg)   {    return ExtensionClass..MODULE$.sealedClassToWrapperClass(arg).extensionMethod();  }  public long test2(SealedClass arg) {    return ExtensionClass..MODULE$.extensionMethod(arg);   }
What you are expecting is that with escape analysis, the JIT should be able to optimize out the extra allocation and the extra call to sealedClassToWrapperClass. So in other words, the JIT has to:

1. Inline the sealedClassToWrapperClass function, so that the WrapperClass it creates is now a local variable in the test1 function. 
2. Then, it has to recognize that this local WrapperClass reference doesn't escape, so it can replace all of its fields with local variables, and inline its constructor in the process (in other words, create a copy of arg). 
3. Then it has to either inline WrapperClass.extensionMethod, using this new temporary copy of arg, or it has to generate a new version of extensionMethod that takes a SealedClass as an argument (and I don't think the JIT would do this). 
4. Then, it has to recognize that the temporary copy of arg is never anything but arg, so it can optimize out the inlined constructor, remove the temporary copy of arg, and change the inlined version of extensionMethod so that it uses arg instead of this temporary copy. 

At this point, the JIT has converted test1 to test2, but only after multiple passes of optimization. It may be that the JIT does some of this (inlining sealedClassToWrapperClass, for example), but if it doesn't do all of the passes, then test1 is always going to be slower than test2. From the tests results, its seems clear that the JIT is not able to completely transform test1 into test2, even with escape analysis enabled, and that the overhead that remains is severe enough to be significant.

i wonder if its really severe in practice though. Were are talking about around 30% increase in time on a loop that does almost nothing (hence the increase in absolute time must be extremely small). Would that influence in the overral execution time of a program? im not sure
its not a criticism (or dismissal) or anything, just a general wonder 

On Wed, Jun 2, 2010 at 1:47 PM, Ismael Juma <mlists@juma.me.uk> wrote:

On Wed, Jun 2, 2010 at 6:34 PM, Marcelo Fukushima <takeshi10@gmail.com> wrote:
> have you tried enabling EscapeAnalysis (JRE6u14+)? it used to be off by
> default and you had to pass
> -XX:+DoEscapeAnalysis
> to enable it
> theoretically, with EA and these scenarios, the overhead would be a lot
> smaller (if not zero)

EA is disabled altogether in u18 (the option doesn't work). It should
be back in u21.

Best,
Ismael

--
http://mapsdev.blogspot.com/
Marcelo Takeshi Fukushima

I agree that this is a useful optimization.  But do we really need to create anything so complicated for it?

Why not just say that

  def methodName(c: ClassA, ...)

can be invoked using

  c.methodName(...)

and that any method from an object can be used that way?  (Or any method from anything--why not, if the calling class can be inferred anyway?)

This gives all of the performance benefit with fewer special cases and less headache.

So, for example, if you import scala.math._ and pick up "max", you could just
  5.0 max 4.2
which would be converted to
  max(5.2,4.2)
without any additional fuss.

This would require one extra check for the compiler--the (5.0).max(4.2) interpretation would come first, but it would see if max(5.2,4.2) was a possibility before seeing if (5.0) could be implicitly converted to something that had a max method.

  --Rex

On Tue, Jun 1, 2010 at 12:36 PM, Jeremy Bell <bell.jeremy@gmail.com> wrote:

First, some background: 
As you know, in Scala the current idiomatic way of adding a new method to an existing type is to:

1. Create a new class that defines the new method. Usually this class takes the existing type as a default constructor parameter.
2. Write an implicit conversion function to convert the existing type into the new "extension" type.
3. Call the new method on an instance of the existing type. Compiler generates an implicit conversion to an instance of this new type, the new method is called on this temporary instance, and then the temporary instance is discarded.

This works great for the most part, but there are drawbacks:

1. It is cumbersome to write both the new class and the implicit conversion, if just a small number of extension methods are needed. 
2. The implicit conversion and the "wrapper" class are highly related, but often separated in the code.
3. For many use cases, the "boxing" operation performed during an implicit conversion is needlessly inefficient. Every time one of the new methods are called, a new temporary object is constructed, then discarded after the method is called. It's possible that escape analysis can reduce the cost of this operation, but not eliminate it completely.

To address these issues, I propose two new features: lightweight extension methods, and a related proposal: extension classes.
Lightweight extension methods:
A lightweight extension method is simply a method defined in an object which takes at least one parameter: an instance of the original type. In addition, the method is marked with an annotation declaring it an extension method:

package somelibobject ExtensionMethods {

@ExtensionMethod def extensionMethod(self: SealedType, args: ArgType): ReturnType = ......

}

 If the ExtensonMethods object is imported (i.e. "import somelib.ExtensionMethods._"), and the extensionMethod method is called on an instance of SealedType, the following transformation occurs:

import somelib.ExtensionMethods._

var original = new SealedType
// original code original.extensionMethod(new ArgType)
// compiler generated code: extensionMethod(original, new ArgType)// in other words: somelib.ExtensonMethods.extensionMethod(original, new ArgType)

As you can see, no temporary object needs to be generated, as we are simply calling an object method on an instance of the original type.
The benefits are:

1. It is a lot simpler to define a single extension method in an object (possibly along with other extension methods of different types) than to do an implicit conversion to a new type. This is especially helpful for defining internal DSLs.
2. It is more efficient, because no boxing of the original instance is necessary to call the extension method.

The drawback is:

1. If in the future you decide to create a wrapper class after all, with an implicit conversion, and you want to include the extension methods you've already defined, then you need to redefine all those methods in the wrapper class and call them on the original instance.

To address this drawback, I propose the concept of an extension class. An extension class is a short-hand for defining both a wrapper class and the implicit conversion needed. However, in addition the compiler can potentially optimize by generating extension methods when possible:

// Note: MyExtensionClass could potentially mixin traits or inherit another class @ExtensionClass(extendedClass = classOf[SealedType])// default constructor must have at least one argument: the wrapped instance. // if more than one argument, then the first argument is used as the wrapped instance// we call it self here as a convention only class MyExtensionClass(val self: SealedType) { 

private var additionalVar = 3 // can define vars

def extensionMethod(arg: ArgType): Unit = println(arg.toString)def implicitConversionMethod(arg: ArgType): Unit = println(additionalVar.toString + arg.toString)

}

The compiler should at a minimum generate an implicit conversion to this extension class with all the methods defined as usual.
However, in addition, the compiler may be able to determine whether a particular method can be implemented using a lightweight extension method. For example, in the above MyExtensionClass, the extensionMethod method can be defined as both a lightweight extension method and a method of MyExtensionClass (which simply calls the lightweight extension method), because the only field of MyExtensionClass it references is the wrapped SealedType. However, the implicitConversionMethod cannot be made into a lightweight extension method, because it references the extension class's additionalVar field, so an implicit conversion will always be done when the implicitConversionMethod is called"

// compiler generated code: @ExtensionClass(extendedClass = classOf[SealedType]) // annotation is retainedobject MyExtensionClass {

@ExtensionMethod def extensionMethod(self: SealedType, arg: ArgType): Unit = ...// implicitConversionMethod is not defined here

implicit def sealedTypeToMyExtensionClass(self: SealedType) = new MyExtensionClass(self)

}
class MyExtensionClass(val self: SealedType) {

private var additionalVar = 3

def extensionMethod(arg: ArgType): Unit = MyExtensionClass.extensionMethod(self, arg)

def implicitConversionMethod(arg: ArgType): Unit = ...

}

The great thing about this is that the code need not care whether the compiler generates lightweight extension methods or not - it will work the same either way. This means it may be possible to convert existing implicit conversions into an extension class without breaking client code:

import somelib._ import somelib.MyExtensionClass._
// code can use references to MyExtensionClass, and they work like you would expect:def foo1(arg: MyExtensionClass) {

arg.implicitConversionMethod(new ArgType)
// code can still call methods that are converted to lightweight extension methods// but in this case it calls the extensionMethod defined in MyExtensionClass, because // we're working on an instance of MyExtensionClass class. That method just redirects to// the lightweight extension method defined in the MyExtensionClass companion object. arg.extensionMethod(new ArgType) 

}
def foo(arg: SealedType): Unit = {

// calling code doesn't care how the compiler implements the extension methods:

arg.extensionMethod(new ArgType) // calls lightweight extension method, no implicit conversion needed

arg.implicitConversionMethod(new ArgType) // implicit conversion to MyExtensionClass required foo2(arg) // implicit conversion to MyExtensionClass required

}

OK, rerunning the tests with the JRE 7 server VM with escape analysis enabled explicitly, shows a massive improvement in test2/test3 performance, though still faster than test1 by 4.5%.
However, I still stand by my proposals. 4.5% is still a respectable performance boost, and there are still the usability benefits to extension classes/methods. In addition, on platforms such as the current JRE 6 and android's dalvik VM, the performance benefits are more pronounced. 
Also, consider this alternative for my proposal that does not require any changes to the syntax: Drop support for user defined lightweight extension methods, and only support extension classes. If you do that, and allow the compiler to optimize extension class method calls into lightweight static extension method calls, then you don't even need to change any scala syntax to get the same benefits, because to the client code it just looks like you are still doing an implicit conversion. And, since the compiler is doing the optimization, you don't need escape analysis to be enabled to get the same performance boost.

On Wed, Jun 2, 2010 at 2:56 PM, Rex Kerr <ichoran@gmail.com> wrote:

  5.0 max 4.2
which would be converted to
  max(5.2,4.2)

Um, hopefully Scala wouldn't make mathematical errors for you!  Should be
  max(5.0,4.2)
of course.

  --Rex

On Wed, Jun 2, 2010 at 3:05 PM, Jeremy Bell <bell.jeremy@gmail.com> wrote:

Also, consider this alternative for my proposal that does not require any changes to the syntax: Drop support for user defined lightweight extension methods, and only support extension classes. If you do that, and allow the compiler to optimize extension class method calls into lightweight static extension method calls, then you don't even need to change any scala syntax to get the same benefits, because to the client code it just looks like you are still doing an implicit conversion. And, since the compiler is doing the optimization, you don't need escape analysis to be enabled to get the same performance boost.

In fact, the only reason you couldn't reliably do this optimization on all existing implicit conversions as written is that it would only work for implicit conversions of a very specific form (the form I'm proposing for extension classes), otherwise the compiler would not be able to do the optimization:
object ImplicitConversions {

// implicit conversion method must follow this simple pattern. Any special logic defined here, and the compiler // wouldn't know it was safe to make the optimization. With extension classes, this implicit conversion// function is just generated for youimplicit def implicitConverterFunc(source: SourceType): DestinationType = new DestinationType(source)

}

// The destination type must follow this constructor pattern, so the compiler knows which field is the wrapped instance class DestinationType(source: SourceType) {

def extensionMethod(<any arguments>) = {

// Compiler traverses the call tree of this method to determine whether source (the wrapped instance) // is the only field that is referenced from this method. If so, this method can // be optimized as a static lightweight extension method.

}

}
object ClientCode {

def foo = (new SourceType).extensionMethod(<any arguments>)

}
Without extension classes, the optimization is limited to a specific form of implicit conversion. With extension classes, this specific form would be enforced by the compiler when you annotate the DestinationType with @ExtensionClass, and in the process you are explicitly documenting the purpose of DestinationType as extending the functionality of SourceType (the default constructor argument type), and also saving yourself the trouble of writing an implicit conversion method from SourceType to DestinationType and from DestinationType to SourceType. 
With lightweight extension methods, you're documenting the fact that there is no wrapper class and no implicit conversion - it's just a static method being called on an existing type, that you can conveniently call as if it were a method of SourceType.
Personally, I think this is less complicated than implicit conversions as they are currently. The complexity is mostly from the compiler's point of view rather than the programmer's. I think it actually simplifies a lot of use cases for the programmer (particularly in internal DSL definitions - one of the major selling points of scala). The performance improvements are really just a bonus. 

On Wed, Jun 02, 2010 at 04:08:56PM -0400, Jeremy Bell wrote:
> With lightweight extension methods, you're documenting the fact that
> there is no wrapper class and no implicit conversion - it's just a
> static method being called on an existing type, that you can
> conveniently call as if it were a method of SourceType.

I haven't followed this super closely so maybe you have brought in more
background than I can see in your opening proposal (which is to say,
none) but I'm wondering if you're aware this has been discussed.
Extensively. On multiple occasions. If you want to accomplish anything
beyond wheel spinning I'd suggest you clearly tailor your arguments
around the previous outcomes.

First Google link for the lazy: http://scala-programming-language.1934581.n4.nabble.com/On-the-topic-of-the-pimp-my-library-pattern-and-performance-td2009581.html

...and a few links down...   http://comments.gmane.org/gmane.comp.lang.scala.debate/3222     On Wed, Jun 2, 2010 at 4:17 PM, Paul Phillips <paulp@improving.org> wrote:

On Wed, Jun 02, 2010 at 04:08:56PM -0400, Jeremy Bell wrote:
> With lightweight extension methods, you're documenting the fact that
> there is no wrapper class and no implicit conversion - it's just a
> static method being called on an existing type, that you can
> conveniently call as if it were a method of SourceType.

I haven't followed this super closely so maybe you have brought in more
background than I can see in your opening proposal (which is to say,
none) but I'm wondering if you're aware this has been discussed.
Extensively.  On multiple occasions.  If you want to accomplish anything
beyond wheel spinning I'd suggest you clearly tailor your arguments
around the previous outcomes.

--
Paul Phillips      | Before a man speaks it is always safe to assume
Apatheist          | that he is a fool.  After he speaks, it is seldom
Empiricist         | necessary to assume it.
i pull his palp!   |     -- H. L. Mencken

--
http://erikengbrecht.blogspot.com/

another thing to be aware of is that in the path without the allocation, JIT may be able to inline all the methods and eventually eliminate the loop entirely (since it basically does nothing with the number returned from the method)... ive seen it happen on micro benchmarks that would become a noop after a while

The modified test code that I mentioned does actual work in the extension method (modifies a field in the SealedType instance), and  after that change test2 dropped from 24% boost to an 18% boost over test1, so I think you are right in that the JIT was doing some clever optimization of the original test. The 4.5% boost with escape analysis that I mentioned is with this modified test rather than the original test, so we are comparing an 18% boost without escape analysis to a 4.5% boost with escape analysis. I apologize for the confusion.

i wonder if its really severe in practice though. Were are talking about around 30% increase in time on a loop that does almost nothing (hence the increase in absolute time must be extremely small). Would that influence in the overral execution time of a program? im not sure

I think most production software is just not going to care that much about performance really. But, if a certain piece of production software is truly concerned with performance, then I think it is likely to be more complicated, use more implicit conversions (maybe even with escaping wrapper class references?), and generate a lot more allocations. So, I think for production software where performance is a true concern, you would see at least this much of a boost and maybe more. There are tons of implicit conversions in the scala library, so it's not difficult to write code that gets a lot of implicit conversions inserted.

On Wed, Jun 2, 2010 at 4:17 PM, Paul Phillips <paulp@improving.org> wrote:

I haven't followed this super closely so maybe you have brought in more
background than I can see in your opening proposal (which is to say,
none) but I'm wondering if you're aware this has been discussed.
Extensively.  On multiple occasions.  If you want to accomplish anything
beyond wheel spinning I'd suggest you clearly tailor your arguments
around the previous outcomes.

Fair enough. Here goes. 
Regarding the discussion here:http://scala-programming-language.1934581.n4.nabble.com/On-the-topic-of-the-pimp-my-library-pattern-and-performance-td2009581.html
Pros of my proposals relative to those in this discussion:

• The only new syntax I introduce is new lookup behavior when invoking a method on an instance. After looking for methods of the instance's type, and before looking for methods in types for which there is an imported implicit conversion from the instance's type, the compiler will look for imported lightweight extension methods for the instance's type. The extension  methods themselves are defined like any other method, except they have an annotation on them. The "this" parameter is implied by it's position in the argument list rather than special syntax.
• If you ignore my user defined lightweight extension method proposal, and look only at the extension class proposal, then no new syntax at all is needed. This could be implemented with a compiler plugin, or even a bytecode instrumentation tool after compilation is finished.
• It's possible that the compiler could detect when an implicit conversion and its associated wrapper type follows the extension class pattern, and do the optimization without even needing the annotations to be explicit. The benefit of the annotations are that the implicit conversions are generated for you, and the extension class form is ensured by the compiler (generates an error when the extension class is not in the right form, etc..). 

My extension class proposal appears to be pretty much the exact same thing as this one:http://comments.gmane.org/gmane.comp.lang.scala.debate/3222
Except for the name of the annotation, and the flexibility to mix methods which can be optimized to be lightweight static methods, with ones that can't.

On Wed, Jun 2, 2010 at 5:07 PM, Jeremy Bell <bell.jeremy@gmail.com> wrote:

On Wed, Jun 2, 2010 at 4:17 PM, Paul Phillips <paulp@improving.org> wrote:

I haven't followed this super closely so maybe you have brought in more
background than I can see in your opening proposal (which is to say,
none) but I'm wondering if you're aware this has been discussed.
Extensively.  On multiple occasions.  If you want to accomplish anything
beyond wheel spinning I'd suggest you clearly tailor your arguments
around the previous outcomes.

I forgot to add one more pro for my proposal (though I think other proposals have this benefit as well):

•  These features have merit outside of the performance benefits. That is, they simplify code for an important use case (namely internal DSL development), and that code using these features more closely expresses its purpose. An implicit conversion could be meant as a way of adding methods to an existing type, or as a true conversion from one type to another (i.e. int to float). Extension methods and extension classes are by nature documenting their purpose specifically to extend another class with additional methods and possibly state. Lightweight extension methods, to a lesser degree, are documenting the fact that there is no implicit conversion to a wrapper class. In other words, there is no need to look up the documentation for some wrapper type - the method itself is all you need to know about.

Out of curiosity (given your quoted interest in Android and Davlik)... Have you attempted any profiling or bytecode analysis following a proguard run?
Nobody is seriously considering using Scala on android without this vital step, owing to the extra size that would be imposed by all the unused classes from the Scala library.  I'd be very interested to see what the effect is of Proguard's static optimisations, which may well eliminate any overhead before the VM even sees the code.  
This would also represent a much more "real world" test.


On 2 June 2010 22:18, Jeremy Bell <bell.jeremy@gmail.com> wrote:

On Wed, Jun 2, 2010 at 5:07 PM, Jeremy Bell <bell.jeremy@gmail.com> wrote:

On Wed, Jun 2, 2010 at 4:17 PM, Paul Phillips <paulp@improving.org> wrote:

I haven't followed this super closely so maybe you have brought in more
background than I can see in your opening proposal (which is to say,
none) but I'm wondering if you're aware this has been discussed.
Extensively.  On multiple occasions.  If you want to accomplish anything
beyond wheel spinning I'd suggest you clearly tailor your arguments
around the previous outcomes.

I forgot to add one more pro for my proposal (though I think other proposals have this benefit as well):

•  These features have merit outside of the performance benefits. That is, they simplify code for an important use case (namely internal DSL development), and that code using these features more closely expresses its purpose. An implicit conversion could be meant as a way of adding methods to an existing type, or as a true conversion from one type to another (i.e. int to float). Extension methods and extension classes are by nature documenting their purpose specifically to extend another class with additional methods and possibly state. Lightweight extension methods, to a lesser degree, are documenting the fact that there is no implicit conversion to a wrapper class. In other words, there is no need to look up the documentation for some wrapper type - the method itself is all you need to know about.

--
Kevin Wright

mail/google talk: kev.lee.wright@gmail.com
wave: kev.lee.wright@googlewave.com
skype: kev.lee.wright
twitter: @thecoda

I think Rex's proposal sounds really interesting. Maybe the automatic
conversion should be confined to methods marked as such (via
annotation, keyword or a special import), but in general it sounds
very useful and doesn't add too much complexity, it seems to me.

It seems much simpler than Jeremy's proposal but with the same
benefits, if not more.

Why is nobody considering his proposal?

Regards,
Ruediger

2010/6/2 Rex Kerr :
> I agree that this is a useful optimization.  But do we really need to create
> anything so complicated for it?
>
> Why not just say that
>
>   def methodName(c: ClassA, ...)
>
> can be invoked using
>
>   c.methodName(...)
>
> and that any method from an object can be used that way?  (Or any method
> from anything--why not, if the calling class can be inferred anyway?)
>
> This gives all of the performance benefit with fewer special cases and less
> headache.
>
> So, for example, if you import scala.math._ and pick up "max", you could
> just
>   5.0 max 4.2
> which would be converted to
>   max(5.2,4.2)
> without any additional fuss.
>
> This would require one extra check for the compiler--the (5.0).max(4.2)
> interpretation would come first, but it would see if max(5.2,4.2) was a
> possibility before seeing if (5.0) could be implicitly converted to
> something that had a max method.
>
>   --Rex
>
> On Tue, Jun 1, 2010 at 12:36 PM, Jeremy Bell wrote:
>>
>> First, some background:
>> As you know, in Scala the current idiomatic way of adding a new method to
>> an existing type is to:
>>
>> Create a new class that defines the new method. Usually this class takes
>> the existing type as a default constructor parameter.
>> Write an implicit conversion function to convert the existing type into
>> the new "extension" type.
>> Call the new method on an instance of the existing type. Compiler
>> generates an implicit conversion to an instance of this new type, the new
>> method is called on this temporary instance, and then the temporary instance
>> is discarded.
>>
>> This works great for the most part, but there are drawbacks:
>>
>> It is cumbersome to write both the new class and the implicit conversion,
>> if just a small number of extension methods are needed.
>> The implicit conversion and the "wrapper" class are highly related, but
>> often separated in the code.
>> For many use cases, the "boxing" operation performed during an implicit
>> conversion is needlessly inefficient. Every time one of the new methods are
>> called, a new temporary object is constructed, then discarded after the
>> method is called. It's possible that escape analysis can reduce the cost of
>> this operation, but not eliminate it completely.
>>
>> To address these issues, I propose two new features: lightweight extension
>> methods, and a related proposal: extension classes.
>> Lightweight extension methods:
>> A lightweight extension method is simply a method defined in an object
>> which takes at least one parameter: an instance of the original type. In
>> addition, the method is marked with an annotation declaring it an extension
>> method:
>>
>> package somelib
>> object ExtensionMethods {
>>
>> @ExtensionMethod def extensionMethod(self: SealedType, args: ArgType):
>> ReturnType = ...
>> ...
>>
>> }
>>
>>  If the ExtensonMethods object is imported (i.e. "import
>> somelib.ExtensionMethods._"), and the extensionMethod method is called on an
>> instance of SealedType, the following transformation occurs:
>>
>> import somelib.ExtensionMethods._
>>
>> var original = new SealedType
>> // original code
>> original.extensionMethod(new ArgType)
>> // compiler generated code:
>> extensionMethod(original, new ArgType)
>> // in other words: somelib.ExtensonMethods.extensionMethod(original, new
>> ArgType)
>>
>> As you can see, no temporary object needs to be generated, as we are
>> simply calling an object method on an instance of the original type.
>> The benefits are:
>>
>> It is a lot simpler to define a single extension method in an object
>> (possibly along with other extension methods of different types) than to do
>> an implicit conversion to a new type. This is especially helpful for
>> defining internal DSLs.
>> It is more efficient, because no boxing of the original instance is
>> necessary to call the extension method.
>>
>> The drawback is:
>>
>> If in the future you decide to create a wrapper class after all, with an
>> implicit conversion, and you want to include the extension methods you've
>> already defined, then you need to redefine all those methods in the wrapper
>> class and call them on the original instance.
>>
>> To address this drawback, I propose the concept of an extension class. An
>> extension class is a short-hand for defining both a wrapper class and the
>> implicit conversion needed. However, in addition the compiler can
>> potentially optimize by generating extension methods when possible:
>>
>> // Note: MyExtensionClass could potentially mixin traits or inherit
>> another class
>> @ExtensionClass(extendedClass = classOf[SealedType])
>> // default constructor must have at least one argument: the wrapped
>> instance.
>> // if more than one argument, then the first argument is used as the
>> wrapped instance
>> // we call it self here as a convention only
>> class MyExtensionClass(val self: SealedType) {
>>
>> private var additionalVar = 3 // can define vars
>>
>> def extensionMethod(arg: ArgType): Unit = println(arg.toString)
>> def implicitConversionMethod(arg: ArgType): Unit =
>> println(additionalVar.toString + arg.toString)
>>
>> }
>>
>> The compiler should at a minimum generate an implicit conversion to this
>> extension class with all the methods defined as usual.
>> However, in addition, the compiler may be able to determine whether a
>> particular method can be implemented using a lightweight extension method.
>> For example, in the above MyExtensionClass, the extensionMethod method can
>> be defined as both a lightweight extension method and a method of
>> MyExtensionClass (which simply calls the lightweight extension method),
>> because the only field of MyExtensionClass it references is the wrapped
>> SealedType. However, the implicitConversionMethod cannot be made into a
>> lightweight extension method, because it references the extension class's
>> additionalVar field, so an implicit conversion will always be done when the
>> implicitConversionMethod is called"
>>
>> // compiler generated code:
>> @ExtensionClass(extendedClass = classOf[SealedType]) // annotation is
>> retained
>> object MyExtensionClass {
>>
>> @ExtensionMethod def extensionMethod(self: SealedType, arg: ArgType): Unit
>> = ...
>> // implicitConversionMethod is not defined here
>>
>> implicit def sealedTypeToMyExtensionClass(self: SealedType) = new
>> MyExtensionClass(self)
>>
>> }
>> class MyExtensionClass(val self: SealedType) {
>>
>> private var additionalVar = 3
>>
>> def extensionMethod(arg: ArgType): Unit =
>> MyExtensionClass.extensionMethod(self, arg)
>>
>> def implicitConversionMethod(arg: ArgType): Unit = ...
>>
>> }
>>
>> The great thing about this is that the code need not care whether the
>> compiler generates lightweight extension methods or not - it will work the
>> same either way. This means it may be possible to convert existing implicit
>> conversions into an extension class without breaking client code:
>>
>> import somelib._
>> import somelib.MyExtensionClass._
>> // code can use references to MyExtensionClass, and they work like you
>> would expect:
>> def foo1(arg: MyExtensionClass) {
>>
>> arg.implicitConversionMethod(new ArgType)
>> // code can still call methods that are converted to lightweight extension
>> methods
>> // but in this case it calls the extensionMethod defined in
>> MyExtensionClass, because
>> // we're working on an instance of MyExtensionClass class. That method
>> just redirects to
>> // the lightweight extension method defined in the MyExtensionClass
>> companion object.
>> arg.extensionMethod(new ArgType)
>>
>> }
>> def foo(arg: SealedType): Unit = {
>>
>> // calling code doesn't care how the compiler implements the extension
>> methods:
>>
>> arg.extensionMethod(new ArgType) // calls lightweight extension method, no
>> implicit conversion needed
>>
>> arg.implicitConversionMethod(new ArgType) // implicit conversion to
>> MyExtensionClass required
>> foo2(arg) // implicit conversion to MyExtensionClass required
>>
>> }
>

But they are!  I'm already working out how hard this would be to do with a plunging :)

On 3 Jun 2010 09:16, "Ruediger Keller" <ruediger@rk42.de> wrote:

I think Rex's proposal sounds really interesting. Maybe the automatic
conversion should be confined to methods marked as such (via
annotation, keyword or a special import), but in general it sounds
very useful and doesn't add too much complexity, it seems to me.

It seems much simpler than Jeremy's proposal but with the same
benefits, if not more.

Why is nobody considering his proposal?

Regards,
Ruediger


2010/6/2 Rex Kerr <ichoran@gmail.com>:

> I agree that this is a useful optimization.  But do we really need to create
> anything so complic...

Out of curiosity (given your quoted interest in Android and Davlik)... Have you attempted any profiling or bytecode analysis following a proguard run?

Nobody is seriously considering using Scala on android without this vital step, owing to the extra size that would be imposed by all the unused classes from the Scala library.  I'd be very interested to see what the effect is of Proguard's static optimisations, which may well eliminate any overhead before the VM even sees the code.  

This would also represent a much more "real world" test.

I have, however the way scala implements objects prevents proguard from optimizing method calls on object singletons (because they could have a side effect, namely the instantiation of the singleton MODULE$ field).
But, just to see what proguard is doing, I ran it and took a look at the bytecode. Here is my proguard configuration file (note I'm using the maximum number of optimization passes, and allowing access modification):

-injars bin;lib/scala-library.jar(!META-INF/MANIFEST.MF,!library.properties)
-outjar gen/scandroid.jar
-libraryjars lib/android.jar
-dontwarn -dontobfuscate-dontskipnonpubliclibraryclasses -dontskipnonpubliclibraryclassmembers -optimizationpasses 9 -allowaccessmodification
-keepattributes Exceptions,InnerClasses,Signature,Deprecated,                 SourceFile,LineNumberTable,*Annotation*,EnclosingMethod -keep public class org.jeb.** { public protected *; } -keep public class org.xml.sax.EntityResolver { public protected *; }

In most situations, proguard can inline the static method calls.
// this code: def main() = { val arg = new SealedClass(); var startTime = System.currentTimeMillis(); var delta: Long = 0; var iters: Long = 0; var accumulator: Long = 0; while(delta < 5000) { accumulator += SandBox.test1(arg); accumulator += SandBox.test2(arg); iters += 1; delta = System.currentTimeMillis() - startTime; } System.out.println("delta: " + delta); System.out.println("iters: " + iters); System.out.println("iters per second: "  + (iters.asInstanceOf[Double] / (delta.asInstanceOf[Double] / 1000.0))); System.out.println("accumulator: " + accumulator); }
// becomes this in the bytecode:  public final void main() {     SealedClass localSealedClass = new SealedClass();
    long l1 = System.currentTimeMillis();    long l2 = 0L;     long l3 = 0L;    long l4 = 0L;    while (l2 < 5000L)     {      // static methods are inlined. Note that the static methods in       // the SandBox class just call the non-static methods in      // the generated SandBox$ class on the MODULE singleton.       // I believe that, since accessing MODULE has the potential      // side effect of instantiating MODULE for the first time,       // proguard can't optimize these calls further      l4 = l4 + SandBox..MODULE$.test1(localSealedClass) +          SandBox..MODULE$.test2(localSealedClass);      l3 += 1L;       l2 = System.currentTimeMillis() - l1;    }     System.out.println(new StringBuilder().append("delta: ").append(Long.valueOf(l2)).toString());    System.out.println(new StringBuilder().append("iters: ").append(Long.valueOf(l3)).toString());     System.out.println(      new StringBuilder().append("iters per second: ")       .append(Double.valueOf(l3 / (l2 / 1000.0D))).toString());    System.out.println(new StringBuilder().append("accumulator: ").append(Long.valueOf(l4)).toString());   }
// however, the test1 and test2 methods in the SandBox$ class remain the same (the implicit conversion isn't optimized out):   public final long test1(SealedClass arg)  {    return ExtensionClass..MODULE$.sealedClassToWrapperClass(arg).extensionMethod();  }  public final long test2(SealedClass arg) {     return ExtensionClass..MODULE$.extensionMethod(arg);  }

So, my theory is that because Scala compiled object methods into non-static methods on a singleton object, proguard (and possibly the JIT as well) can't optimize any further than a reference to MODULE$. So to test this theory, I wrote some code in java where that situation is reversed. In other words, the non-static methods of the SandBox$ class would simply forward the call to the static methods of SandBox, so that MODULE$ is only referenced if the object is passed to a function or a reference is assigned to it. Then I checked how proguard optimizes the result. Here is the original code:
// original code (in java)public class Main { static class MySealedClass { private long i = 1; public long getI() { return i; } } static class MyWrapperClass { private MySealedClass source; public MyWrapperClass(MySealedClass source) { this.source = source; } public long wrappedExtensionMethod() { return source.getI() + 3; } } // object ExtensionMethods { static MyWrapperClass mySealedClassToMyWrapperClass(MySealedClass source) { return new MyWrapperClass(source); } static long staticExtensionMethod(MySealedClass arg) { return arg.getI() + 3; } // } // object SandBox { static long test1(MySealedClass arg) { return (new MyWrapperClass(arg)).wrappedExtensionMethod(); } static long test2(MySealedClass arg) { return staticExtensionMethod(arg); } // } // this would be in the bytecode too, but since we don't use it in this code, // I'll just put what it would be here // public final class SandBox$ { //   public static final MODULE$; //   static { new (); } //   public final long test1(MySealedClass arg) { //     return SandBox.test1(arg); // forwarded to the static method //   } //   public final long test2(MySealedClass arg) { //     return SandBox.test2(arg); //   } //   private SandBox$() { MODULE$ = this; } // }  public static void main() { MySealedClass arg = new MySealedClass(); long startTime = System.currentTimeMillis(); long delta = 0; long iters = 0; long accum1 = 0; long accum2 = 0; while(delta < 5000) { accum1 += test1(arg); accum2 += test2(arg); iters++; delta = System.currentTimeMillis() - startTime; } System.out.println("delta: " + delta); System.out.println("iters: " + iters); System.out.println("iters per second: " + ((double)iters / ((double)delta / 1000.0))); System.out.println("accum1: " + accum1); System.out.println("accum2: " + accum2); }
}
// proguard optimizes this down to this in the bytecodepublic class Main{   public static void main()  {    MySealedClass localMySealedClass1 = new MySealedClass();
    long l1 = System.currentTimeMillis();     long l2 = 0L;    long l4 = 0L;    long l5 = 0L;     long l6 = 0L;    long l3;    while (l3 < 5000L) {       // test1 call, inlined in two passes.      // proguard can't do escape analysis, so the implicit       // conversion remains, although both the conversion method and      // the extension method in MyWrapperClass are inlined.        MySealedClass localMySealedClass2 = localMySealedClass1;       l5 += new MyWrapperClass(localMySealedClass2).source.i + 3L;
      // test2 call, inlined in one pass       l6 += localMySealedClass1.i + 3L;             l4 += 1L;      l3 = System.currentTimeMillis() - l1;     }    System.out.println("delta: " + l3);     System.out.println("iters: " + l4);    System.out.println("iters per second: " + l4 / (l3 / 1000.0D));     System.out.println("accum1: " + l5);    System.out.println("accum2: " + l6);   }
  static class MySealedClass   {    // proguard conveniently eliminates the generated access methods     // if they don't have side-effects, or if the side-effects can be inlined    long i = 1L;    }
  static class MyWrapperClass {     Main.MySealedClass source;
    public MyWrapperClass(Main.MySealedClass source) {       this.source = source;    }  } }

I thought I would add another alternative to the mix for lightweight extension methods (because I don't know whether one implementation would be easier to implement than the other). This time I am reusing the implicit keyword in the method definition, and doing a small transformation at the call site. So, this is what it would look like to the programmer:

class MySourceType(var value: String)
object ExtensionMethods {
  def extensionMethod(implicit source: MySourceType) = source.value
}
import ExtensionMethods._
object ClientCode {
  def clientMethod: Unit = {
    var x = new MySourceType("hello")
    var y = x.extensionMethod
    println(y)
  }
}

And of course, the extension method could be defined at any scope that is visible:

object ClientCode {
  var counter = 0
  def countingExtensionMethod(implicit x: MySourceType) = counter += 1
  def clientMethod: Unit = {
    var x = new MySourceType("hello")
    x.countingExtensionMethod
    var y = 0
    def methodScopeExtensionMethod(implicit source: MySourceType) = y += 1
    x.methodScopeExtensionMethod
  }
}


The transformation happens at the call site (the x.extensionMethod call in clientMethod). What happens is that when the compiler sees the x.extensionMethod call, and it can't find a method extensionMethod defined in MySourceType, it pretends x is an implicit var, and looks for a method extensionMethod with an implicit var of type MySourceType (specifically, a method where the FIRST argument is an implicit argument of type MySourceType). If there is one, it calls the method, substituting x for the implied argument:

object ClientCode {
  def clientMethod: Unit = {
    var x = new MySourceType("hello")
    var y = extensionMethod(x)
    println(y)
  }
}