no-specialization viral?

Hi Nils,
On Mon, Aug 23, 2010 at 2:52 PM, Nils Kilden-Pedersen <nilskp@gmail.com> wrote:

Because of the early problems with @specialize, I made a habit of compiling O/R Broker with 'no-specialization'. It now appears that if a project is compiled against the jar, they get the following error, unless it too have 'no-specialization' enabled: 

"error: class file needed by BrokerBuilder is missing.
reference value apache of package org refers to nonexisting symbol."

As indicated, this goes away if 'no-specialization' is enabled.

O/R Broker has two optional runtime dependencies, one of them being Apache Velocity, which is where the 'apache' reference comes from. However I don't know how that relates to specialization and why it goes away when disabled.

It is very hard for me to see that as well, but I can have some guesses. Specialization needs to know if a class inherits any method that needs to be specialized. For instance, overriding a specializable method needs to override both the generic and its specialized variant. In that sense, specialization has to look up the hierarchy of classes, and may need to know the types of classes that are not directly referenced in your code. There is no way around it, so this may be what you experience. But if this is true, compilation would fail even when both modules are compiled with specialization, simply because the compiler needs more type information than before.  

I assume that 'no-specialization' is not viral and a jar compiled with specialization disabled should be perfectly usable in a project that has it enabled (and vice versa), and I also assume that one should be able to compile against a jar that has runtime dependencies, so long as one does not directly reference those dependencies.

This is not exactly the case, for the reason outline earlier. There is another issue, which was discovered too late for inclusion in 2.8.0, and which will be fixed for the next release: code depending on a specialized library compiled with no-specializaition won't work when specialized itself (it will try to use specialized methods that are not there). 
iulian

If those assumptions are correct, this looks like a bug.

Nils

--
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais

On Mon, Aug 23, 2010 at 8:29 AM, iulian dragos <iulian.dragos@epfl.ch> wrote:

On Mon, Aug 23, 2010 at 2:52 PM, Nils Kilden-Pedersen <nilskp@gmail.com> wrote:

Because of the early problems with @specialize, I made a habit of compiling O/R Broker with 'no-specialization'. It now appears that if a project is compiled against the jar, they get the following error, unless it too have 'no-specialization' enabled: 

"error: class file needed by BrokerBuilder is missing.
reference value apache of package org refers to nonexisting symbol."

As indicated, this goes away if 'no-specialization' is enabled.

O/R Broker has two optional runtime dependencies, one of them being Apache Velocity, which is where the 'apache' reference comes from. However I don't know how that relates to specialization and why it goes away when disabled.

It is very hard for me to see that as well, but I can have some guesses. Specialization needs to know if a class inherits any method that needs to be specialized. For instance, overriding a specializable method needs to override both the generic and its specialized variant. In that sense, specialization has to look up the hierarchy of classes, and may need to know the types of classes that are not directly referenced in your code. There is no way around it, so this may be what you experience. But if this is true, compilation would fail even when both modules are compiled with specialization, simply because the compiler needs more type information than before.  

I assume that 'no-specialization' is not viral and a jar compiled with specialization disabled should be perfectly usable in a project that has it enabled (and vice versa), and I also assume that one should be able to compile against a jar that has runtime dependencies, so long as one does not directly reference those dependencies.

This is not exactly the case, for the reason outline earlier. There is another issue, which was discovered too late for inclusion in 2.8.0, and which will be fixed for the next release: code depending on a specialized library compiled with no-specializaition won't work when specialized itself (it will try to use specialized methods that are not there). 

Was this fixed for 2.8.1? If so, does it mean that something compiled with 2.8.0 no-specialization will work with a project compiled with 2.8.1 with specialization, or do both need to be compiled with 2.8.1 (or later going forwards)?

On Thu, Nov 11, 2010 at 2:40 PM, Nils Kilden-Pedersen <nilskp@gmail.com> wrote:

On Mon, Aug 23, 2010 at 8:29 AM, iulian dragos <iulian.dragos@epfl.ch> wrote:

On Mon, Aug 23, 2010 at 2:52 PM, Nils Kilden-Pedersen <nilskp@gmail.com> wrote:

Because of the early problems with @specialize, I made a habit of compiling O/R Broker with 'no-specialization'. It now appears that if a project is compiled against the jar, they get the following error, unless it too have 'no-specialization' enabled: 

"error: class file needed by BrokerBuilder is missing.
reference value apache of package org refers to nonexisting symbol."

As indicated, this goes away if 'no-specialization' is enabled.

O/R Broker has two optional runtime dependencies, one of them being Apache Velocity, which is where the 'apache' reference comes from. However I don't know how that relates to specialization and why it goes away when disabled.

It is very hard for me to see that as well, but I can have some guesses. Specialization needs to know if a class inherits any method that needs to be specialized. For instance, overriding a specializable method needs to override both the generic and its specialized variant. In that sense, specialization has to look up the hierarchy of classes, and may need to know the types of classes that are not directly referenced in your code. There is no way around it, so this may be what you experience. But if this is true, compilation would fail even when both modules are compiled with specialization, simply because the compiler needs more type information than before.  

I assume that 'no-specialization' is not viral and a jar compiled with specialization disabled should be perfectly usable in a project that has it enabled (and vice versa), and I also assume that one should be able to compile against a jar that has runtime dependencies, so long as one does not directly reference those dependencies.

This is not exactly the case, for the reason outline earlier. There is another issue, which was discovered too late for inclusion in 2.8.0, and which will be fixed for the next release: code depending on a specialized library compiled with no-specializaition won't work when specialized itself (it will try to use specialized methods that are not there). 

Was this fixed for 2.8.1? If so, does it mean that something compiled with 2.8.0 no-specialization will work with a project compiled with 2.8.1 with specialization, or do both need to be compiled with 2.8.1 (or later going forwards)?

This was bug #3650, which appears in the list for 2.8.1, so it should be there. Unfortunately, code compiled with 2.8.0 -no-specialization won't benefit from this fix. That's because the jar files in 2.8.0 did not have any indication whether the library was compiled with specialization or not.
cheers,iulian 


--
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais

On Thu, Nov 11, 2010 at 11:44 AM, iulian dragos <iulian.dragos@epfl.ch> wrote:

This was bug #3650, which appears in the list for 2.8.1, so it should be there. Unfortunately, code compiled with 2.8.0 -no-specialization won't benefit from this fix. That's because the jar files in 2.8.0 did not have any indication whether the library was compiled with specialization or not.

Ok, good to know. That should explain why, using 2.8.1, I cannot compile against a 2.8.0 no-specialization jar, regardless of no-specialization on the 2.8.1 compile.
If I compile a jar with 2.8.1 will it be compatible with 2.8.0 dependent code? In any combination of specialization?
2.8.0 no-specalization=true => 2.8.0 no-specialization=true: Works 2.8.0 no-specalization=true => 2.8.0 no-specialization=false: Does not work2.8.0 no-specialization=true => 2.8.1: Does not work, regardless of no-specialization2.8.0 no-specialization=false => 2.8.1: Haven't tested, but presumably works
2.8.1 => 2.8.0 ?

On Thu, Nov 11, 2010 at 7:46 PM, Nils Kilden-Pedersen <nilskp@gmail.com> wrote:

On Thu, Nov 11, 2010 at 11:44 AM, iulian dragos <iulian.dragos@epfl.ch> wrote:

This was bug #3650, which appears in the list for 2.8.1, so it should be there. Unfortunately, code compiled with 2.8.0 -no-specialization won't benefit from this fix. That's because the jar files in 2.8.0 did not have any indication whether the library was compiled with specialization or not.

Ok, good to know. That should explain why, using 2.8.1, I cannot compile against a 2.8.0 no-specialization jar, regardless of no-specialization on the 2.8.1 compile.
If I compile a jar with 2.8.1 will it be compatible with 2.8.0 dependent code? In any combination of specialization?

You mean you have a binary compiled with 2.8.1, and you compile against it using 2.8.0? It should work in all combinations.
iulian 

2.8.0 no-specalization=true => 2.8.0 no-specialization=true: Works 2.8.0 no-specalization=true => 2.8.0 no-specialization=false: Does not work2.8.0 no-specialization=true => 2.8.1: Does not work, regardless of no-specialization2.8.0 no-specialization=false => 2.8.1: Haven't tested, but presumably works
2.8.1 => 2.8.0 ?

--
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais

On Mon, Aug 23, 2010 at 8:29 AM, iulian dragos <iulian.dragos@epfl.ch> wrote:

Hi Nils,
On Mon, Aug 23, 2010 at 2:52 PM, Nils Kilden-Pedersen <nilskp@gmail.com> wrote:

Because of the early problems with @specialize, I made a habit of compiling O/R Broker with 'no-specialization'. It now appears that if a project is compiled against the jar, they get the following error, unless it too have 'no-specialization' enabled: 

"error: class file needed by BrokerBuilder is missing.
reference value apache of package org refers to nonexisting symbol."

As indicated, this goes away if 'no-specialization' is enabled.

O/R Broker has two optional runtime dependencies, one of them being Apache Velocity, which is where the 'apache' reference comes from. However I don't know how that relates to specialization and why it goes away when disabled.

It is very hard for me to see that as well, but I can have some guesses. Specialization needs to know if a class inherits any method that needs to be specialized. For instance, overriding a specializable method needs to override both the generic and its specialized variant. In that sense, specialization has to look up the hierarchy of classes, and may need to know the types of classes that are not directly referenced in your code. There is no way around it, so this may be what you experience. But if this is true, compilation would fail even when both modules are compiled with specialization, simply because the compiler needs more type information than before.

Unfortunately, I now have a situation where, with 2.8.1, it's impossible to deliver a jar with optional dependencies. This is a huge regression in my opinion. I have tried to compile O/R Broker with both 2.8.0 and 2.8.1 using no-specialization. And in both cases It is now impossible to compile against either when using 2.8.1 without supplying all optional dependencies on the classpath. 
If I understood your older comment above correctly, this should only be the case when using specialization, but that no longer appears to be the case.
I hope this is a bug.

On Sat, Nov 13, 2010 at 11:58 PM, Nils Kilden-Pedersen <nilskp@gmail.com> wrote:

On Mon, Aug 23, 2010 at 8:29 AM, iulian dragos <iulian.dragos@epfl.ch> wrote:

Hi Nils,
On Mon, Aug 23, 2010 at 2:52 PM, Nils Kilden-Pedersen <nilskp@gmail.com> wrote:

Because of the early problems with @specialize, I made a habit of compiling O/R Broker with 'no-specialization'. It now appears that if a project is compiled against the jar, they get the following error, unless it too have 'no-specialization' enabled: 

"error: class file needed by BrokerBuilder is missing.
reference value apache of package org refers to nonexisting symbol."

As indicated, this goes away if 'no-specialization' is enabled.

O/R Broker has two optional runtime dependencies, one of them being Apache Velocity, which is where the 'apache' reference comes from. However I don't know how that relates to specialization and why it goes away when disabled.

It is very hard for me to see that as well, but I can have some guesses. Specialization needs to know if a class inherits any method that needs to be specialized. For instance, overriding a specializable method needs to override both the generic and its specialized variant. In that sense, specialization has to look up the hierarchy of classes, and may need to know the types of classes that are not directly referenced in your code. There is no way around it, so this may be what you experience. But if this is true, compilation would fail even when both modules are compiled with specialization, simply because the compiler needs more type information than before.

Unfortunately, I now have a situation where, with 2.8.1, it's impossible to deliver a jar with optional dependencies. This is a huge regression in my opinion. I have tried to compile O/R Broker with both 2.8.0 and 2.8.1 using no-specialization. And in both cases It is now impossible to compile against either when using 2.8.1 without supplying all optional dependencies on the classpath. 

There are two different issues at play. One was related to invalid references to specialized symbols inside a library compiled with -no-specialization (which was fixed). The other was that the compilation classpath has to have the classes that your code (indirectly) refers to. Maybe this can be improved, but as I explained earlier, the compiler needs to look at classes in your classpath for potential specializations. For instance, to see if your code overrides a specialized method. Can you give me a code example when this happens and you think it's not necessary?
iulian 

If I understood your older comment above correctly, this should only be the case when using specialization, but that no longer appears to be the case.
I hope this is a bug.

--
« Je déteste la montagne, ça cache le paysage »
Alphonse Allais

On Sun, Nov 14, 2010 at 11:15 AM, iulian dragos <iulian.dragos@epfl.ch> wrote:

There are two different issues at play. One was related to invalid references to specialized symbols inside a library compiled with -no-specialization (which was fixed). The other was that the compilation classpath has to have the classes that your code (indirectly) refers to. Maybe this can be improved,

It's something that ought to be looked into.

 

but as I explained earlier, the compiler needs to look at classes in your classpath for potential specializations. For instance, to see if your code overrides a specialized method.

In my case, the BrokerBuilder has private lazy references to both FreeMarker and Velocity classes. They will not be instantiated unless required by the code using O/R Broker. Is specialization not part of the byte code signature for a class? If not, maybe it should be, because that would indicate to the compiler that any dependent classes of a no-specialization class would not need to be checked, right?
 

Can you give me a code example when this happens and you think it's not necessary?

O/R Broker provides support for two template engines, Velocity and FreeMarker, for dynamic SQL generation. Neither is required, but typically only one of them will be used. Using both would be a bit weird. Yet it now appears that for someone to use O/R Broker they would have to include a jar on the classpath that they have no use for. Furthermore, both Velocity and FreeMarker are Java libraries, which is probably the most common case for jar dependencies in Scala for a long time to come, so there is no concept of specialization. Not sure if there's any way to detect that, when the classes are unavailable, other than what I wrote above.

Another solution could be some compiler override to tell the compiler to not do this check.

BTW, any idea why this was not a problem in 2.8.0 with no-specialization enabled?