Re: Tracking changed files for recompilation

On Mon, Feb 9, 2009 at 1:53 PM, Sean McDirmid <sean.mcdirmid@gmail.com> wrote:

Sigh. Should Scalac cater to scalac?

In a word: No.

It certainly should be able to *handle* scalac, but scalac's design is highly atypical. The ability to cater to the remainder of Scala projects which don't look like scalac is much more important.

I'll certainly look into improving the behaviour on scalac and similar. I'm just not willing to compromise on correctness in other cases in order to do so.

On Mon, Feb 9, 2009 at 2:03 PM, David MacIver wrote:
> I'll certainly look into improving the behaviour on scalac and similar. I'm
> just not willing to compromise on correctness in other cases in order to do
> so.

Do we have any evidence that there's even an issue with building
scalac under the scheme you're proposing? I'd be very surprised if the
extra complications provided any significant measurable benefit.

Cheers,

Miles

On Mon, Feb 09, 2009 at 09:53:18PM +0800, Sean McDirmid wrote:
> Sigh. Should Scalac cater to scalac?

If that's supposed to be self-evidently yes, I would disagree.

On Mon, Feb 9, 2009 at 2:17 PM, David MacIver <david.maciver@gmail.com> wrote:

On Mon, Feb 9, 2009 at 2:11 PM, Miles Sabin <miles@milessabin.com> wrote:

On Mon, Feb 9, 2009 at 2:03 PM, David MacIver <david.maciver@gmail.com> wrote:
> I'll certainly look into improving the behaviour on scalac and similar. I'm
> just not willing to compromise on correctness in other cases in order to do
> so.

Do we have any evidence that there's even an issue with building
scalac under the scheme you're proposing? I'd be very surprised if the
extra complications provided any significant measurable benefit.

Unfortunately it's obvious.

Consider components Foo and Bar.

Foo changes. Global depends on Foo. Therefore Global needs recompiling. Foo depends on Global. Therefore Foo needs recompiling.

Err. That was supposed to be "Bar depends on Global. Therefore Bar needs recompiling. "

On Mon, Feb 9, 2009 at 2:11 PM, Miles Sabin <miles@milessabin.com> wrote:

On Mon, Feb 9, 2009 at 2:03 PM, David MacIver <david.maciver@gmail.com> wrote:
> I'll certainly look into improving the behaviour on scalac and similar. I'm
> just not willing to compromise on correctness in other cases in order to do
> so.

Do we have any evidence that there's even an issue with building
scalac under the scheme you're proposing? I'd be very surprised if the
extra complications provided any significant measurable benefit.

Unfortunately it's obvious.

Consider components Foo and Bar.

Foo changes. Global depends on Foo. Therefore Global needs recompiling. Foo depends on Global. Therefore Foo needs recompiling.

Ok. So what is a typical Scala design then? It seems like given 1000 large systems written in Scala, 999 will be atypical. 
I think its very important to define separate compilation in Scala's specification with respect to when does a file need to be recompiled when another file changes. Actually, there is always a trivial correct answer to this question: recompile everything. This is actually the answer in C#, but they have a really fast compiler and can afford it. But its a useless for Scala. So then what is the right answer for Scala? Do we require transitive closure in the separate compilation spec, or should we be more precise?  
From what I know about the Scala compiler, Scala file dependency tracking is pretty simple, with the only tricky part being traits since classes will copy private data from the traits they extend, and add bridge methods to allow for super calls. Is there anything else that I didn't catch? Definitely its possible. We don't have a spec so its hard to know. 

On Mon, Feb 9, 2009 at 10:03 PM, David MacIver <david.maciver@gmail.com> wrote:

In a word: No.

It certainly should be able to *handle* scalac, but scalac's design is highly atypical. The ability to cater to the remainder of Scala projects which don't look like scalac is much more important.

I'll certainly look into improving the behaviour on scalac and similar. I'm just not willing to compromise on correctness in other cases in order to do so.

I'll put my two separate comments in separate emails.

On Monday 09 February 2009 09:18, Sean McDirmid wrote:
> Ok. So what is a typical Scala design then? It seems like given 1000 large
> systems written in Scala, 999 will be atypical.
> I think its very important to define separate compilation in Scala's
> specification with respect to when does a file need to be recompiled when
> another file changes. Actually, there is always a trivial correct answer to
> this question: recompile everything. This is actually the answer in C#, but
> they have a really fast compiler and can afford it. But its a useless for
> Scala.

My first comment is related to Sean's remark about recompiling everything.
Perhaps we should take the compiler performance more seriously. I think the
people in this discussion have looked at speeding up the compiler at one time
or another and have seen that the typer and the backend take the most time (I
think parsing is a distant third). My initial investigations into this
suggested that perhaps a substantial time is spent loading classes and not
necessarily typechecking. I believe fsc is supposed to be faster because it
reuses the compiler instance, the benefit of which is presumably that classes
are cached (in addition to a warm jvm). Is this correct?

Anyway, I stopped using fsc, however, because I didn't think I could trust its
outputs. I don't think it invalidates its cache of loaded classes. With
dependency analysis, I think the cache could be properly managed.

With respect to the backend, if it is a computational bottleneck, it seems
like it could run in parallel since I don't think it modifies in-memory data
structures. Alternatively, if it is an I/O bottleneck, ... ideas? I imagine
that reducing the number of classes generated would help (I think my ratio is
an ugly 20:1 classes:sources in sbt), but I think that has been discussed
elsewhere to no avail.

-Mark

On Monday 09 February 2009 09:18, Sean McDirmid wrote:
> Do we require transitive
> closure in the separate compilation spec, or should we be more precise?
>
> From what I know about the Scala compiler, Scala file dependency tracking is
> pretty simple, with the only tricky part being traits since classes will
> copy private data from the traits they extend, and add bridge methods to
> allow for super calls. Is there
an/ad/eng/users/h/a/harrah/computing/code/scala/temp/./src/main/scala/A1.scala:1:ything
else that I didn't catch?
> Definitely its possible. We don't have a spec so its hard to know.

My second comment is that I think that it is very important to do as good a
job at proper recompilation as possible in as many cases as possible and to
put this before speed. I don't know about addressing the problem brought up
by Eric (in the thread on scala-user, I think), but I think that anything
that can be addressed should be addressed. For example, I improperly
minimized my counterexample for trait recompilation. I don't think this
works properly with the plugin's approach:

trait A { val x = "a" }
trait B extends A { val y = x }
trait C extends B { val z = y.length }

C no longer depends directly on A. This might be what David meant about type
signatures. Given this, I would be inclined to leave the recompilation
strategy at transitive closure. I'm happy to be proven wrong, of course.

After you get a runtime error because of something like this, you either: a)
move on b) you run the equivalent of clean before each compile you want to
have confidence c) go write some code to try to keep it from happening again.
It looks like I'm not the only one to have done a-c.

-Mark

On Mon, Feb 9, 2009 at 4:09 PM, Mark Harrah <harrah@bu.edu> wrote:

I'll put my two separate comments in separate emails.

On Monday 09 February 2009 09:18, Sean McDirmid wrote:
> Ok. So what is a typical Scala design then? It seems like given 1000 large
> systems written in Scala, 999 will be atypical.
> I think its very important to define separate compilation in Scala's
> specification with respect to when does a file need to be recompiled when
> another file changes. Actually, there is always a trivial correct answer to
> this question: recompile everything. This is actually the answer in C#, but
> they have a really fast compiler and can afford it. But its a useless for
> Scala.

My first comment is related to Sean's remark about recompiling everything.
Perhaps we should take the compiler performance more seriously.  I think the
people in this discussion have looked at speeding up the compiler at one time
or another and have seen that the typer and the backend take the most time (I
think parsing is a distant third).  My initial investigations into this
suggested that perhaps a substantial time is spent loading classes and not
necessarily typechecking.  I believe fsc is supposed to be faster because it
reuses the compiler instance, the benefit of which is presumably that classes
are cached (in addition to a warm jvm).  Is this correct?

Anyway, I stopped using fsc, however, because I didn't think I could trust its
outputs.  I don't think it invalidates its cache of loaded classes.  With
dependency analysis, I think the cache could be properly managed.

With respect to the backend, if it is a computational bottleneck, it seems
like it could run in parallel since I don't think it modifies in-memory data
structures.  Alternatively, if it is an I/O bottleneck, ... ideas?  I imagine
that reducing the number of classes generated would help (I think my ratio is
an ugly 20:1 classes:sources in sbt), but I think that has been discussed
elsewhere to no avail.

I don't know if this is still the case, but at one point I had a discussion with someone (I don't remember who) about the binary pickler combinators, the conclusion of which was that their design was really painfully limiting for performance (it basically involves creating lots of byte arrays and concatenating them as I recall). At one point they were being used for saving signatures for scala methods, etc. in the class files. Is this still the case? If so, anyone know if these are a bottle neck?

Because if they are, it might be worth ressurecting a plan that was vaguely floated a while ago about merging SBinary into the standard library and using it to replace the pickler combinators. Its design is stream based, and should be significantly more efficient. At least I hope so. :-)

Another possibility is to investigate whether FJBG is the bottleneck and whether replacing it with ASM would be an improvement.

I can't really comment on speeding up the typer. It's scary. :-)

On Monday 09 February 2009 11:18, David MacIver wrote:
> I don't know if this is still the case, but at one point I had a discussion
> with someone (I don't remember who) about the binary pickler combinators,
> the conclusion of which was that their design was really painfully limiting
> for performance (it basically involves creating lots of byte arrays and
> concatenating them as I recall). At one point they were being used for
> saving signatures for scala methods, etc. in the class files. Is this still
> the case? If so, anyone know if these are a bottle neck?
>
> Because if they are, it might be worth ressurecting a plan that was vaguely
> floated a while ago about merging SBinary into the standard library and
> using it to replace the pickler combinators. Its design is stream based, and
> should be significantly more efficient. At least I hope so. :-)
>
> Another possibility is to investigate whether FJBG is the bottleneck and
> whether replacing it with ASM would be an improvement.

I'm not too familiar with the details of this: what applies to the
backend/class generation and what applies to reading classes?

> I can't really comment on speeding up the typer. It's scary. :-)

I'm not interested in touching typechecking either; my comment/question was
that perhaps it is not the typechecking itself but reading classes from
jars/class files that is a problem that could be addressed. Hence my
question about the possibility that either of your above suggestions would
reduce the time spent reading classes in addition to writing them.

Thanks,
Mark

On Mon, Feb 9, 2009 at 4:47 PM, Mark Harrah <harrah@bu.edu> wrote:

On Monday 09 February 2009 11:18, David MacIver wrote:
> I don't know if this is still the case, but at one point I had a discussion
> with someone (I don't remember who) about the binary pickler combinators,
> the conclusion of which was that their design was really painfully limiting
> for performance (it basically involves creating lots of byte arrays and
> concatenating them as I recall). At one point they were being used for
> saving signatures for scala methods, etc. in the class files. Is this still
> the case? If so, anyone know if these are a bottle neck?
>
> Because if they are, it might be worth ressurecting a plan that was vaguely
> floated a while ago about merging SBinary into the standard library and
> using it to replace the pickler combinators. Its design is stream based, and
> should be significantly more efficient. At least I hope so. :-)
>
> Another possibility is to investigate whether FJBG is the bottleneck and
> whether replacing it with ASM would be an improvement.

I'm not too familiar with the details of this: what applies to the
backend/class generation and what applies to reading classes?

If it's pickler related, I think it's mostly about improving writing. It's possible the pickler does something stupid for reading, but I'm only aware of problems in the writing.

If it's FJBG related, similarly (the classfile reading code in FJBG is non-functioning and not used in the compiler), but there are other places where custom code might be profitable replaced with something ASM based and that might cause a performance improvement.
 

> I can't really comment on speeding up the typer. It's scary. :-)

I'm not interested in touching typechecking either; my comment/question was

Which is a shame, because as far as I can tell it really is a big bottleneck. I wonder if there's anything that can be done to make it more accessible?
 

that perhaps it is not the typechecking itself but reading classes from
jars/class files that is a problem that could be addressed.  Hence my
question about the possibility that either of your above suggestions would
reduce the time spent reading classes in addition to writing them.

It's definitely possible. It would need further investigation before I could say whether it was likely.

On Mon, Feb 09, 2009 at 11:09:51AM -0500, Mark Harrah wrote:
> I believe fsc is supposed to be faster because it reuses the compiler instance, the benefit of which is presumably
> that classes are cached (in addition to a warm jvm). Is this correct?

Importantly including the "supposed to", yes. Until a few days ago fsc was not reusing compiler instances because of
this bug:

http://lampsvn.epfl.ch/trac/scala/ticket/1683

I don't know how long it was broken (I'm presuming at some point it was working) but existing measurements should be
considered suspect.

> I don't know if this is still the case, but at one point I had a discussion with someone
> (I don't remember who) about the binary pickler combinators,
I think this was me.

> the conclusion of which was that their design was really painfully limiting for performance
> (it basically involves creating lots of byte arrays and concatenating them as I recall).
This was indeed the case, but it was also fairly easy to fix. I am now happily using the combinators (with some modifications) to read and write multi-megabyte files of complex genetic data in just a few seconds.

> At one point they were being used for saving signatures for scala methods, etc. in the class files.
> Is this still the case? If so, anyone know if these are a bottle neck?
If this is the case, that would be good. As I said, the fix is not hard, and I will be happy to assist with patches.

Andreas



From: David MacIver <david.maciver@gmail.com>
To: Mark Harrah <harrah@bu.edu>
Cc: scala-tools@listes.epfl.ch
Sent: Monday, February 9, 2009 11:18:47 AM
Subject: Re: [scala-tools] Tracking changed files for recompilation

On Mon, Feb 9, 2009 at 4:09 PM, Mark Harrah <harrah [at] bu [dot] edu" target="_blank" href="mailto:harrah@bu.edu" rel="nofollow">harrah@bu.edu> wrote:

I'll put my two separate comments in separate emails.

On Monday 09 February 2009 09:18, Sean McDirmid wrote:
> Ok. So what is a typical Scala design then? It seems like given 1000 large
> systems written in Scala, 999 will be atypical.
> I think its very important to define separate compilation in Scala's
> specification with respect to when does a file need to be recompiled when
> another file changes. Actually, there is always a trivial correct answer to
> this question: recompile everything. This is actually the answer in C#, but
> they have a really fast compiler and can afford it. But its a useless for
> Scala.

My first comment is related to Sean's remark about recompiling everything.
Perhaps we should take the compiler performance more seriously.  I think the
people in this discussion have looked at speeding up the compiler at one time
or another and have seen that the typer and the backend take the most time (I
think parsing is a distant third).  My initial investigations into this
suggested that perhaps a substantial time is spent loading classes and not
necessarily typechecking.  I believe fsc is supposed to be faster because it
reuses the compiler instance, the benefit of which is presumably that classes
are cached (in addition to a warm jvm).  Is this correct?

Anyway, I stopped using fsc, however, because I didn't think I could trust its
outputs.  I don't think it invalidates its cache of loaded classes.  With
dependency analysis, I think the cache could be properly managed.

With respect to the backend, if it is a computational bottleneck, it seems
like it could run in parallel since I don't think it modifies in-memory data
structures.  Alternatively, if it is an I/O bottleneck, ... ideas?  I imagine
that reducing the number of classes generated would help (I think my ratio is
an ugly 20:1 classes:sources in sbt), but I think that has been discussed
elsewhere to no avail.

I don't know if this is still the case, but at one point I had a discussion with someone (I don't remember who) about the binary pickler combinators, the conclusion of which was that their design was really painfully limiting for performance (it basically involves creating lots of byte arrays and concatenating them as I recall). At one point they were being used for saving signatures for scala methods, etc. in the class files. Is this still the case? If so, anyone know if these are a bottle neck?

Because if they are, it might be worth ressurecting a plan that was vaguely floated a while ago about merging SBinary into the standard library and using it to replace the pickler combinators. Its design is stream based, and should be significantly more efficient. At least I hope so. :-)

Another possibility is to investigate whether FJBG is the bottleneck and whether replacing it with ASM would be an improvement.

I can't really comment on speeding up the typer. It's scary. :-)

On Mon, Feb 9, 2009 at 8:41 PM, Windemuth Andreas <windemut@yahoo.com> wrote:

> I don't know if this is still the case, but at one point I had a discussion with someone
> (I don't remember who) about the binary pickler combinators,
I think this was me.

> the conclusion of which was that their design was really painfully limiting for performance
> (it basically involves creating lots of byte arrays and concatenating them as I recall).
This was indeed the case, but it was also fairly easy to fix. I am now happily using the combinators (with some modifications) to read and write multi-megabyte files of complex genetic data in just a few seconds.

> At one point they were being used for saving signatures for scala methods, etc. in the class files.
> Is this still the case? If so, anyone know if these are a bottle neck?
If this is the case, that would be good. As I said, the fix is not hard, and I will be happy to assist with patches.

Andreas

Good to know. Thanks.

To refresh our memories, below is my initial monologue on the byte picklers. It is only fair to say that after I sent off the cricket bit I did get a few very helpful responses, including from David and Philipp Haller, the original author. I played with thoughts of tearing them out and replacing them by Sbinary, but in the end I just fixed them and have used them happily ever after. I still think they are great.

If they are indeed still part of the compiler, I think they should be fixed, definitely. Let me know off-line where to send patches.

Andreas


On Mon, Jun 9, 2008 at 4:16 PM, Windemuth Andreas <windemut@yahoo.com> wrote:

It appears nobody is interested in the BytePickle library, since I did not see any response. That is a shame, because I think it is a really great idea, with only some minor fatal flaws in the current Scala implementation :-)

Anyway, since my last message I have discovered a more serious defect than the one mentioned originally: Due to the use of Array[Byte] for the bytestream, coupled with the functional style due to the Haskell heritage, the entire byte stream is copied every time bytes are added. Needless to say, this runs O(n^2) and slows things down to an intolerable crawl when pickling anything larger than a few small objects. I have reworked the library a bit, keeping the functional style but ditching Array[Byte], and it now runs nice and fast (O(n), anyway). If there is interest, I'll spend some more effort preparing a patch or something. I'd like someone (the current BytePickle expert, if there is such a person) to review and help with integration, though.

Who still cares about BytePickle? What do people use instead? Am I missing something?  

(cue the cricket chirps ....)

Andreas


----- Original Message ----
From: Windemuth Andreas <windemut@yahoo.com>
To: scala <scala@listes.epfl.ch>
Sent: Thursday, June 5, 2008 12:53:13 PM
Subject: [scala] Trouble with BytePickle


I have had a good look at the BytePickle library, and seems like a very useful tool. I have stumbled over the following implementation of the "wrap" method:

  def wrap[a,b](i: a => b, j: b => a, pa: SPU[a]): SPU[b] =
    sequ(j, pa, (x: a) => lift(i(x)))

I do not fully understand the workings of sequ and lift, but I have noticed that the function i(x) gets called during the pickling process. From what I understand, i(x) is used to convert an underlying data presentation (such as a Pair) to a higher order object (such as a case class instance). This operation should only be needed upon unpickling, not pickling. This can be a problem if pickling large objects, as the instantiation and conversion can waste a lot of resources during pickling.

I have rewritten wrap as follows (using d: data, and o: object instead of a and b, which I get confused too easily):

  def wrap[d,o](f: d => o, g: o => d, pa: SPU[d]): SPU[o] = new SPU[o] {
    def appP(v: o, state: PicklerState): PicklerState = pa.appP(g(v), state)
    def appU(state: UnPicklerState): (o, UnPicklerState) = {
      val p = pa.appU(state)
      (f(p._1), p._2)
    }
  }

and it seems to work great for both pickling and unpickling, without the unnecessary creation of an extra object. I also find it much easier to understand, but I am not at all steeped in combinator theory.

Is there a reason that I am missing for the current implementation, or would it make sense to replace it with the more straightforward one above?

Cheers,

Andreas

I don't know. I agree that one option is improving scalac's performance by about an order of magnitude, then we can be as sloppy as we want in recompiling dependencies (as long as we are overly conservative). I don't think this is going to happen, scalac is already tuned now, and most of the time during compilation is inside the type checker. Given what scalac does for the programmer, we should just accept that its going to be slower than other compilers and be smarter about incremental and resident compilation.
There are lots of complexities in building a resident compiler like fsc or the plugin's build compiler (which are basically the same thing). One issue is invalidating stale dependencies, which isn't exactly easy to do in scalac, another object is coordinating changes in dependencies between projects. David MacIver's assertion that dependency calculator is broken in the plugin is difficult to believe because so many things in the plugin could break first leading to the same problems. Without actual bug reports on the plugin or FSC, though, its hard to tell. 

On Tue, Feb 10, 2009 at 12:09 AM, Mark Harrah <harrah@bu.edu> wrote:
 

My first comment is related to Sean's remark about recompiling everything.
Perhaps we should take the compiler performance more seriously.  I think the
people in this discussion have looked at speeding up the compiler at one time
or another and have seen that the typer and the backend take the most time (I
think parsing is a distant third).  My initial investigations into this
suggested that perhaps a substantial time is spent loading classes and not
necessarily typechecking.  I believe fsc is supposed to be faster because it
reuses the compiler instance, the benefit of which is presumably that classes
are cached (in addition to a warm jvm).  Is this correct?

Anyway, I stopped using fsc, however, because I didn't think I could trust its
outputs.  I don't think it invalidates its cache of loaded classes.  With
dependency analysis, I think the cache could be properly managed.

With respect to the backend, if it is a computational bottleneck, it seems
like it could run in parallel since I don't think it modifies in-memory data
structures.  Alternatively, if it is an I/O bottleneck, ... ideas?  I imagine
that reducing the number of classes generated would help (I think my ratio is
an ugly 20:1 classes:sources in sbt), but I think that has been discussed
elsewhere to no avail.

-Mark

On Tue, Feb 10, 2009 at 12:14 AM, Mark Harrah <harrah@bu.edu> wrote:

My second comment is that I think that it is very important to do as good a
job at proper recompilation as possible in as many cases as possible and to
put this before speed.  

This is a noble statement but not very pragmatic. To be 100% safe you have to recompile the world on each change. To be 90% safe you can do something like transitive closure recompilation. And so on...It just turns out that the 90% option is about as slow as the 100% option on many projects (because anything involving transitive closure in Scala is going to choke a lot of projects). 

I don't know about addressing the problem brought up
by Eric (in the thread on scala-user, I think), but I think that anything
that can be addressed should be addressed.  For example, I improperly
minimized my counterexample for trait recompilation.  I don't think this
works properly with the plugin's approach:

trait A { val x = "a" }
trait B extends A { val y = x }
trait C extends B { val z = y.length }

C no longer depends directly on A.  This might be what David meant about type
signatures.  Given this, I would be inclined to leave the recompilation
strategy at transitive closure.  I'm happy to be proven wrong, of course.

This will work in the plugin, at it works in the version on my machine. In the compilation of B, the inferred type will appear in the signature of y, so a change in the type of x will trigger a recompilation of B, which then triggers a recompilation of C because the type of y changes. This is easy. There are cases in scalac that are broken though, e.g.,  assume A and B are in different files:
trait A { val x = "a" }class B extends A { val y = x }
Compile A and B, now change A so...trait A { val x = (new B).y }
The plugin will not detect an error between A  and B, and neither would David's dependency calculator. The bug is not detected unless A and B are always compiled together, and won't be detected when A and B are compiled separately. 
 

After you get a runtime error because of something like this, you either: a)
move on b) you run the equivalent of clean before each compile you want to
have confidence c) go write some code to try to keep it from happening again.
It looks like I'm not the only one to have done a-c.

I don't understand, what runtime error are you getting? The above code compiles fine in the plugin, can you be more specific?

On Tue, Feb 10, 2009 at 7:40 AM, Sean McDirmid <sean.mcdirmid@gmail.com> wrote:

On Tue, Feb 10, 2009 at 12:14 AM, Mark Harrah <harrah@bu.edu> wrote:

My second comment is that I think that it is very important to do as good a
job at proper recompilation as possible in as many cases as possible and to
put this before speed.  

This is a noble statement but not very pragmatic. To be 100% safe you have to recompile the world on each change. To be 90% safe you can do something like transitive closure recompilation. And so on...It just turns out that the 90% option is about as slow as the 100% option on many projects (because anything involving transitive closure in Scala is going to choke a lot of projects). 

I want to add that for a large project where recompile all takes 5-10 minutes, speed is definitely very important. You always want correctness, but speed can be just as important in determining whether something is useful or not. 

trait A { val x = "a" }class B extends A { val y = x }
Compile A and B, now change A so...trait A { val x = (new B).y }
The plugin will not detect an error between A  and B, and neither would David's dependency calculator. The bug is not detected unless A and B are always compiled together, and won't be detected when A and B are compiled separately. 

How is David's dependency calculator not compiling A and B together?  If not, perhaps here is an issue with how scalac creates the "depends" attribute on a compilation unit, rather than an issue with David's code.

Any bug from scalac not recompiling correctly is wasted time for me, and I'd rather wait through one build then two.  In fact, before you realize what's happening, you may spend time "debugging the issue" before you figure out what happened.  I'd rather wait the 5-10 minutes than 10-20 minutes of a failed build, analysis and recompile all.  It's thoughts like this that make me unable to "sell" scala to my current company.  If scala hopes to get out of the academic business, I seriously hope more people change their attitudes.   Oour customers want something that works right the first time, not something fast that breaks down... (Speaking of failing car companies...)   I'd hope that scala would choose correctness first, then speed.

This functionality *does not exist in scalac*, and therefore is already only improving speed for users of scalac.  IF the eclipse plugin has a better solution *GREAT*, don't use the provided default in scalac.  If you believe scalac should be fast, perhaps an option that allows you to use the "dangerous" dependency analysis would be a much better solution (but once aagain not the default)


Also I'm moving this conversation to scala-debate.

trait A { val x = "a" }class B extends A { val y = x }
Compile A and B, now change A so...trait A { val x = (new B).y }
The plugin will not detect an error between A  and B, and neither would David's dependency calculator. The bug is not detected unless A and B are always compiled together, and won't be detected when A and B are compiled separately. 

How is David's dependency calculator not compiling A and B together?  If not, perhaps here is an issue with how scalac creates the "depends" attribute on a compilation unit, rather than an issue with David's code.

Any bug from scalac not recompiling correctly is wasted time for me, and I'd rather wait through one build then two.  In fact, before you realize what's happening, you may spend time "debugging the issue" before you figure out what happened.  I'd rather wait the 5-10 minutes than 10-20 minutes of a failed build, analysis and recompile all.  It's thoughts like this that make me unable to "sell" scala to my current company.  If scala hopes to get out of the academic business, I seriously hope more people change their attitudes.   Oour customers want something that works right the first time, not something fast that breaks down... (Speaking of failing car companies...)   I'd hope that scala would choose correctness first, then speed.

This functionality *does not exist in scalac*, and therefore is already only improving speed for users of scalac.  IF the eclipse plugin has a better solution *GREAT*, don't use the provided default in scalac.  If you believe scalac should be fast, perhaps an option that allows you to use the "dangerous" dependency analysis would be a much better solution (but once aagain not the default)


Also I'm moving this conversation to scala-debate.

I'm not on scala-debate so this reply will bounce there. 

On Tue, Feb 10, 2009 at 9:30 AM, Josh Suereth <joshua.suereth@gmail.com> wrote:

How is David's dependency calculator not compiling A and B together?  If not, perhaps here is an issue with how scalac creates the "depends" attribute on a compilation unit, rather than an issue with David's code.

I don't think David's dependency calculator will eagerly compile things together, and that it would wait to detect that A's new and old class files are different before recompiling B. The problem is really with the way scalac detects cycles in a way that doesn't support separate compilation. The problem is pretty hypothetical (e.g., it won't happen accidentally in most cases), so its not a high priority to fix it.   

Any bug from scalac not recompiling correctly is wasted time for me, and I'd rather wait through one build then two.  In fact, before you realize what's happening, you may spend time "debugging the issue" before you figure out what happened.  I'd rather wait the 5-10 minutes than 10-20 minutes of a failed build, analysis and recompile all.  It's thoughts like this that make me unable to "sell" scala to my current company.  If scala hopes to get out of the academic business, I seriously hope more people change their attitudes.   Oour customers want something that works right the first time, not something fast that breaks down... (Speaking of failing car companies...)   I'd hope that scala would choose correctness first, then speed.

If you can stomache a long 5 minute build on each change then you are very lucky. As for me, I need correctness and performance in the build process, otherwise Scala is useless and I'll use C#. In an IDE, you are locked out until the compile is finished, you can still edit files, but certain operations will lock you out until the compile is done. If the compile last 5 minutes when it only needed to last 5 seconds, its a huge usability problem. Definitely, correctness is important, but its not the only thing that is important. The problem is, we need correctness and speed, at least in the IDE.   

This functionality *does not exist in scalac*, and therefore is already only improving speed for users of scalac.  IF the eclipse plugin has a better solution *GREAT*, don't use the provided default in scalac.  If you believe scalac should be fast, perhaps an option that allows you to use the "dangerous" dependency analysis would be a much better solution (but once aagain not the default)

Slippery slope. The problem eventually degrades into should scalac be useful or useless. If scalac took 2 hours to compile a file just to be safe, obviously no one would bother using it. If Martin didn't add all the dangerous performance optimizations in scalac that create bugs, scalac would be too slow to use and therefore useless. Better to have a fast buggy scalac (bugs can be fixed BTW) then a slow perfect scalac. I would rather we focus efforts on getting dangerous dependency analysis fixed to work correctly rather than starting at the other end of getting safe perfect dependency analysis to work with adequate performance. But this is up to whoever writes the code.
BTW, the dependency calculator in the IDE was supposed to be a part of scalac but never got there. The infrastructure to track dependencies does exist in the compiler though, it just needs to be plugged into.   

+1
One of the most frustrating aspects of working with Scala is the build, realize something's broken with the build, clean, and build again cycle.  IMHO correctness trumps speed every time.

On Mon, Feb 9, 2009 at 8:30 PM, Josh Suereth <joshua.suereth@gmail.com> wrote:

trait A { val x = "a" }class B extends A { val y = x }
Compile A and B, now change A so...trait A { val x = (new B).y }
The plugin will not detect an error between A  and B, and neither would David's dependency calculator. The bug is not detected unless A and B are always compiled together, and won't be detected when A and B are compiled separately. 

How is David's dependency calculator not compiling A and B together?  If not, perhaps here is an issue with how scalac creates the "depends" attribute on a compilation unit, rather than an issue with David's code.

Any bug from scalac not recompiling correctly is wasted time for me, and I'd rather wait through one build then two.  In fact, before you realize what's happening, you may spend time "debugging the issue" before you figure out what happened.  I'd rather wait the 5-10 minutes than 10-20 minutes of a failed build, analysis and recompile all.  It's thoughts like this that make me unable to "sell" scala to my current company.  If scala hopes to get out of the academic business, I seriously hope more people change their attitudes.   Oour customers want something that works right the first time, not something fast that breaks down... (Speaking of failing car companies...)   I'd hope that scala would choose correctness first, then speed.

This functionality *does not exist in scalac*, and therefore is already only improving speed for users of scalac.  IF the eclipse plugin has a better solution *GREAT*, don't use the provided default in scalac.  If you believe scalac should be fast, perhaps an option that allows you to use the "dangerous" dependency analysis would be a much better solution (but once aagain not the default)


Also I'm moving this conversation to scala-debate.

--
http://erikengbrecht.blogspot.com/

+1
One of the most frustrating aspects of working with Scala is the build, realize something's broken with the build, clean, and build again cycle.  IMHO correctness trumps speed every time.

On Mon, Feb 9, 2009 at 8:30 PM, Josh Suereth <joshua.suereth@gmail.com> wrote:

trait A { val x = "a" }class B extends A { val y = x }
Compile A and B, now change A so...trait A { val x = (new B).y }
The plugin will not detect an error between A  and B, and neither would David's dependency calculator. The bug is not detected unless A and B are always compiled together, and won't be detected when A and B are compiled separately. 

How is David's dependency calculator not compiling A and B together?  If not, perhaps here is an issue with how scalac creates the "depends" attribute on a compilation unit, rather than an issue with David's code.

Any bug from scalac not recompiling correctly is wasted time for me, and I'd rather wait through one build then two.  In fact, before you realize what's happening, you may spend time "debugging the issue" before you figure out what happened.  I'd rather wait the 5-10 minutes than 10-20 minutes of a failed build, analysis and recompile all.  It's thoughts like this that make me unable to "sell" scala to my current company.  If scala hopes to get out of the academic business, I seriously hope more people change their attitudes.   Oour customers want something that works right the first time, not something fast that breaks down... (Speaking of failing car companies...)   I'd hope that scala would choose correctness first, then speed.

This functionality *does not exist in scalac*, and therefore is already only improving speed for users of scalac.  IF the eclipse plugin has a better solution *GREAT*, don't use the provided default in scalac.  If you believe scalac should be fast, perhaps an option that allows you to use the "dangerous" dependency analysis would be a much better solution (but once aagain not the default)


Also I'm moving this conversation to scala-debate.

--
http://erikengbrecht.blogspot.com/

On Tue, Feb 10, 2009 at 1:30 AM, Josh Suereth wrote:
> IF the eclipse plugin has a better solution *GREAT*, don't use the
> provided default in scalac.

It doesn't, but even if it did it would be highly desirable to have
the plugin share an actively maintained codebase with scalac.

Cheers,

Miles

On Tue, Feb 10, 2009 at 1:30 AM, Josh Suereth wrote:
> IF the eclipse plugin has a better solution *GREAT*, don't use the
> provided default in scalac.

It doesn't, but even if it did it would be highly desirable to have
the plugin share an actively maintained codebase with scalac.

Cheers,

Miles

It does. Miles, if you are so sure that dependency tracking is broken in the plugin, just provide me with a bug report. 

On Tue, Feb 10, 2009 at 9:44 AM, Miles Sabin <miles@milessabin.com> wrote:

On Tue, Feb 10, 2009 at 1:30 AM, Josh Suereth <joshua.suereth@gmail.com> wrote:
> IF the eclipse plugin has a better solution *GREAT*, don't use the
> provided default in scalac.

It doesn't, but even if it did it would be highly desirable to have
the plugin share an actively maintained codebase with scalac.

Cheers,


Miles

--
Miles Sabin
tel:    +44 (0)1273 720 779
mobile: +44 (0)7813 944 528
skype:  milessabin

Ah! I understand. 
So you won't even wait to see if A has changed or not, and you'll aggressively recompile the transitive closure in bulk? So if A changes, then B will automatically be recompiled in the same run as A even if B hasn't change?
Ok, in that case you won't run into separate compilation problems. My approach totally depends on separate compilation working in scalac, which is mostly true (though as mentioned there are a few corner cases where it doesn't work). 

On Tue, Feb 10, 2009 at 9:50 AM, David MacIver <david.maciver@gmail.com> wrote:

Of course it will. It might have difficulties given some of the refinements I've been thinking about to optimise the process, but there's a reason I've not implemented them yet - they're all precarious for this sort of reason.

I think you're under the impression that the way it currently works is as follows:

A.scala has changed. Let's recompile it.

(we recompile it)

Oh, hey. B.scala depends on A.scala and we recompiled A.scala. Let's recompile that.

(we compile it).

This isn't the case. The way it actually works is you pass it both A.scala and B.scala and it determines which need to be recompiled and doesn't do anything to the remainder.
 

The bug is not detected unless A and B are always compiled together, and won't be detected when A and B are compiled separately. 

Which is a large part of why the way my approach is currently implemented recompiles all necessary source files in bulk.

Huh? In this case:
// A.scalatrait A { val x = "hello" }// B.scalaclass B extends A { val y = x }

There is no cyclic dependency. Its only when A is changed as: // A.scalatrait A { val x = (new B).y }
that the cyclic dependency is introduced, but you don't know that before you recompile A, so you don't know that B needs to be recompiled with A until after you've recompiled A.  
I have a strong hunch that your system will also fail to detect this problem. (Unless you are going recompile A then recompile A and B.) 
On Tue, Feb 10, 2009 at 9:57 AM, David MacIver <david.maciver@gmail.com> wrote:

Well, mine also depends on separate compilation working in scalac. It's just that it only assumes it for non-cyclic dependencies (because it's only actually true for non-cyclic dependencies). 

On Mon, Feb 9, 2009 at 11:40 PM, Sean McDirmid <sean.mcdirmid@gmail.com> wrote:

trait A { val x = "a" }
trait B extends A { val y = x }
trait C extends B { val z = y.length }

C no longer depends directly on A.  This might be what David meant about type
signatures.  Given this, I would be inclined to leave the recompilation
strategy at transitive closure.  I'm happy to be proven wrong, of course.

This will work in the plugin, at it works in the version on my machine. In the compilation of B, the inferred type will appear in the signature of y, so a change in the type of x will trigger a recompilation of B, which then triggers a recompilation of C because the type of y changes. This is easy. There are cases in scalac that are broken though, e.g.,  assume A and B are in different files:
trait A { val x = "a" }class B extends A { val y = x }
Compile A and B, now change A so...trait A { val x = (new B).y }
The plugin will not detect an error between A  and B, and neither would David's dependency calculator.

Of course it will. It might have difficulties given some of the refinements I've been thinking about to optimise the process, but there's a reason I've not implemented them yet - they're all precarious for this sort of reason.

I think you're under the impression that the way it currently works is as follows:

A.scala has changed. Let's recompile it.

(we recompile it)

Oh, hey. B.scala depends on A.scala and we recompiled A.scala. Let's recompile that.

(we compile it).

This isn't the case. The way it actually works is you pass it both A.scala and B.scala and it determines which need to be recompiled and doesn't do anything to the remainder.
 

The bug is not detected unless A and B are always compiled together, and won't be detected when A and B are compiled separately. 

Which is a large part of why the way my approach is currently implemented recompiles all necessary source files in bulk.

On Tue, Feb 10, 2009 at 1:55 AM, Sean McDirmid <sean.mcdirmid@gmail.com> wrote:

Ah! I understand. 
So you won't even wait to see if A has changed or not, and you'll aggressively recompile the transitive closure in bulk? So if A changes, then B will automatically be recompiled in the same run as A even if B hasn't change?

Yes.
 

Ok, in that case you won't run into separate compilation problems. My approach totally depends on separate compilation working in scalac, which is mostly true (though as mentioned there are a few corner cases where it doesn't work). 

Well, mine also depends on separate compilation working in scalac. It's just that it only assumes it for non-cyclic dependencies (because it's only actually true for non-cyclic dependencies).

This works fine. B extends A and so it will get recompiled whenever A gets
recompiled.

-Mark

On Monday 09 February 2009, Sean McDirmid wrote:
> Huh? In this case:
> // A.scala
> trait A { val x = "hello" }
> // B.scala
> class B extends A { val y = x }
>
> There is no cyclic dependency. Its only when A is changed as:
> // A.scala
> trait A { val x = (new B).y }
>
> that the cyclic dependency is introduced, but you don't know that before
> you recompile A, so you don't know that B needs to be recompiled with A
> until after you've recompiled A.
>
> I have a strong hunch that your system will also fail to detect this
> problem. (Unless you are going recompile A then recompile A and B.)
>
> On Tue, Feb 10, 2009 at 9:57 AM, David MacIver
wrote:
> > Well, mine also depends on separate compilation working in scalac. It's
> > just that it only assumes it for non-cyclic dependencies (because it's
> > only actually true for non-cyclic dependencies).

Oops, your right. So whether or not a cyclic dependency is introduced during compilation isn't important since the dependency already exists downstream. Ok, David's approach seems correct when scalac doesn't support the separate compilation of cyclic dependencies. 
On Tue, Feb 10, 2009 at 10:53 AM, Mark Harrah <harrah@bu.edu> wrote:

This works fine.  B extends A and so it will get recompiled whenever A gets
recompiled.

-Mark

On Monday 09 February 2009, Sean McDirmid wrote:
> Huh? In this case:
> // A.scala
> trait A { val x = "hello" }
> // B.scala
> class B extends A { val y = x }
>
> There is no cyclic dependency. Its only when A is changed as:
> // A.scala
> trait A { val x = (new B).y }
>
> that the cyclic dependency is introduced, but you don't know that before
> you recompile A, so you don't know that B needs to be recompiled with A
> until after you've recompiled A.
>
> I have a strong hunch that your system will also fail to detect this
> problem. (Unless you are going recompile A then recompile A and B.)
>
> On Tue, Feb 10, 2009 at 9:57 AM, David MacIver
<david.maciver@gmail.com>wrote:
> > Well, mine also depends on separate compilation working in scalac. It's
> > just that it only assumes it for non-cyclic dependencies (because it's
> > only actually true for non-cyclic dependencies).

On Mon, Feb 9, 2009 at 10:09 PM, Windemuth Andreas wrote:
>
> To refresh our memories, below is my initial monologue on the byte picklers.
> It is only fair to say that after I sent off the cricket bit I did get a few
> very helpful responses, including from David and Philipp Haller, the
> original author. I played with thoughts of tearing them out and replacing
> them by Sbinary, but in the end I just fixed them and have used them happily
> ever after. I still think they are great.
>
> If they are indeed still part of the compiler, I think they should be fixed,
> definitely. Let me know off-line where to send patches.

They were never part of the compiler. Scalac uses some custom code to
read/write type information (which is called pickles, causing
confusion). I doubt that code is slowing down anything. But patches
for the standard library are most welcome. If there's no ticket for
this issue, please open one and attach your patch.

cheers,
iulian

The issue already seems to be fixed. I think it was dealt with back
when this was originally discussed.

On 2/12/09, Iulian Dragos wrote:
> On Mon, Feb 9, 2009 at 10:09 PM, Windemuth Andreas
> wrote:
>>
>> To refresh our memories, below is my initial monologue on the byte
>> picklers.
>> It is only fair to say that after I sent off the cricket bit I did get a
>> few
>> very helpful responses, including from David and Philipp Haller, the
>> original author. I played with thoughts of tearing them out and replacing
>> them by Sbinary, but in the end I just fixed them and have used them
>> happily
>> ever after. I still think they are great.
>>
>> If they are indeed still part of the compiler, I think they should be
>> fixed,
>> definitely. Let me know off-line where to send patches.
>
> They were never part of the compiler. Scalac uses some custom code to
> read/write type information (which is called pickles, causing
> confusion). I doubt that code is slowing down anything. But patches
> for the standard library are most welcome. If there's no ticket for
> this issue, please open one and attach your patch.
>
> cheers,
> iulian
>
>
> --
> « Je déteste la montagne, ça cache le paysage »
> Alphonse Allais
>

David McIver wrote:
> The issue already seems to be fixed. I think it was dealt with back
> when this was originally discussed.

I do not think so. If this is the up-to-date source:

http://scala-tools.org/scaladocs/scala-library/2.7.1/io/BytePickle.scala...

then neither of the two problems has been fixed. As the following code illustrates:

1. def appP(n: Int, s: Array[Byte]): Array[Byte] =
2. Array.concat(s, nat2Bytes(n));
bytes are appended to the output by taking all of the existing stream and adding
a few bytes using Array.concat(). Array.concat() will copy the entire stream, and
pickling slows down to a crawl after the first few kilobytes have been pickled.

What I did is replace Array[Byte] with abstract traits ByteInStream and ByteOutStream as follows:

- def appP(n: Int, s: Array[Byte]): Array[Byte] =
- Array.concat(s, nat2Bytes(n));
+ def appP(n: Int, s: ByteOutStream): ByteOutStream = s ++ nat2Bytes(n)

- def appU(s: Array[Byte]): (Int, Array[Byte]) = {
+ def appU(s: ByteInStream): (Int, ByteInStream) = {

with

trait ByteInStream {
def apply(i: Int): Byte
def head(n: Int): Seq[Byte]
def tail(n: Int): ByteInStream
}

trait ByteOutStream {
def ++(b: Byte): ByteOutStream
def ++(ba: Seq[Byte]): ByteOutStream
}

These traits can then be implemented efficiently, and I have produced implementations
that wrap java.io's InputStream and OutputStream, so that no memory is wasted to hold
the byte stream. There are some issues with Mutability, but they have been solved. I have
used the modified library to read and write complex multi-megabyte files very efficiently.

Unfortunately this change needs to be made in pretty much every single method
defined in BytePickle.scala, which makes the patch unwieldy. Also, I have not taken
care to preserve the API, so there may be some code breaking unless some more
work is done. I am not even sure if the API is even well defined.

I will try and create a ticket for this, although it does not seem so urgent now that
it appears that I am the only serious user of the code.

Andreas

Suddenly, without any discernable cause, the plug-in is reliably crashing on me during
project build. I am trying to backtrack my changes to see if I can identify the offending
code. Obviously, I cannot provide an isolated example before then.

I am hoping that the below stack trace can ring some bells with a plugin coder. Please let
me know if you have any idea what could be causing this, so I have a better chance to work
around and/or provide an isolated test case.

This is on Eclipse version: 3.4.1, build id: M20080911-1700, with the 2.7.3-final plugin.

Andreas

java.lang.Error
at lampion.core.Nodes$class.assert(Nodes.scala:25)
at scala.tools.eclipse.Driver$Project.assert(Driver.scala:16)
at scala.tools.eclipse.ScalaPlugin$ProjectImpl$class.lampionToNSC(ScalaPlugin.scala:389)
at scala.tools.eclipse.Driver$Project.lampionToNSC(Driver.scala:16)
at scala.tools.eclipse.ScalaPlugin$ProjectImpl$$anonfun$build$1.apply(ScalaPlugin.scala:403)
at scala.tools.eclipse.ScalaPlugin$ProjectImpl$$anonfun$build$1.apply(ScalaPlugin.scala:402)
at scala.Iterator$class.foreach(Iterator.scala:414)
at scala.collection.jcl.MutableIterator$Wrapper.foreach(MutableIterator.scala:14)
at scala.Iterable$class.foreach(Iterable.scala:256)
at scala.collection.jcl.LinkedHashSet.foreach(LinkedHashSet.scala:18)
at scala.tools.eclipse.ScalaPlugin$ProjectImpl$class.build(ScalaPlugin.scala:402)
at scala.tools.eclipse.Driver$Project.build(Driver.scala:16)
at lampion.eclipse.Builder.build(Builder.scala:87)
at scala.tools.eclipse.Builder.build(Builder.scala:32)
at org.eclipse.core.internal.events.BuildManager$2.run(BuildManager.java:633)
at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:37)
at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:170)
at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:201)
at org.eclipse.core.internal.events.BuildManager$1.run(BuildManager.java:253)
at org.eclipse.core.runtime.SafeRunner.run(SafeRunner.java:37)
at org.eclipse.core.internal.events.BuildManager.basicBuild(BuildManager.java:256)
at org.eclipse.core.internal.events.BuildManager.basicBuildLoop(BuildManager.java:309)
at org.eclipse.core.internal.events.BuildManager.build(BuildManager.java:341)
at org.eclipse.core.internal.events.AutoBuildJob.doBuild(AutoBuildJob.java:140)
at org.eclipse.core.internal.events.AutoBuildJob.run(AutoBuildJob.java:238)
at org.eclipse.core.internal.jobs.Worker.run(Worker.java:55)

On Fri, Feb 13, 2009 at 4:15 PM, Windemuth Andreas wrote:
> Suddenly, without any discernable cause, the plug-in is reliably crashing on me during
> project build. I am trying to backtrack my changes to see if I can identify the offending
> code. Obviously, I cannot provide an isolated example before then.
>
> I am hoping that the below stack trace can ring some bells with a plugin coder. Please let
> me know if you have any idea what could be causing this, so I have a better chance to work
> around and/or provide an isolated test case.
>
> This is on Eclipse version: 3.4.1, build id: M20080911-1700, with the 2.7.3-final plugin.

Most likely your workspace needs a refresh. I suggest,

* Turn off automatic building

* Refresh

* Turn automatic building back on

Let me know how you get on ...

Cheers,

Miles

I had tried refreshing, rebuilding and restarting eclipse several different ways without success. However, after undoing a few changes, and some more refreshing, restarting and rebuilding, the problem is gone. Even after putting the changes back in. Go figure. While it works, the plugin is great, and it works 95% of the time, for me.

I have not tried turning off automatic building. Will try that if I run into problems again.

Thanks,

Andreas

Go, plugin!