Tracking changed files for recompilation

Most of the traditional build tools for scala have a bit of a problem: It's really hard to map class files to their source files and determine which ones need recompiling. It gets even harder if you want to recompile dependent files. The solution taken by ant is dramatically inadequate in that it assumes you follow a Java file convention and doesn't recompile dependent files.

It turns out that it's not very hard to write extensions to scalac which have sensible behaviour for recompiling changed files. Unfortunately it does seem to have to be an extension - as far as I can tell it's not possible to do as a plugin because you need to be able to cancel the compilation of unchanged source files. Mark Harrah's http://code.google.com/p/simple-build-tool/ does this for example and it works rather well. I have some code which does similar, but I never really went anywhere with it (though Josh Suereth has taken it up as part of his scalac-dependency-plugin project).

It would be nice to get similar behaviour to this integrated into scalac so that it does this by default: It would make all tools get the benefit of it for free, and it would make the compile times for scalac much more manageable. Frankly it would also make developing scalac much nicer - at the moment it's painfully easy to get into a state where things are broken because they were compiled against an older classfile which has since been replaced.

I'm happy to volunteer to implement this if I get the go ahead. I don't think it would be very hard at all, and it would make everyones' lives easier.

+1 (if I'm allowed any say)

On Sun, Feb 8, 2009 at 10:09 AM, David MacIver <david.maciver@gmail.com> wrote:

Most of the traditional build tools for scala have a bit of a problem: It's really hard to map class files to their source files and determine which ones need recompiling. It gets even harder if you want to recompile dependent files. The solution taken by ant is dramatically inadequate in that it assumes you follow a Java file convention and doesn't recompile dependent files.

It turns out that it's not very hard to write extensions to scalac which have sensible behaviour for recompiling changed files. Unfortunately it does seem to have to be an extension - as far as I can tell it's not possible to do as a plugin because you need to be able to cancel the compilation of unchanged source files. Mark Harrah's http://code.google.com/p/simple-build-tool/ does this for example and it works rather well. I have some code which does similar, but I never really went anywhere with it (though Josh Suereth has taken it up as part of his scalac-dependency-plugin project).

It would be nice to get similar behaviour to this integrated into scalac so that it does this by default: It would make all tools get the benefit of it for free, and it would make the compile times for scalac much more manageable. Frankly it would also make developing scalac much nicer - at the moment it's painfully easy to get into a state where things are broken because they were compiled against an older classfile which has since been replaced.

I'm happy to volunteer to implement this if I get the go ahead. I don't think it would be very hard at all, and it would make everyones' lives easier.

On Sun, Feb 8, 2009 at 3:09 PM, David MacIver wrote:
> I'm happy to volunteer to implement this if I get the go ahead. I don't
> think it would be very hard at all, and it would make everyones' lives
> easier.

Yes, please do. While you're digging around in that sort of area, it'd
be interesting to know if,

http://lampsvn.epfl.ch/trac/scala/ticket/683

of its fix is at all related.

Cheers,

Miles

Most of the traditional build tools for scala have a bit of a problem: It's really hard to map class files to their source files and determine which ones need recompiling. It gets even harder if you want to recompile dependent files. The solution taken by ant is dramatically inadequate in that it assumes you follow a Java file convention and doesn't recompile dependent files.

It turns out that it's not very hard to write extensions to scalac which have sensible behaviour for recompiling changed files. Unfortunately it does seem to have to be an extension - as far as I can tell it's not possible to do as a plugin because you need to be able to cancel the compilation of unchanged source files. Mark Harrah's http://code.google.com/p/simple-build-tool/ does this for example and it works rather well. I have some code which does similar, but I never really went anywhere with it (though Josh Suereth has taken it up as part of his scalac-dependency-plugin project).

It would be nice to get similar behaviour to this integrated into scalac so that it does this by default: It would make all tools get the benefit of it for free, and it would make the compile times for scalac much more manageable. Frankly it would also make developing scalac much nicer - at the moment it's painfully easy to get into a state where things are broken because they were compiled against an older classfile which has since been replaced.

I'm happy to volunteer to implement this if I get the go ahead. I don't think it would be very hard at all, and it would make everyones' lives easier.

He didn't specify who could give him the go ahead, and he did post to an open list....  It rather sounded like something I would hear at work:  Hi, I have a great idea.  It would benefit the entire world, wouldn't take much time to implement, and I will do it myself.  Will someone [with someone not being directed at a specific individual or decision maker] give me the go ahead to do it?
+1 - You have the go ahead.

On Sun, Feb 8, 2009 at 10:47 AM, Josh Suereth <joshua.suereth@gmail.com> wrote:

+1 (if I'm allowed any say)

On Sun, Feb 8, 2009 at 10:09 AM, David MacIver <david.maciver@gmail.com> wrote:

Most of the traditional build tools for scala have a bit of a problem: It's really hard to map class files to their source files and determine which ones need recompiling. It gets even harder if you want to recompile dependent files. The solution taken by ant is dramatically inadequate in that it assumes you follow a Java file convention and doesn't recompile dependent files.

It turns out that it's not very hard to write extensions to scalac which have sensible behaviour for recompiling changed files. Unfortunately it does seem to have to be an extension - as far as I can tell it's not possible to do as a plugin because you need to be able to cancel the compilation of unchanged source files. Mark Harrah's http://code.google.com/p/simple-build-tool/ does this for example and it works rather well. I have some code which does similar, but I never really went anywhere with it (though Josh Suereth has taken it up as part of his scalac-dependency-plugin project).

It would be nice to get similar behaviour to this integrated into scalac so that it does this by default: It would make all tools get the benefit of it for free, and it would make the compile times for scalac much more manageable. Frankly it would also make developing scalac much nicer - at the moment it's painfully easy to get into a state where things are broken because they were compiled against an older classfile which has since been replaced.

I'm happy to volunteer to implement this if I get the go ahead. I don't think it would be very hard at all, and it would make everyones' lives easier.

--
http://erikengbrecht.blogspot.com/

Heh. Sorry, what happened there was that this was originally going to scala-internals with the assumption that it would mostly be compiler committers reading this, but I changed it at the last minute to go to scala-tools instead on grounds that it seemed more appropriate there.

On Sun, Feb 8, 2009 at 3:56 PM, Erik Engbrecht <erik.engbrecht@gmail.com> wrote:

He didn't specify who could give him the go ahead, and he did post to an open list....  It rather sounded like something I would hear at work:  Hi, I have a great idea.  It would benefit the entire world, wouldn't take much time to implement, and I will do it myself.  Will someone [with someone not being directed at a specific individual or decision maker] give me the go ahead to do it?
+1 - You have the go ahead.

On Sun, Feb 8, 2009 at 10:47 AM, Josh Suereth <joshua.suereth@gmail.com> wrote:

+1 (if I'm allowed any say)

On Sun, Feb 8, 2009 at 10:09 AM, David MacIver <david.maciver@gmail.com> wrote:

Most of the traditional build tools for scala have a bit of a problem: It's really hard to map class files to their source files and determine which ones need recompiling. It gets even harder if you want to recompile dependent files. The solution taken by ant is dramatically inadequate in that it assumes you follow a Java file convention and doesn't recompile dependent files.

It turns out that it's not very hard to write extensions to scalac which have sensible behaviour for recompiling changed files. Unfortunately it does seem to have to be an extension - as far as I can tell it's not possible to do as a plugin because you need to be able to cancel the compilation of unchanged source files. Mark Harrah's http://code.google.com/p/simple-build-tool/ does this for example and it works rather well. I have some code which does similar, but I never really went anywhere with it (though Josh Suereth has taken it up as part of his scalac-dependency-plugin project).

It would be nice to get similar behaviour to this integrated into scalac so that it does this by default: It would make all tools get the benefit of it for free, and it would make the compile times for scalac much more manageable. Frankly it would also make developing scalac much nicer - at the moment it's painfully easy to get into a state where things are broken because they were compiled against an older classfile which has since been replaced.

I'm happy to volunteer to implement this if I get the go ahead. I don't think it would be very hard at all, and it would make everyones' lives easier.

--
http://erikengbrecht.blogspot.com/

Ok. I'm not committing this to trunk just yet because I want to put it through the review board, but here's a patch.

Defined behaviour:

We add two new compiler flags. -compileall and -dependenciesfile. The first turns off the dependency tracking behaviour. The second allows you to specify a different file. By default dependency tracking is on and the dependencies file is .scala_dependencies in the current directory.

When dependency tracking is on, dependencies between source and class files will be tracked and only files which have changed or which depend on files which have changed will be recompiled (this includes transitively - if A depends on B and B is recompiled, A will be). This happens irrespective of any naming or source convention you might choose to follow - no correspondence between class and source names is assumed.

That's all. Have fun. :-)

David

On Sun, Feb 8, 2009 at 4:28 PM, David MacIver <david.maciver@gmail.com> wrote:

Heh. Sorry, what happened there was that this was originally going to scala-internals with the assumption that it would mostly be compiler committers reading this, but I changed it at the last minute to go to scala-tools instead on grounds that it seemed more appropriate there.

On Sun, Feb 8, 2009 at 3:56 PM, Erik Engbrecht <erik.engbrecht@gmail.com> wrote:

He didn't specify who could give him the go ahead, and he did post to an open list....  It rather sounded like something I would hear at work:  Hi, I have a great idea.  It would benefit the entire world, wouldn't take much time to implement, and I will do it myself.  Will someone [with someone not being directed at a specific individual or decision maker] give me the go ahead to do it?
+1 - You have the go ahead.

On Sun, Feb 8, 2009 at 10:47 AM, Josh Suereth <joshua.suereth@gmail.com> wrote:

+1 (if I'm allowed any say)

On Sun, Feb 8, 2009 at 10:09 AM, David MacIver <david.maciver@gmail.com> wrote:

Most of the traditional build tools for scala have a bit of a problem: It's really hard to map class files to their source files and determine which ones need recompiling. It gets even harder if you want to recompile dependent files. The solution taken by ant is dramatically inadequate in that it assumes you follow a Java file convention and doesn't recompile dependent files.

It turns out that it's not very hard to write extensions to scalac which have sensible behaviour for recompiling changed files. Unfortunately it does seem to have to be an extension - as far as I can tell it's not possible to do as a plugin because you need to be able to cancel the compilation of unchanged source files. Mark Harrah's http://code.google.com/p/simple-build-tool/ does this for example and it works rather well. I have some code which does similar, but I never really went anywhere with it (though Josh Suereth has taken it up as part of his scalac-dependency-plugin project).

It would be nice to get similar behaviour to this integrated into scalac so that it does this by default: It would make all tools get the benefit of it for free, and it would make the compile times for scalac much more manageable. Frankly it would also make developing scalac much nicer - at the moment it's painfully easy to get into a state where things are broken because they were compiled against an older classfile which has since been replaced.

I'm happy to volunteer to implement this if I get the go ahead. I don't think it would be very hard at all, and it would make everyones' lives easier.

--
http://erikengbrecht.blogspot.com/

On Sun, Feb 8, 2009 at 8:13 PM, David MacIver wrote:
> We add two new compiler flags. -compileall and -dependenciesfile. The first
> turns off the dependency tracking behaviour. The second allows you to
> specify a different file. By default dependency tracking is on and the
> dependencies file is .scala_dependencies in the current directory.

Hmm ... this is great, but is there any chance that we could have an
API which doesn't commit to files on the filesystem?

Cheers,

Miles

Impressive stuff!  I like the fact that it's all handled within the compiler itself, no need to track the dependency graph externally.  If this is accepted into the trunk/, it should mean that every build tool will reap the benefits right out of the box.

Daniel

On Sun, Feb 8, 2009 at 2:13 PM, David MacIver <david.maciver@gmail.com> wrote:

Ok. I'm not committing this to trunk just yet because I want to put it through the review board, but here's a patch.

Defined behaviour:

We add two new compiler flags. -compileall and -dependenciesfile. The first turns off the dependency tracking behaviour. The second allows you to specify a different file. By default dependency tracking is on and the dependencies file is .scala_dependencies in the current directory.

When dependency tracking is on, dependencies between source and class files will be tracked and only files which have changed or which depend on files which have changed will be recompiled (this includes transitively - if A depends on B and B is recompiled, A will be). This happens irrespective of any naming or source convention you might choose to follow - no correspondence between class and source names is assumed.

That's all. Have fun. :-)

David

On Sun, Feb 8, 2009 at 4:28 PM, David MacIver <david.maciver@gmail.com> wrote:

Heh. Sorry, what happened there was that this was originally going to scala-internals with the assumption that it would mostly be compiler committers reading this, but I changed it at the last minute to go to scala-tools instead on grounds that it seemed more appropriate there.

On Sun, Feb 8, 2009 at 3:56 PM, Erik Engbrecht <erik.engbrecht@gmail.com> wrote:

He didn't specify who could give him the go ahead, and he did post to an open list....  It rather sounded like something I would hear at work:  Hi, I have a great idea.  It would benefit the entire world, wouldn't take much time to implement, and I will do it myself.  Will someone [with someone not being directed at a specific individual or decision maker] give me the go ahead to do it?
+1 - You have the go ahead.

On Sun, Feb 8, 2009 at 10:47 AM, Josh Suereth <joshua.suereth@gmail.com> wrote:

+1 (if I'm allowed any say)

On Sun, Feb 8, 2009 at 10:09 AM, David MacIver <david.maciver@gmail.com> wrote:

Most of the traditional build tools for scala have a bit of a problem: It's really hard to map class files to their source files and determine which ones need recompiling. It gets even harder if you want to recompile dependent files. The solution taken by ant is dramatically inadequate in that it assumes you follow a Java file convention and doesn't recompile dependent files.

It turns out that it's not very hard to write extensions to scalac which have sensible behaviour for recompiling changed files. Unfortunately it does seem to have to be an extension - as far as I can tell it's not possible to do as a plugin because you need to be able to cancel the compilation of unchanged source files. Mark Harrah's http://code.google.com/p/simple-build-tool/ does this for example and it works rather well. I have some code which does similar, but I never really went anywhere with it (though Josh Suereth has taken it up as part of his scalac-dependency-plugin project).

It would be nice to get similar behaviour to this integrated into scalac so that it does this by default: It would make all tools get the benefit of it for free, and it would make the compile times for scalac much more manageable. Frankly it would also make developing scalac much nicer - at the moment it's painfully easy to get into a state where things are broken because they were compiled against an older classfile which has since been replaced.

I'm happy to volunteer to implement this if I get the go ahead. I don't think it would be very hard at all, and it would make everyones' lives easier.

--
http://erikengbrecht.blogspot.com/

On Sun, Feb 8, 2009 at 8:13 PM, David MacIver <david.maciver@gmail.com> wrote:
> We add two new compiler flags. -compileall and -dependenciesfile. The first
> turns off the dependency tracking behaviour. The second allows you to
> specify a different file. By default dependency tracking is on and the
> dependencies file is .scala_dependencies in the current directory.

Hmm ... this is great, but is there any chance that we could have an
API which doesn't commit to files on the filesystem?

On Sun, Feb 8, 2009 at 10:32 PM, Miles Sabin <miles@milessabin.com> wrote:

On Sun, Feb 8, 2009 at 8:13 PM, David MacIver <david.maciver@gmail.com> wrote:
> We add two new compiler flags. -compileall and -dependenciesfile. The first
> turns off the dependency tracking behaviour. The second allows you to
> specify a different file. By default dependency tracking is on and the
> dependencies file is .scala_dependencies in the current directory.

Hmm ... this is great, but is there any chance that we could have an
API which doesn't commit to files on the filesystem?

Sure. I'll think about how the details should work, but 95% of it should be there already.

How is this related to the dependency tracker we already have in the plugin which already generates a depends file?

Also, you've got to be very careful here, I've been burned more than once by traits (you have to recompile done the entire trait change when a super trait changes) given the way traits are compiled. 

On Sun, Feb 8, 2009 at 11:24 PM, Sean McDirmid <sean.mcdirmid@gmail.com> wrote:

How is this related to the dependency tracker we already have in the plugin which already generates a depends file?

The two aren't related.
 

Also, you've got to be very careful here, I've been burned more than once by traits (you have to recompile done the entire trait change when a super trait changes) given the way traits are compiled. 

This should work fine. If a file is recompiled all files which use things from that file are also recompiled. So when the trait is recompiled so is anything that extends it (including things which extend it indirectly because the recompiles are propagated transitively).

This should work fine. If a file is recompiled all files which use things from that file are also recompiled. So when the trait is recompiled so is anything that extends it (including things which extend it indirectly because the recompiles are propagated transitively).

Actually, sorry, the plugin tries to be a bit smarter about this. If a trait changes, then its immediate dependencies are recompiled (the files that directly depend on its files) and its end point dependencies are recompiled (the files that contain CLASSES that extend the trait). This is because true transitive closure isn't necessary, and is really too expensive (especially when recompiling the trait heavy plugin): you only need to recompile the extending classes so Scalac can do its bridge/super magic. 
Transitive closure recompilation really sucks, I'm glad we wouldn't have to use it. When I had this on in the plugin development ground to a halt as I was just recompiling the world on every save. 

On Mon, Feb 9, 2009 at 7:45 AM, David MacIver <david.maciver@gmail.com> wrote:
 

This should work fine. If a file is recompiled all files which use things from that file are also recompiled. So when the trait is recompiled so is anything that extends it (including things which extend it indirectly because the recompiles are propagated transitively).

Perhaps this is deemed an acceptable tradeoff or perhaps I misunderstand, but
I think the following is a minimal example where this will fail:

Given the following traits in three separate files:

trait A { val x = "a" }
trait B extends A
trait C extends B { val z = x.length }

$ scalac *.scala

now change A to:

trait A { val x = 5 }

The strategy you describe would then only compile A and B, equivalent to

$ scalac A.scala B.scala

but this succeeds when it should produce the error you see when recompiling
all files:

$ scalac *.scala
C.scala:3: error: value length is not a member of Int
val z = x.length

-Mark

On Sunday 08 February 2009, Sean McDirmid wrote:
> Actually, sorry, the plugin tries to be a bit smarter about this. If a
> trait changes, then its immediate dependencies are recompiled (the files
> that directly depend on its files) and its end point dependencies are
> recompiled (the files that contain CLASSES that extend the trait). This is
> because true transitive closure isn't necessary, and is really too
> expensive (especially when recompiling the trait heavy plugin): you only
> need to recompile the extending classes so Scalac can do its bridge/super
> magic.
>
> Transitive closure recompilation really sucks, I'm glad we wouldn't have to
> use it. When I had this on in the plugin development ground to a halt as I
> was just recompiling the world on every save.
>
> On Mon, Feb 9, 2009 at 7:45 AM, David MacIver
wrote:
> > This should work fine. If a file is recompiled all files which use things
> > from that file are also recompiled. So when the trait is recompiled so is
> > anything that extends it (including things which extend it indirectly
> > because the recompiles are propagated transitively).

Perhaps this is deemed an acceptable tradeoff or perhaps I misunderstand, but
I think the following is a minimal example where this will fail:

Given the following traits in three separate files:

trait A { val x = "a" }
trait B extends A
trait C extends B { val z = x.length }

$ scalac *.scala

now change A to:

trait A { val x = 5 }

The strategy you describe would then only compile A and B, equivalent to

$ scalac A.scala B.scala

but this succeeds when it should produce the error you see when recompiling
all files:

$ scalac *.scala
C.scala:3: error: value length is not a member of Int
       val z = x.length

-Mark

On Sunday 08 February 2009, Sean McDirmid wrote:
> Actually, sorry, the plugin tries to be a bit smarter about this. If a
> trait changes, then its immediate dependencies are recompiled (the files
> that directly depend on its files) and its end point dependencies are
> recompiled (the files that contain CLASSES that extend the trait). This is
> because true transitive closure isn't necessary, and is really too
> expensive (especially when recompiling the trait heavy plugin): you only
> need to recompile the extending classes so Scalac can do its bridge/super
> magic.
>
> Transitive closure recompilation really sucks, I'm glad we wouldn't have to
> use it. When I had this on in the plugin development ground to a halt as I
> was just recompiling the world on every save.
>
> On Mon, Feb 9, 2009 at 7:45 AM, David MacIver
<david.maciver@gmail.com>wrote:
> > This should work fine. If a file is recompiled all files which use things
> > from that file are also recompiled. So when the trait is recompiled so is
> > anything that extends it (including things which extend it indirectly
> > because the recompiles are propagated transitively).

On Mon, Feb 9, 2009 at 3:08 AM, Josh Suereth wrote:
> If this is indeed 100% correct, then perhaps the code should be merged with
> what David has provided. I believe anything is a step up from nothing, and
> the eclipse plugin can ignore the changes to scalac, and continue to use its
> own method. I believe the Ant/Maven/Builder community can benefit greatly
> form this. (PROPS to SBT for already supporting it).

I think it would make sense for as many tools as possible to converge
on the same mechanism and all efforts go into improving and
maintaining that. It'd be great if anything of additional value in the
plugins dependency manager could be merged into David's work.

Cheers,

Miles

Actually, sorry, the plugin tries to be a bit smarter about this. If a trait changes, then its immediate dependencies are recompiled (the files that directly depend on its files) and its end point dependencies are recompiled (the files that contain CLASSES that extend the trait). This is because true transitive closure isn't necessary, and is really too expensive (especially when recompiling the trait heavy plugin): you only need to recompile the extending classes so Scalac can do its bridge/super magic. 
Transitive closure recompilation really sucks, I'm glad we wouldn't have to use it. When I had this on in the plugin development ground to a halt as I was just recompiling the world on every save. 

On Mon, Feb 9, 2009 at 7:45 AM, David MacIver <david.maciver@gmail.com> wrote:
 

This should work fine. If a file is recompiled all files which use things from that file are also recompiled. So when the trait is recompiled so is anything that extends it (including things which extend it indirectly because the recompiles are propagated transitively).

Yay for transitive dependencies.  I know of another good tool that does this; Could it be.... MAVEN?


I do have one question/statement:  To use this new functionality the maven-scala-plugin should simply pass *all* source files to scalac all the time.  Not to be devil's advocate, but what about things like "fsc" where there is a resident compiler?  David Bernard spent some time making a goal for the maven plugin that watches a directory for changes to

a file and passes that into scalac.   To keep this working, should I instead pass *all* files when *any* change is detected? 

Well, not to sound harsh, but I've never heard anyone have a positive thing to say about the reliability of this method. Recompiling transitive dependencies is guaranteed to work and is required for correctness in sufficiently many cases (changes of type signatures for example) that you can't correctly determine whether a full transitive recompile is needed without detailed analysis of the source files. Which, as far as I can tell, the eclipse plugin either doesn't do or gets wrong.

On Mon, Feb 9, 2009 at 2:18 AM, Sean McDirmid <sean.mcdirmid@gmail.com> wrote:

Actually, sorry, the plugin tries to be a bit smarter about this. If a trait changes, then its immediate dependencies are recompiled (the files that directly depend on its files) and its end point dependencies are recompiled (the files that contain CLASSES that extend the trait). This is because true transitive closure isn't necessary, and is really too expensive (especially when recompiling the trait heavy plugin): you only need to recompile the extending classes so Scalac can do its bridge/super magic. 
Transitive closure recompilation really sucks, I'm glad we wouldn't have to use it. When I had this on in the plugin development ground to a halt as I was just recompiling the world on every save. 

On Mon, Feb 9, 2009 at 7:45 AM, David MacIver <david.maciver@gmail.com> wrote:
 

This should work fine. If a file is recompiled all files which use things from that file are also recompiled. So when the trait is recompiled so is anything that extends it (including things which extend it indirectly because the recompiles are propagated transitively).

> We add two new compiler flags. -compileall and -dependenciesfile. The first
> turns off the dependency tracking behaviour. The second allows you to
> specify a different file. By default dependency tracking is on and the
> dependencies file is .scala_dependencies in the current directory.
>
> When dependency tracking is on, dependencies between source and class files
> will be tracked and only files which have changed or which depend on files
> which have changed will be recompiled (this includes transitively - if A
> depends on B and B is recompiled, A will be). This happens irrespective of
> any naming or source convention you might choose to follow - no
> correspondence between class and source names is assumed.

Hi David,

I'm not sure that the following properly converts nested classes to the right
file (line 303 of your patch):

settings.outdir.value / (name.toString.replace(".", "/") + ".class")

Also, I think you miss generating the class file for a companion object in
certain cases.

Finally, if this is going to make it into scalac, it should track all
dependencies scalac is capable of tracking, such as dependencies on class
files not in the output directory and jar files.

Can partest test this kind (modifying source files and verifying proper
compilation) of thing well?

Thanks,
Mark

> We add two new compiler flags. -compileall and -dependenciesfile. The first
> turns off the dependency tracking behaviour. The second allows you to
> specify a different file. By default dependency tracking is on and the
> dependencies file is .scala_dependencies in the current directory.
>
> When dependency tracking is on, dependencies between source and class files
> will be tracked and only files which have changed or which depend on files
> which have changed will be recompiled (this includes transitively - if A
> depends on B and B is recompiled, A will be). This happens irrespective of
> any naming or source convention you might choose to follow - no
> correspondence between class and source names is assumed.

Hi David,

I'm not sure that the following properly converts nested classes to the right
file (line 303 of your patch):

settings.outdir.value / (name.toString.replace(".", "/") + ".class")

Also, I think you miss generating the class file for a companion object in
certain cases.

Finally, if this is going to make it into scalac, it should track all
dependencies scalac is capable of tracking, such as dependencies on class
files not in the output directory and jar files.

Can partest test this kind (modifying source files and verifying proper
compilation) of thing well?

> >
> > I'm not sure that the following properly converts nested classes to the
> > right
> > file (line 303 of your patch):
> >
> > settings.outdir.value / (name.toString.replace(".", "/") + ".class")
>
> Hm. I'll look into it. Could you give a minimal example where you think it
> gets this wrong?

I believe it doesn't get nested/anonymous classes right. I'm pretty sure
there was a good reason for the code at the end of sbt.Analyzer.

>
> > Also, I think you miss generating the class file for a companion object
> > in certain cases.
>
> I'm not sure what you mean. scalac should be generating class files as
> normal.

Sometimes scalac generates an extra class file that isn't listed in
unit.icode. I think it is for a companion object.

>
> > Finally, if this is going to make it into scalac, it should track all
> > dependencies scalac is capable of tracking, such as dependencies on class
> > files not in the output directory and jar files.
>
> Yes. There are definitely refinements that could be applied still. I'm not
> planning to dump this in scalac and forget about it. :-) My goal is to just
> get a basic working proof of concept in there and then improve upon it.

Sure, when I added this in sbt, I had to add another file for external
dependencies, so I figured I'd mention it in case thinking about it now helps
you keep everything in one file.

>
> > Can partest test this kind (modifying source files and verifying proper
> > compilation) of thing well?
>
> No, unfortunately not. I'm having trouble figuring out a good way to test
> it. If you've any suggestions I'd love to hear them.

Well, for sbt I implemented a scripted test framework where you write scripts
like:

> compile [success]
$ copy-file changes/A.scala src/main/scala/A.scala [success]
> compile [failure]

You could write something similar, steal it (see sbt's project directory), or
use sbt for testing (probably requires some tweaks that I'm happy to help
with). One problem you might encounter is that last modified often only has
1 second resolution, which creates difficulties in automated testing like
this (the file doesn't actually look changed).

I've written more about this at
http://code.google.com/p/simple-build-tool/wiki/ChangeDetectionAndTesting.

Thanks,
Mark

> >
> > I'm not sure that the following properly converts nested classes to the
> > right
> > file (line 303 of your patch):
> >
> > settings.outdir.value / (name.toString.replace(".", "/") + ".class")
>
> Hm. I'll look into it. Could you give a minimal example where you think it
> gets this wrong?

I believe it doesn't get nested/anonymous classes right.  I'm pretty sure
there was a good reason for the code at the end of sbt.Analyzer.

> > Also, I think you miss generating the class file for a companion object
> > in certain cases.
>
> I'm not sure what you mean. scalac should be generating class files as
> normal.

Sometimes scalac generates an extra class file that isn't listed in
unit.icode.  I think it is for a companion object.

> > Finally, if this is going to make it into scalac, it should track all
> > dependencies scalac is capable of tracking, such as dependencies on class
> > files not in the output directory and jar files.
>
> Yes. There are definitely refinements that could be applied still. I'm not
> planning to dump this in scalac and forget about it. :-) My goal is to just
> get a basic working proof of concept in there and then improve upon it.

Sure, when I added this in sbt, I had to add another file for external
dependencies, so I figured I'd mention it in case thinking about it now helps
you keep everything in one file.

>
> > Can partest test this kind (modifying source files and verifying proper
> > compilation) of thing well?
>
> No, unfortunately not. I'm having trouble figuring out a good way to test
> it. If you've any suggestions I'd love to hear them.

Well, for sbt I implemented a scripted test framework where you write scripts
like:

> compile [success]
$ copy-file changes/A.scala src/main/scala/A.scala [success]
> compile [failure]

You could write something similar, steal it (see sbt's project directory), or
use sbt for testing (probably requires some tweaks that I'm happy to help
with).  One problem you might encounter is that last modified often only has
1 second resolution, which creates difficulties in automated testing like
this (the file doesn't actually look changed).

I've written more about this at
http://code.google.com/p/simple-build-tool/wiki/ChangeDetectionAndTesting.

> > > > Also, I think you miss generating the class file for a companion
> > > > object in certain cases.
> > >
> > > I'm not sure what you mean. scalac should be generating class files as
> > > normal.
> >
> > Sometimes scalac generates an extra class file that isn't listed in
> > unit.icode. I think it is for a companion object.
>
> Oh. How perplexing. I wonder if that's a bug or a feature. Can anyone who
> knows that aspect of the compiler a bit better comment?
>
> Hopefully this shouldn't matter for dependency tracking (as there will be
> other class files), but it probably will for cleaning up old class files.

I believe it is intentional. You can see it happen somewhere in GenJVM or
something like it I think. I think it is important to properly map sources
to classes because you don't want stale classes lying around. You can mask
errors that way.

>
> > > > Finally, if this is going to make it into scalac, it should track all
> > > > dependencies scalac is capable of tracking, such as dependencies on
> >
> > class
> >
> > > > files not in the output directory and jar files.
> > >
> > > Yes. There are definitely refinements that could be applied still. I'm
> >
> > not
> >
> > > planning to dump this in scalac and forget about it. :-) My goal is to
> >
> > just
> >
> > > get a basic working proof of concept in there and then improve upon it.
> >
> > Sure, when I added this in sbt, I had to add another file for external
> > dependencies, so I figured I'd mention it in case thinking about it now
> > helps
> > you keep everything in one file.
>
> Thanks. It's appreciated. :-)
>
> RE dependency files: I'm probably going to do something mildly evil and
> switch to an XML based format, which should give more flexibility about how
> I track features of individual files, as I can see that the file format is
> probably going to get complicated very quickly. This might also make it in
> some sense easier for other people to work with.

I think I liked your format last time I looked, but as long as the XML format
doesn't add any time to compilation, I have no major objections.

-Mark

> Oh. How perplexing. I wonder if that's a bug or a feature. Can anyone who
> knows that aspect of the compiler a bit better comment?
>
> Hopefully this shouldn't matter for dependency tracking (as there will be
> other class files), but it probably will for cleaning up old class files.

I believe it is intentional.  You can see it happen somewhere in GenJVM or
something like it I think.  I think it is important to properly map sources
to classes because you don't want stale classes lying around.  You can mask
errors that way.

> RE dependency files: I'm probably going to do something mildly evil and
> switch to an XML based format, which should give more flexibility about how
> I track features of individual files, as I can see that the file format is
> probably going to get complicated very quickly. This might also make it in
> some sense easier for other people to work with.

I think I liked your format last time I looked, but as long as the XML format
doesn't add any time to compilation, I have no major objections.

Not seem harsh also, its incredibly way too slow to recompile transitive dependencies with scalac, scalac is just not that kind of compiler that can be re-run over and over again on the same code without consequence. This is fine as a strategy for gcc or even javac, which are pretty fast, much faster than scalac line per line, but for scalac we have to be smarter, there just isn't any other way. If transitive dependencies must be recompiled in Scala, then it is essentially useless for large projects. 
If you have an example of where the plugin is falling short and missing dependencies, please let me know, it could be that we aren't tracing a new or changed construct correctly. The algorithm for tracking dependencies is quite simple, no detailed source analysis is necessary. 
Sean

On Mon, Feb 9, 2009 at 5:50 PM, David MacIver <david.maciver@gmail.com> wrote:

Well, not to sound harsh, but I've never heard anyone have a positive thing to say about the reliability of this method. Recompiling transitive dependencies is guaranteed to work and is required for correctness in sufficiently many cases (changes of type signatures for example) that you can't correctly determine whether a full transitive recompile is needed without detailed analysis of the source files. Which, as far as I can tell, the eclipse plugin either doesn't do or gets wrong.

On Mon, Feb 9, 2009 at 2:18 AM, Sean McDirmid <sean.mcdirmid@gmail.com> wrote:

Actually, sorry, the plugin tries to be a bit smarter about this. If a trait changes, then its immediate dependencies are recompiled (the files that directly depend on its files) and its end point dependencies are recompiled (the files that contain CLASSES that extend the trait). This is because true transitive closure isn't necessary, and is really too expensive (especially when recompiling the trait heavy plugin): you only need to recompile the extending classes so Scalac can do its bridge/super magic. 
Transitive closure recompilation really sucks, I'm glad we wouldn't have to use it. When I had this on in the plugin development ground to a halt as I was just recompiling the world on every save. 

On Mon, Feb 9, 2009 at 7:45 AM, David MacIver <david.maciver@gmail.com> wrote:
 

This should work fine. If a file is recompiled all files which use things from that file are also recompiled. So when the trait is recompiled so is anything that extends it (including things which extend it indirectly because the recompiles are propagated transitively).