State of XML projects

On Tue, Dec 8, 2009 at 4:15 AM, Jürgen Purtz <juergen@purtz.de> wrote:

Hi all, is there any regular (online) meeting of the XML team with a status
report similar to "Scala Meeting report" of gmane.comp.lang.scala.internals?

What is the actual state of the XML projects
(http://lamp.epfl.ch/~emir/projects/)? Any time schedule or release plans?

Burak graduated and is no longer at EPFL.  I think the release status of the projects is likely to be never.

Many of Burak's ideas made it into Lift via Burak's awesome coaching of me in my early Scala days.
 

Jürgen

--
Lift, the simply functional web framework http://liftweb.net
Beginning Scala http://www.apress.com/book/view/1430219890
Follow me: http://twitter.com/dpp
Surf the harmonics

David Pollak writes:

> I think the release status of the projects is likely to be never.

If this statement holds true it's essential that we find a consensus on which
XML tasks we should focus our future development activities.

On 09-12-08 08:04 AM, Jürgen Purtz wrote:
> David Pollak writes:
>
>> I think the release status of the projects is likely to be never.
>>
> If this statement holds true it's essential that we find a consensus on which
> XML tasks we should focus our future development activities.
>
>
With the best of intentions, I volunteered earlier this year to
coordinate the maintenance and evolution of Scala's XML infrastructure,
but have been unable to devote much time to it, due to various
circumstances partly beyond my control. If anyone would like to grab
the baton and try to get this parade marching again, it's yours, just
say the word!

Several times, we've started discussing various ways that the situation
could be improved, but nothing seems to have come of it. Unfortunately,
the XML infrastructure in Scala suffers from a variety of issues that
have conspired to make it much more difficult to make forward progress
than I expected at the outset:

- Library and data model are extremely tightly coupled to the compiler,
in a non-modular way;
- A large and economically important installed base (in the form of Lift
and its users);
- A complex, pseudo-cyclic inheritance graph that makes my head hurt;
- Relatively low general interest in the community, compared to
basically every other aspect of Scala;
- A great career move on the part of the original architect

None of these can really be blamed on anything or anyone (except my
headache, that's my problem ;), they're just the way things have ended up.

Early on, the threat of breaking binary compatibility put a serious
damper on things, but since then we've learned that binary compatibility
in Scala is, uh... problematic at best, and source compatibility is the
ideal we strive for.

As for how to move forward, I think we need to achieve sanity in the
absolute basics before trying to attack any new functionality.

I believe we've achieved consensus in scala-internals that the "bizarre
love triangle" of inheritance between Node, NodeSeq and Seq[Node] has to
go, but AFAIK that's the extent of the discussion to this time.

At this point what's needed is for someone who has a lot more time than
me to fork it, hack it, and see what comes of it.

Thanks,

-0xe1a

Hi. I'd like to volunteer to help with getting the XML support in Scala
up to scratch, since I'm being switching to Scala of late and most of my
work is XML-related. What would be the best next steps? Thanks a lot in
advance,

Cheers, Tony.
http://xmlguild.org//members#coates

On Tue, 08 Dec 2009 18:53:11 -0000, Alex Cruise wrote:

> With the best of intentions, I volunteered earlier this year to
> coordinate the maintenance and evolution of Scala's XML infrastructure,
> but have been unable to devote much time to it, due to various
> circumstances partly beyond my control. If anyone would like to grab
> the baton and try to get this parade marching again, it's yours, just
> say the word!

Good to see more people adopting Scala commercially :)We'll have a job market soon, any day now...

On Tue, Dec 8, 2009 at 9:25 PM, Anthony B. Coates (Londata) <abcoates@londata.com> wrote:

Hi.  I'd like to volunteer to help with getting the XML support in Scala up to scratch, since I'm being switching to Scala of late and most of my work is XML-related.  What would be the best next steps?  Thanks a lot in advance,

Cheers, Tony.
http://xmlguild.org//members#coates

On Tue, 08 Dec 2009 18:53:11 -0000, Alex Cruise <alex@cluonflux.com> wrote:

With the best of intentions, I volunteered earlier this year to coordinate the maintenance and evolution of Scala's XML infrastructure, but have been unable to devote much time to it, due to various circumstances partly beyond my control.  If anyone would like to grab the baton and try to get this parade marching again, it's yours, just say the word!

On 09-12-08 01:25 PM, Anthony B. Coates (Londata) wrote:
> Hi. I'd like to volunteer to help with getting the XML support in
> Scala up to scratch, since I'm being switching to Scala of late and
> most of my work is XML-related. What would be the best next steps?
> Thanks a lot in advance,
Great! I don't know if anyone else might be interested, but you seem
qualified and motivated. Toni (CC'd) can set you up with the
Contributor License Agreement etc.

Thanks!

-0xe1a

Hi,

A few thoughts for the wishlist from someone with a fairly extensive
XSLT background who is new to Scala.

1: My biggest concern is the following, from the Burak's Scala Tutorial
(which, incidentally, appears to be HTML served with a PDF mimetype):

***

The current implementation will not properly stratify namespace bindings
when elements from different scopes are combined. This is not a problem
when querying or processing XML data, but it might lead to wrong
namespace bindings when serializing XML. A modified version of the
serializing algorithm can solve the problem by introducing namespace
declarations and undeclarations in the right place. Since it seems a
rare problem and developers can stratify namespaces themselves in a
given XML application, your humble author and scala.xml maintainer did
not consider this issue a priority.

***

I don't know exactly what "will not properly stratify namespace
bindings" means. I ran it past an XML Guild member who didn't recognise
the terminology. But if it means that Scala XML isn't currently 100% W3C
compliant wrt namespaces, that seems to me like a huge issue in the long
term.

It manifestly hasn't been an issue for cranking out HTML, and it
probably isn't a problem for application programmers who want to do a
bit of XML when they have to (which, tellingly, is the line taken in the
XML chapters of all four Scala books I've looked at, ie "XML sucks but
since you might have to use it sometimes we give you tools that let you
engage with XML standards as little as possible").

But nesting XML documents, namespaces and potentially overlapping
prefixes do need to work for some XML applications, and it would be sad
to end up telling people that for full W3C compliance regarding
namespaces the user should use a legacy Java API such as Xerces from
within Scala, especially if the limitations are hard-baked.

Like I say, I may just be misreading the tutorial, in which case any
clarification anyone can offer would be most appreciated.

2: SAX-style parsing. I'm currently using Xerces SAX2 within Scala to
build an XDM representation. Scala represents the algorithm very neatly,
and I can even see some scope for concurrency while parsing serialized
data, but having to fight with a Java interface that, for example,
provides all attributes as one array reference results in more lines of
code than anything else. But it doesn't look to me like current Scala
pull support would give me all the events I need for full XDM
implementation, even before considering support for DTD and Schema
validation, which Xerces offers out of the box.

3: XPath support. Scala seems to be heading down the "All the useful
bits of XPath" route that Adobe has so far done at least two ways. But,
from the tutorial,

biblio \\ "_"

isn't any kind of XPath. Part of the beauty of XPath is that it's a
standard across a very wide range of technology. Programmers who don't
know XPath may prefer to learn the Scala alternative if it does what
they want. But for any programmer coming to Scala with some general XML
baggage, standard XPath would be a huge benefit. Also, none of the
examples I've seen suggest to me that XPath projection is as even as
powerful as XPath 1.0. Do we get predicates, for example, or do we have
to build the equivalent using mapping?

I suspect that getting XPath working properly (at least XPath 2.0) would
be hard without a 100% W3C data model, which brings me back to my first
point. But it's not hard to imagine a situation where in a few years'
time Scala could be the only major programming environment around that
doesn't support XPath. I struggle to see this as a feature.

I'm no Scala internals expert, but I'd be interested in contributing to
SAX/XDM/XPath-type developments if anyone else is interested in those
areas.

Regards,

Mark

Sure!

Anthony, if you would like to be an active contributor the first step is
filling in and sending us a Contributor License Agreement. You can find it,
together with the instructions, at:

http://www.scala-lang.org/sites/default/files/contributor_agreement.pdf

This step is necessary so that we can make sure no commercial or otherwise
unlicensed code ends up in the repository.

Once that is done, you can send a couple of initial patches to any of the
active contributors for a preliminary review (Paul Phillips, for example,
or Alex Cruise). Not much is needed, it's just a basic check.

Once they can vouch that those first contributions are ok, we'll set up
the repository privileges for you; just let us know your Trac account (you
can create one at https://lampsvn.epfl.ch/trac/scala/register), and you'll
have the required commit privileges in no time.

More details: you can find the list of pending XML bugs by following the
link "Scala XML" on the front page of the Scala Trac server:
http://lampsvn.epfl.ch/trac/scala

The newly created tickets that refer to the XML component are automatically
assigned to the Trac user "scala-xml_team"; they also get posted to the
scala-xml mailing list.

Once a ticket is closed, you should check whether the bug was present in a
previously released version. If so, while closing you should set the field
"Fixed in version" to "Unreleased-2.8.x". The field is then used to build
the "what's new" list at: http://www.scala-lang.org/node/299
If the ticket was never present in a released version (for instance it
refers to a new feature, or it is a regression), the field should not be set.

When a new revision is committed via SVN (we will probably switch to Git in
the near future), the commit message is automatically parsed for references
to tickets. If one is found, the commit message is also appended to the
corresponding ticket/tickets. For instance: "Closes #1e1e1e, #202020. See #222222"
will close the first two tickets, but not close the third. The commit
message is appended to all three. In detail:

- 'close', closed', 'closes' => adds a reference, and closes the ticket
- 'fix', 'fixed', 'fixes', 'addresses', 're', 'references', 'refs', 'see'
=> adds a reference, but does not close the ticket

We are currently setting up a system to automatically create review
requests by using a similar mechanism, but it's not completely ready yet.

I think that's basically it. If you need additional details, feel free to
send me a message at any time!
Toni

Alex Cruise wrote:
> On 09-12-08 01:25 PM, Anthony B. Coates (Londata) wrote:
>> Hi. I'd like to volunteer to help with getting the XML support in
>> Scala up to scratch, since I'm being switching to Scala of late and
>> most of my work is XML-related. What would be the best next steps?
>> Thanks a lot in advance,
> Great! I don't know if anyone else might be interested, but you seem
> qualified and motivated. Toni (CC'd) can set you up with the
> Contributor License Agreement etc.
>
> Thanks!
>
> -0xe1a
>

Out of curiosity, have you considered starting initial work via the scala-incubator?
You can work there and produce patches to submit to SVN as normal, but can also bring in collaborators while it's work-in-progress
The contributor agreement is still a requirement, but there's no need to prove yourself via patches before getting access rights, as the incubator is the place where you'd prove yourself...

On Wed, Dec 9, 2009 at 10:32 AM, Antonio Cunei <antonio.cunei@epfl.ch> wrote:

Sure!

Anthony, if you would like to be an active contributor the first step is filling in and sending us a Contributor License Agreement. You can find it, together with the instructions, at:

http://www.scala-lang.org/sites/default/files/contributor_agreement.pdf

This step is necessary so that we can make sure no commercial or otherwise unlicensed code ends up in the repository.

Once that is done, you can send a couple of initial patches to any of the active contributors for a preliminary review (Paul Phillips, for example, or Alex Cruise). Not much is needed, it's just a basic check.

Once they can vouch that those first contributions are ok, we'll set up the repository privileges for you; just let us know your Trac account (you can create one at https://lampsvn.epfl.ch/trac/scala/register), and you'll have the required commit privileges in no time.


More details: you can find the list of pending XML bugs by following the link "Scala XML" on the front page of the Scala Trac server: http://lampsvn.epfl.ch/trac/scala

The newly created tickets that refer to the XML component are automatically assigned to the Trac user "scala-xml_team"; they also get posted to the scala-xml mailing list.

Once a ticket is closed, you should check whether the bug was present in a previously released version. If so, while closing you should set the field "Fixed in version" to "Unreleased-2.8.x". The field is then used to build the "what's new" list at: http://www.scala-lang.org/node/299
If the ticket was never present in a released version (for instance it refers to a new feature, or it is a regression), the field should not be set.

When a new revision is committed via SVN (we will probably switch to Git in the near future), the commit message is automatically parsed for references to tickets. If one is found, the commit message is also appended to the corresponding ticket/tickets. For instance: "Closes #1e1e1e, #202020. See #222222" will close the first two tickets, but not close the third. The commit message is appended to all three. In detail:

- 'close', closed', 'closes' => adds a reference, and closes the ticket
- 'fix', 'fixed', 'fixes', 'addresses', 're', 'references', 'refs', 'see'
    => adds a reference, but does not close the ticket

We are currently setting up a system to automatically create review requests by using a similar mechanism, but it's not completely ready yet.

I think that's basically it. If you need additional details, feel free to send me a message at any time!
Toni


Alex Cruise wrote:

On 09-12-08 01:25 PM, Anthony B. Coates (Londata) wrote:

Hi.  I'd like to volunteer to help with getting the XML support in Scala up to scratch, since I'm being switching to Scala of late and most of my work is XML-related.  What would be the best next steps?  Thanks a lot in advance,

Great!  I don't know if anyone else might be interested, but you seem qualified and motivated.  Toni (CC'd) can set you up with the Contributor License Agreement etc.

Thanks!

-0xe1a

On Wed, Dec 9, 2009 at 10:57 AM, Kevin Wright
wrote:
> Out of curiosity, have you considered starting initial work via the
> scala-incubator?
>
> You can work there and produce patches to submit to SVN as normal, but can
> also bring in collaborators while it's work-in-progress
>
> The contributor agreement is still a requirement, but there's no need to
> prove yourself via patches before getting access rights, as the incubator is
> the place where you'd prove yourself...

Yes, indeed ... the incubator would be the right place to get this
work underway.

http://groups.google.com/group/scala-incubator

Cheers,

Miles

Alex Cruise writes:

>
> On 09-12-08 08:04 AM, Jürgen Purtz wrote:
> > David Pollak writes:
> >
> >> I think the release status of the projects is likely to be never.
> >>
> > If this statement holds true it's essential that we find a consensus on which
> > XML tasks we should focus our future development activities.
> >
> >
> With the best of intentions, I volunteered earlier this year to
> coordinate the maintenance and evolution of Scala's XML infrastructure,
> but have been unable to devote much time to it, due to various
> circumstances partly beyond my control. If anyone would like to grab
> the baton and try to get this parade marching again, it's yours, just
> say the word!
>
> Several times, we've started discussing various ways that the situation
> could be improved, but nothing seems to have come of it. Unfortunately,
> the XML infrastructure in Scala suffers from a variety of issues that
> have conspired to make it much more difficult to make forward progress
> than I expected at the outset:
>
> - Library and data model are extremely tightly coupled to the compiler,
> in a non-modular way;
> - A large and economically important installed base (in the form of Lift
> and its users);
> - A complex, pseudo-cyclic inheritance graph that makes my head hurt;
> - Relatively low general interest in the community, compared to
> basically every other aspect of Scala;
> - A great career move on the part of the original architect
>
> None of these can really be blamed on anything or anyone (except my
> headache, that's my problem ;), they're just the way things have ended up.
>
> Early on, the threat of breaking binary compatibility put a serious
> damper on things, but since then we've learned that binary compatibility
> in Scala is, uh... problematic at best, and source compatibility is the
> ideal we strive for.
>
> As for how to move forward, I think we need to achieve sanity in the
> absolute basics before trying to attack any new functionality.
>
> I believe we've achieved consensus in scala-internals that the "bizarre
> love triangle" of inheritance between Node, NodeSeq and Seq[Node] has to
> go, but AFAIK that's the extent of the discussion to this time.
>
> At this point what's needed is for someone who has a lot more time than
> me to fork it, hack it, and see what comes of it.
>
> Thanks,
>
> -0xe1a
>
>

Alex,

your are right pointing out that we need
- a summary of what is going wrong
- sanity in the absolute basics ... before ... any new functionality
- a person coodinating things.

And I like to add that we need a consensus about the platform for our
discussion. I'm confused about the great number of communication channels: EMail
lists, news and buglist at EPFL; Gmane; private E-Mails; Scala Incubator at
Google. Parts of them are redundant as a result of forwarding mechanisms. There
is a recommendation from Martin how to organize some things:

http://permalink.gmane.org/gmane.comp.lang.scala.internals/2284

Can you make a proposal which communication infrastructure we should use? I like
to post some ideas about design questions of package scala.xml.

Jürgen

On 09-12-09 05:36 AM, Jürgen Purtz wrote:
> your are right pointing out that we need
> - a summary of what is going wrong
> - sanity in the absolute basics ... before ... any new functionality
> - a person coodinating things.
>
> And I like to add that we need a consensus about the platform for our
> discussion. I'm confused about the great number of communication channels: EMail
> lists, news and buglist at EPFL; Gmane; private E-Mails; Scala Incubator at
> Google. Parts of them are redundant as a result of forwarding mechanisms. There
> is a recommendation from Martin how to organize some things:
>
> http://permalink.gmane.org/gmane.comp.lang.scala.internals/2284
>
> Can you make a proposal which communication infrastructure we should use? I like
> to post some ideas about design questions of package scala.xml.
>

My preference, for what it's worth, is to have design discussions on
this list (i.e. scala-xml). Then, when we have a rough consensus on a
concrete design and preliminary implementations start to happen, we
should have the discussion about how to integrate the changes into the
main distribution on the incubator list.

Also note that I don't think we necessarily need to have a big
discussion about design before any work gets started; people should feel
free to wade right in and start hacking and slashing, as a great way to
explore the design space if nothing else. But I do think it would be
best to solicit feedback regularly from EPFL and other community members
in the most official channel we have.

As for a coordinator, I said yesterday that I'm happy to abdicate in
favour of anyone who seems more likely than me to get the job done, and
it looks like Anthony Coates is ready to step in.

What's most important is that people are sufficiently motivated to
engage with the discussion and put in some time and effort, which I'm
happy to say seems to be the case now.

Thanks!

-0xe1a

Miles, what is there to this incubator at the moment. I had a look at the
Google Group (& joined it), but I don't feel I have the big picture yet.
The Google Groups gives you a mailing list, and I see that there is a
Lighthouse account for issues (are you moving away from Trac?). I'm
presuming that are matching repositories for the incubator projects,
perhaps in Git(?).

Is there some big picture somewhere is what is going on the the incubator,
a tracking page or something? Did I miss something obvious? Thanks a lot
in advance,

Cheers, Tony.

On Wed, 09 Dec 2009 11:17:33 -0000, Miles Sabin
wrote:

> On Wed, Dec 9, 2009 at 10:57 AM, Kevin Wright
> wrote:
>> Out of curiosity, have you considered starting initial work via the
>> scala-incubator?
>>
>> You can work there and produce patches to submit to SVN as normal, but
>> can
>> also bring in collaborators while it's work-in-progress
>>
>> The contributor agreement is still a requirement, but there's no need to
>> prove yourself via patches before getting access rights, as the
>> incubator is
>> the place where you'd prove yourself...
>
> Yes, indeed ... the incubator would be the right place to get this
> work underway.
>
> http://groups.google.com/group/scala-incubator
>
> Cheers,
>
>
> Miles

Yes indeed :)Our github account is here: http://github.com/scala-incubator


On Wed, Dec 9, 2009 at 9:29 PM, Anthony B. Coates (Londata) <abcoates@londata.com> wrote:

Miles, what is there to this incubator at the moment.  I had a look at the Google Group (& joined it), but I don't feel I have the big picture yet.  The Google Groups gives you a mailing list, and I see that there is a Lighthouse account for issues (are you moving away from Trac?).  I'm presuming that are matching repositories for the incubator projects, perhaps in Git(?).

Is there some big picture somewhere is what is going on the the incubator, a tracking page or something?  Did I miss something obvious?  Thanks a lot in advance,

Cheers, Tony.

On Wed, 09 Dec 2009 11:17:33 -0000, Miles Sabin <miles@milessabin.com> wrote:

On Wed, Dec 9, 2009 at 10:57 AM, Kevin Wright
<kev.lee.wright@googlemail.com> wrote:

Out of curiosity, have you considered starting initial work via the
scala-incubator?

You can work there and produce patches to submit to SVN as normal, but can
also bring in collaborators while it's work-in-progress

The contributor agreement is still a requirement, but there's no need to
prove yourself via patches before getting access rights, as the incubator is
the place where you'd prove yourself...

Yes, indeed ... the incubator would be the right place to get this
work underway.

 http://groups.google.com/group/scala-incubator

Cheers,


Miles

On Wed, Dec 9, 2009 at 9:29 PM, Anthony B. Coates (Londata)
wrote:
> Miles, what is there to this incubator at the moment.  I had a look at the
> Google Group (& joined it), but I don't feel I have the big picture yet.
>  The Google Groups gives you a mailing list, and I see that there is a
> Lighthouse account for issues (are you moving away from Trac?).  I'm
> presuming that are matching repositories for the incubator projects, perhaps
> in Git(?).
>
> Is there some big picture somewhere is what is going on the the incubator, a
> tracking page or something?  Did I miss something obvious?  Thanks a lot in
> advance,

The background is in this mail from Martin,

http://thread.gmane.org/gmane.comp.lang.scala.internals/2284

The Scala Incubator and Greenhouse are both in their very early stages
at the moment ... your input would be most welcome.

Cheers,

Miles