Re: Re: Looks like I found a scala bug, a memory leak, but perhaps I missed something...

Note the distinction between an Iterable and an Iterator. An Iterable may be traversed many times, while an Iterator may not.

On Thu, Feb 3, 2011 at 10:29, dacr <crosson.david@gmail.com> wrote:

THANK you very much, if I remove the reference on the head of stream,
I didn't have an out of memory anymore ! I understand what happened,
it was quite a normal situation.

So now, with code as follow, everything is ok :

   reader.toIterable map { ...
   } foreach {
     _ match {
       case None=>
       case Some(strevent) =>
         val xml=XML.loadString(strevent)
     }
   }


thanks,
David.


On Feb 3, 1:09 pm, "Tommaso Galleri" <Tommaso.Gall...@bjss.com> wrote:
> Hi,
>
> Not tried your code, but have a look at this:
>
> scala> val a = (0 to 2000000000).toStream.toIterable
> a: Iterable[Int] = Stream(0, ?)
>
> scala> a.foreach{x=>}
> Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
>
> Imagine (0 to 2000000000).toStream is your reader; then I get an
> iterable on it.
>
> In this case keeping "val a" around causes the head of the stream to be
> referenced and so nothing can be garbage collected.
>
> If we lose "val a", then everything is fine...
>
> scala> (0 to 2000000000).toStream.toIterable.foreach{x=>}
>
> scala>
>
> Could be down to this or not, not done enough analysis.
>
> Have a go,
>
> Tommaso
>
> -----Original Message-----
> From: scala-user@googlegroups.com [mailto:scala-user@googlegroups.com]
>
> On Behalf Of dacr
> Sent: 03 February 2011 11:14
> To: scala-user
> Subject: [scala-user] Looks like I found a scala bug, a memory leak, but
> perhaps I missed something...
>
> Hi all,
>
> Consider the following code, which extract from a 40Mo xml log file,
> very small xml "events" parts :
>
> import java.io.FileInputStream
> import scala.xml.XML
> import javax.xml.stream._
> import javax.xml.stream.events._
> import scala.collection.JavaConversions._
>
> object XMLPullCost {
>   def main(args: Array[String]): Unit = {
>     val factory = XMLInputFactory.newFactory
>     val reader = factory.createXMLEventReader(new
> FileInputStream("totoxml.log"))
>     def stst(s:StartElement, v:String) = s.getName.getLocalPart==v
>     def etst(e:EndElement, v:String) = e.getName.getLocalPart==v
>     val buffer = new StringBuilder()
>     val strevents = reader.toIterable map {
>       case i:StartDocument => None
>       case i:EndDocument => None
>       case s:StartElement if stst(s,"verbosegc") => None
>       case e:EndElement   if etst(e,"verbosegc") => None
>       case s:StartElement if stst(s,"sys") =>
> {println("delayed?");buffer.clear; buffer.append(s); None}
>       case e:EndElement   if etst(e,"sys") => {buffer.append(e);
> Some(buffer.toString)}
>       case s:StartElement if stst(s,"af")  =>
> {println("delayed?");buffer.clear; buffer.append(s); None}
>       case e:EndElement   if etst(e,"af")  => {buffer.append(e);
> Some(buffer.toString)}
>       case c@_ => buffer.append(c); None
>     }
>     System.out.println("events processing...")
>     strevents foreach {
>       _ match {
>         case None=>
>         case Some(strevent) =>
>           val xml=XML.loadString(strevent)
>       }
>     }
>   }
> }
>
> Using a 256Mo sized, which should be large enough I got Memory leak.
> When started I got the following output (-Xmx256M -Xms256M -XX:
> +HeapDumpOnOutOfMemoryError ):
>
> events processing...
> delayed?
> delayed?
> delayed?
> delayed?
> ...
>
> So the streaming processing approach is working fine, my xml small
> part are only extracted when I need the next one, within the strevents
> foreach loop. But I'm getting an OutOfMemory, looks like the problem
> is with "val xml=XML.loadString(strevent)", the GarbageCollector can't
> free the memory allocated by this call.
>
> Did I miss something ? Would just like your point of view before
> filling a bug.
>
> regards,
> david.
>
> The information included in this email and any files transmitted with it may contain information that is confidential and it must not be used by, or its contents or attachments copied or disclosed, to persons other than the intended addressee. If you have received this email in error, please notify BJSS. In the absence of written agreement to the contrary BJSS' relevant standard terms of contract for any work to be undertaken will apply. Please carry out virus or such other checks as you consider appropriate in respect of this email. BJSS do not accept responsibility for any adverse effect upon your system or data in relation to this email or any files transmitted with it. BJSS Limited, a company registered in England and Wales (Company Number 2777575), VAT Registration Number 613295452, Registered Office Address, First Floor, Coronet House, Queen Street, Leeds, LS1 2TW

--
Daniel C. Sobral

I travel to the future all the time.