- About Scala
- Documentation
- Code Examples
- Software
- Scala Developers
memory limit
Thu, 2010-10-28, 20:18
Hi all,
I'm writing a small program which will count repeated string sequence in a text file.
My program heavily rely on List to processing intermediate results. It works fine for small input size, but will raise exceptions for large input size.
Is it possible that the runtime has not allocated enough memory for my program? If so, how could I request for more memory?
Jiansen
I'm writing a small program which will count repeated string sequence in a text file.
My program heavily rely on List to processing intermediate results. It works fine for small input size, but will raise exceptions for large input size.
Is it possible that the runtime has not allocated enough memory for my program? If so, how could I request for more memory?
Jiansen
Thu, 2010-10-28, 20:57
#2
Re: memory limit
Thank you.
I would more like to run my program through command line.
concordance_0_1.scala is the file where I put my main method through a singleton concordance_0_1 object.
Usually, I compile it by scalac concordance_0_1.scala, and run by scala concordance_0_1 2 myText.txt.
I tried java -Xmx4G -cp /usr/local/scala/lib/scala-library.jar:.concordance_0_1, but got the same result as I run java -help. Perhaps this is due to the -X option.
When I tried java -Xmx4G -cp /usr/local/scala/lib/scala-library.jar:.concordance_0_1 2 myText.txt, it raised java.lang.ClassNotFoundException
Any further suggestions?
Jiansen
I would more like to run my program through command line.
concordance_0_1.scala is the file where I put my main method through a singleton concordance_0_1 object.
Usually, I compile it by scalac concordance_0_1.scala, and run by scala concordance_0_1 2 myText.txt.
I tried java -Xmx4G -cp /usr/local/scala/lib/scala-library.jar:.concordance_0_1, but got the same result as I run java -help. Perhaps this is due to the -X option.
When I tried java -Xmx4G -cp /usr/local/scala/lib/scala-library.jar:.concordance_0_1 2 myText.txt, it raised java.lang.ClassNotFoundException
Any further suggestions?
Jiansen
Thu, 2010-10-28, 21:17
#3
Re: memory limit
You don't have spaces in the right place. I will write a new line each place there should be a space
java
-Xmx4G
-cp
/usr/local/scala/lib/scala-library.jar:.
concordance_0_1
2
myText.txt
Also, make sure you use the correct path to your scala-library.jar. It almost certainly is NOT at /usr/local/scala/lib. Mine isn't--that was just an example.
Finally, if you're running on a 32 bit machine (or with a 32 bit JVM), 4G is certainly too much memory for the JVM. Try 1G or 2G. (If you want more resolution, use e.g. 1500M (for megabytes).)
--Rex
On Thu, Oct 28, 2010 at 3:48 PM, Jiansen He <jiansenhe@googlemail.com> wrote:
java
-Xmx4G
-cp
/usr/local/scala/lib/scala-library.jar:.
concordance_0_1
2
myText.txt
Also, make sure you use the correct path to your scala-library.jar. It almost certainly is NOT at /usr/local/scala/lib. Mine isn't--that was just an example.
Finally, if you're running on a 32 bit machine (or with a 32 bit JVM), 4G is certainly too much memory for the JVM. Try 1G or 2G. (If you want more resolution, use e.g. 1500M (for megabytes).)
--Rex
On Thu, Oct 28, 2010 at 3:48 PM, Jiansen He <jiansenhe@googlemail.com> wrote:
Thank you.
I would more like to run my program through command line.
concordance_0_1.scala is the file where I put my main method through a singleton concordance_0_1 object.
Usually, I compile it by scalac concordance_0_1.scala, and run by scala concordance_0_1 2 myText.txt.
I tried java -Xmx4G -cp /usr/local/scala/lib/scala-library.jar:.concordance_0_1, but got the same result as I run java -help. Perhaps this is due to the -X option.
When I tried java -Xmx4G -cp /usr/local/scala/lib/scala-library.jar:.concordance_0_1 2 myText.txt, it raised java.lang.ClassNotFoundException
Any further suggestions?
Jiansen
Thu, 2010-10-28, 21:57
#4
Re: memory limit
Thanks again. I can run the program in your way.
But it seems that I'm too confident about the correctness of my program :-(
I got following exception:
dyn-95-175-140-17:src jiansenhe$ java -mx4G -cp /Users/jiansenhe/scala/lib/scala-library.jar:. concordance_0_1 1 TMM.txt
File loaded in 0.162 secs
Exception in thread "main" java.lang.StackOverflowError
at scala.collection.immutable.List.length(List.scala:45)
at scala.collection.SeqLike$class.size(SeqLike.scala:221)
at scala.collection.immutable.List.size(List.scala:45)
at concordance_0_1$.gen_(concordance_0_1.scala:36)
at concordance_0_1$.gen_(concordance_0_1.scala:37)
at concordance_0_1$.gen_(concordance_0_1.scala:37)
...many same exceptions ..
Line 32 to line 38 are cited as follows:
32 def gen_(i:Int, len:Int, l:List[String]): List[(List [String], Int)] = l match {
33 case Nil => Nil
34 case h::t =>
35 // println("i="+i+" l.size=" + l.size + " len=" + len + " l.take(len)=" +l.take(len))
36 if (len>l.size) Nil
37 else insert((l.take(len),i), gen_(i+1, len, t))
38 }
After putting back the commend in Line 35, I got follows:
...
i=7000 l.size=4354 len=1 l.take(len)=List(say)
i=7001 l.size=4353 len=1 l.take(len)=List(a)
Exception in thread "main" java.lang.StackOverflowError
at scala.runtime.AbstractFunction1.<init>(AbstractFunction1.scala:16)
at scala.collection.TraversableOnce$$anonfun$addString$1.<init>(TraversableOnce.scala:502)
at scala.collection.TraversableOnce$class.addString(TraversableOnce.scala:502)
at scala.collection.immutable.List.addString(List.scala:45)
at scala.collection.TraversableOnce$class.mkString(TraversableOnce.scala:467)
at scala.collection.immutable.List.mkString(List.scala:45)
at scala.collection.TraversableLike$class.toString(TraversableLike.scala:708)
at scala.collection.SeqLike$class.toString(SeqLike.scala:869)
at scala.collection.immutable.List.toString(List.scala:45)
at java.lang.String.valueOf(String.java:2826)
at scala.collection.mutable.StringBuilder.append(StringBuilder.scala:178)
at concordance_0_1$.gen_(concordance_0_1.scala:35)
at concordance_0_1$.gen_(concordance_0_1.scala:37)
at concordance_0_1$.gen_(concordance_0_1.scala:37)
at concordance_0_1$.gen_(concordance_0_1.scala:37)
...many same exceptions...
The gen_ function is used as follows:
gen_(0, len, strings) will generate all sequences at length len, together with index of each sequence.
Any ideas where could go wrong?
Ideally, a list in scala might have infinite length, right? Besides, the exception is raised really soon.
Jiansen
But it seems that I'm too confident about the correctness of my program :-(
I got following exception:
dyn-95-175-140-17:src jiansenhe$ java -mx4G -cp /Users/jiansenhe/scala/lib/scala-library.jar:. concordance_0_1 1 TMM.txt
File loaded in 0.162 secs
Exception in thread "main" java.lang.StackOverflowError
at scala.collection.immutable.List.length(List.scala:45)
at scala.collection.SeqLike$class.size(SeqLike.scala:221)
at scala.collection.immutable.List.size(List.scala:45)
at concordance_0_1$.gen_(concordance_0_1.scala:36)
at concordance_0_1$.gen_(concordance_0_1.scala:37)
at concordance_0_1$.gen_(concordance_0_1.scala:37)
...many same exceptions ..
Line 32 to line 38 are cited as follows:
32 def gen_(i:Int, len:Int, l:List[String]): List[(List [String], Int)] = l match {
33 case Nil => Nil
34 case h::t =>
35 // println("i="+i+" l.size=" + l.size + " len=" + len + " l.take(len)=" +l.take(len))
36 if (len>l.size) Nil
37 else insert((l.take(len),i), gen_(i+1, len, t))
38 }
After putting back the commend in Line 35, I got follows:
...
i=7000 l.size=4354 len=1 l.take(len)=List(say)
i=7001 l.size=4353 len=1 l.take(len)=List(a)
Exception in thread "main" java.lang.StackOverflowError
at scala.runtime.AbstractFunction1.<init>(AbstractFunction1.scala:16)
at scala.collection.TraversableOnce$$anonfun$addString$1.<init>(TraversableOnce.scala:502)
at scala.collection.TraversableOnce$class.addString(TraversableOnce.scala:502)
at scala.collection.immutable.List.addString(List.scala:45)
at scala.collection.TraversableOnce$class.mkString(TraversableOnce.scala:467)
at scala.collection.immutable.List.mkString(List.scala:45)
at scala.collection.TraversableLike$class.toString(TraversableLike.scala:708)
at scala.collection.SeqLike$class.toString(SeqLike.scala:869)
at scala.collection.immutable.List.toString(List.scala:45)
at java.lang.String.valueOf(String.java:2826)
at scala.collection.mutable.StringBuilder.append(StringBuilder.scala:178)
at concordance_0_1$.gen_(concordance_0_1.scala:35)
at concordance_0_1$.gen_(concordance_0_1.scala:37)
at concordance_0_1$.gen_(concordance_0_1.scala:37)
at concordance_0_1$.gen_(concordance_0_1.scala:37)
...many same exceptions...
The gen_ function is used as follows:
gen_(0, len, strings) will generate all sequences at length len, together with index of each sequence.
Any ideas where could go wrong?
Ideally, a list in scala might have infinite length, right? Besides, the exception is raised really soon.
Jiansen
Thu, 2010-10-28, 22:07
#5
Re: memory limit
On 10/28/2010 01:50 PM, Jiansen He wrote:
> Thanks again. I can run the program in your way.
>
> But it seems that I'm too confident about the correctness of my program :-(
>
>
> I got following exception:
>
> dyn-95-175-140-17:src jiansenhe$ java -mx4G -cp
> /Users/jiansenhe/scala/lib/scala-library.jar:. concordance_0_1 1 TMM.txt
> File loaded in 0.162 secs
> Exception in thread "main" java.lang.StackOverflowError
> [snip]
A stack overflow in Java/Scala is often the result of a recursive
function gone awry (e.g. infinite recursion). In this case, it's
probably just that your list is too large for the available stack space.
There are a couple ways to fix this:
1. increase stack size (I think its the -Xss java parameter)
2. rewrite your function using iteration rather than recursion
3. rewrite your function using tail recursion
> Any ideas where could go wrong?
>
> Ideally, a list in scala might have infinite length, right? Besides, the
> exception is raised really soon.
I assume you mean that a list can have arbitrary length, which is true.
It cannot have infinite length, but a Stream can.
Fri, 2010-10-29, 09:07
#6
Re: memory limit
Thanks Aaron.
Since I'm actually running a benchmark for different languages, I'm not going to change the code to tail recursive style.
My program works fine after using -Xss java parameters.
Thanks again and again
Jiansen
Since I'm actually running a benchmark for different languages, I'm not going to change the code to tail recursive style.
My program works fine after using -Xss java parameters.
Thanks again and again
Jiansen
Fri, 2010-10-29, 09:17
#7
Re: memory limit
Sorry for forgetting thank you, Rex.
My apologize~~
Jiansen
On Fri, Oct 29, 2010 at 8:58 AM, Jiansen He <jiansenhe@googlemail.com> wrote:
My apologize~~
Jiansen
On Fri, Oct 29, 2010 at 8:58 AM, Jiansen He <jiansenhe@googlemail.com> wrote:
Thanks Aaron.
Since I'm actually running a benchmark for different languages, I'm not going to change the code to tail recursive style.
My program works fine after using -Xss java parameters.
Thanks again and again
Jiansen
JAVA_OPTS=-Xmx4G scala
scala> Runtime.getRuntime.maxMemory
res0: Long = 3817799680
This also works when calling a compiled file.
Alternatively, you can run the class directly with java:
java -Xmx4G -cp /usr/local/scala/lib/scala-library.jar:. MyClass
(obviously, use the correct path to your scala-library.jar, and scala-swing.jar if you use swing, etc.) and then you can put whatever options you like on the command-line.
--Rex
On Thu, Oct 28, 2010 at 3:18 PM, Jiansen He <jiansenhe@googlemail.com> wrote: