how do I save a scala object to disk and then unpack it later?

Hi,

scala.util.Marshal serializes objects to byte arrays. You can see the
source code for that object here:
https://lampsvn.epfl.ch/trac/scala/browser/scala/tags/R_2_9_1_final/src/...

Here's a quick example that writes and reads an object:
https://gist.github.com/1480346

I wonder if there's a better way of reading bytes from a file, though...

Regards,
Vetle

On Thu, Dec 15, 2011 at 03:05, Kevin Murphy wrote:
> Hi
>
> I want to construct an object with a lot of internal state,  save it
> to disk, then have some other program read it and
> unpickle it. How can I do this?
>
> Kevin
>
> PS. I don't know Java, so I would appreciate a Scala solution :)

I use db4o for this, and find that it works very well.

http://www.db4o.com

Sample usage with Scala shown here:

http://www.matthewtodd.info/?p=68

>
> I want to construct an object with a lot of internal state, save it
> to disk, then have some other program read it and
> unpickle it. How can I do this?
>
> Kevin
>
> PS. I don't know Java, so I would appreciate a Scala solution :)
Just a few comments:
1) In the Java world, 'pickling' objects is called 'Serialization'.
2) To serialize a Java/Scala object, the object should extend Serializable. http://docs.oracle.com/javase/6/docs/api/java/io/Serializable.html
3) You might want to look at http://docs.oracle.com/javase/6/docs/api/java/io/ObjectOutputStream.html and http://java.sun.com/developer/technicalArticles/Programming/serialization/ for a bit more understanding. Scala's Marshall object is just using ObjectOutputStream under the hood.
4) Serialization in Java is fraught with gotchas; if you're doing enterprise-y type projects you might want to serialize objects using something more robust, like Google's Protocol Buffers.

Cheers

Thanks. Using you example from https://gist.github.com/1480346 ,
I am able to save and load a specific class (see code below).
But if i replace the specific class name Foo with a generic type T,
I get the following error in writeObjectToFile[T]:

:26: error: could not find implicit value for parameter m:
scala.reflect.ClassManifest[T]
out.write(Marshal.dump(obj))
^

How can I save/ load generically?

Thanks
Kevin

PS. This is for small-ish obejcts (say under 1MB), not enterprise-
level.
I'm familiar with protobufs, albeit not in scala.
I could convert the object to a string, but this seems very
inefficient
(since it will have lots of doubles internally).

// The following code works but is not generic.

class Foo(m: String) extends scala.Serializable {  val message = m 
val arr =  Array(1D,2D,3D)}
def writeObjectToFile(filename: String, obj: Foo) = {  // Code from
https://gist.github.com/1480346  import scala.util.Marshal  import
java.io._  val out = new FileOutputStream(filename) 
out.write(Marshal.dump(obj))  out.close}
def readObjectFromFile(filename: String): Foo = {  // https://gist.github.com/1480346 
import scala.util.Marshal  import java.io._  val in = new
FileInputStream(filename)  val bytes =
Stream.continually(in.read).takeWhile(-1 !=).map(_.toByte).toArray 
Marshal.load[Foo](bytes)}

val foo = new Foo("qweqwe")val filename =
"tmp.dat"writeObjectToFile(filename, foo)val bar =
readObjectFromFile(filename)bar.arr

On Dec 15, 12:23 am, Vetle Roeim wrote:
> Hi,
>
> scala.util.Marshal serializes objects to byte arrays. You can see the
> source code for that object here:https://lampsvn.epfl.ch/trac/scala/browser/scala/tags/R_2_9_1_final/s...
>
> Here's a quick example that writes and reads an object:https://gist.github.com/1480346
>
> I wonder if there's a better way of reading bytes from a file, though...
>
> Regards,
> Vetle
>
> On Thu, Dec 15, 2011 at 03:05, Kevin Murphy wrote:
> > Hi
>
> > I want to construct an object with a lot of internal state,  save it
> > to disk, then have some other program read it and
> > unpickle it. How can I do this?
>
> > Kevin
>
> > PS. I don't know Java, so I would appreciate a Scala solution :)
>
> --
> vr

Boy, I've got to watch my formatting!
Sorry that was completely unreadable - my cut and paste seems messed
up.

Anyway, what I was saying is that I can write save and load methods
specific to a class,
as in the code below (based on https://gist.github.com/1480346 ),
but I don't know how to write a generic save/ load function for any
class.
Ideas? (The obvious approach of replacing type Model with type T,
and defining polymorphic functions, does not work, maybe because
Java cannot handle this?)

object Model { // companion
import scala.util.Marshal
import java.io._

def writeModelToFile(filename: String, obj: Model) = {
val out = new FileOutputStream(filename)
out.write(Marshal.dump(obj))
out.close
}

def readModelFromFile(filename: String): Model = {
val in = new FileInputStream(filename)
val bytes = Stream.continually(in.read).takeWhile(-1 !
=).map(_.toByte).toArray
Marshal.load[Model](bytes)
}

} // end object

class Model(...) extends scala.Serializable {
// stuff
}

The other problem I have is that even with the above class-specific
code,
I get this error when trying to save the object:

java.io.NotSerializableException: scala.collection.immutable.MapLike
$ImmutableDefaultKeySet

I assume this is because my Model class contains a Map[String,Double]
field (in 'stuff' above)
How am I supposed to save/load such beasts?

Tx!
Kevin

On Thu, Dec 15, 2011 at 2:21 PM, Kevin Murphy <murphyk2@gmail.com> wrote:

Thanks. Using you example from https://gist.github.com/1480346 ,
I am able to save and load a specific class (see code below).
But if i replace the specific class name Foo with a generic type T,
I get the following error in writeObjectToFile[T]:

 <console>:26: error: could not find implicit value for parameter m:
scala.reflect.ClassManifest[T]
        out.write(Marshal.dump(obj))
                              ^

Since your using an API that requires a Manifest, you have to declare that you provide it, so that the compiler knows that one will be available.In other words, a ClassManifest[T] has to get into the (implicit) scope somehow. So for instance:
def writeObjectToFile[T: ClassManifest](filename: String, obj: T) = ...// syntactic sugar for: def writeObjectToFile[T](filename: String, obj: T)(implicit cm: ClassManifest[T])
Now you shifted the burden to your callers.

At this point, I'm more concerned that I can't serialize my particular
object;
I'll worry about the generic case later.
Here is my class:

type Mid = String
type Name = String
type Word = Stringclass Model(val entityNames: Map[Mid, Name],    val
aliases: Map[Mid, Set[Name]],    val wordCountsForEntity: Map[Mid,
Map[Word, Int]],    val wordCountsOverall: Map[Word, Int])   extends
scala.Serializable {  // constructor body  ()
// extra fields  val dummy = 0.5  //val words =
wordCountsOverall.keys.toSet // not serializable!

}

This can be saved and loaded (using Marshal) just fine.
But if I uncomment the words field, I get the following error
when trying to save this:

java.io.NotSerializableException: scala.collection.immutable.MapLike
$ImmutableDefaultKeySet
at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:
1180)
...

How can I tell the error is due to the words field?
In this case it's obvious, but I had to do binary search by hand
to find the culprit, each time commenting out fields, recompiling and
re-running!
Is there a better way?
How can I print the inferred type signature of a class (methods and
fields)?
Also, what's so hard about serializing a set of strings??

(In Matlab, I'd just type 'save("foo.mat", obj)' and 'obj =
load("foo.mat")' and I'd be done by now...
Very frustrating!)

Kevin

On Dec 15, 11:21 pm, Naftoli Gugenheim wrote:
> On Thu, Dec 15, 2011 at 2:21 PM, Kevin Murphy wrote:
> > Thanks. Using you example fromhttps://gist.github.com/1480346,
> > I am able to save and load a specific class (see code below).
> > But if i replace the specific class name Foo with a generic type T,
> > I get the following error in writeObjectToFile[T]:
>
> >  :26: error: could not find implicit value for parameter m:
> > scala.reflect.ClassManifest[T]
> >         out.write(Marshal.dump(obj))
> >                               ^
>
> Since your using an API that requires a Manifest, you have to declare that
> you provide it, so that the compiler knows that one will be available.
> In other words, a ClassManifest[T] has to get into the (implicit) scope
> somehow. So for instance:
>
> def writeObjectToFile[T: ClassManifest](filename: String, obj: T) = ...
> // syntactic sugar for: def writeObjectToFile[T](filename: String, obj:
> T)(implicit cm: ClassManifest[T])
>
> Now you shifted the burden to your callers.

On Thu, Dec 15, 2011 at 6:18 PM, Kevin Murphy <murphyk2@gmail.com> wrote:

I get this error when trying to save the object:

java.io.NotSerializableException: scala.collection.immutable.MapLike
$ImmutableDefaultKeySet

Oh, that guy.  mySetOfKeys.toList.toSet will work around it.  I could've sworn I filed a bug about those before, but maybe not...
-0xe1a

Awesome, thanks!

Summarizing, the following seems to work:

class Model(val wordCountsOverall: Map[String, Int], ...)
extends Serializable {
//val words = wordCountsOverall.keys.toSet
val words = wordCountsOverall.keys.toList.toSet // hack around the
bug
}

BTW, I figured out how to read/write generically,
following http://stackoverflow.com/questions/3442171/how-do-i-use-a-serializable-s....
Here's the code:

def writeObjectToFile(filename: String, obj: AnyRef) = { // obj must
have serialiazable trait
import java.io._
val fos = new FileOutputStream(filename)
val oos = new ObjectOutputStream(fos)
oos.writeObject(obj)
oos.close()
}

def readObjectFromFile[T](filename: String): T = {
import java.io._
val fis = new FileInputStream(filename)
val ois = new ObjectInputStream(fis)
val obj = ois.readObject()
ois.close()
obj.asInstanceOf[T]
}

Example usage:

val model = new Model(...)
va filename = "..."
writeObjectToFile(filename, model)
val model2 = readObjectFromFile[Model](flename)

K.

On Dec 16, 10:38 am, Alex Cruise wrote:
> On Thu, Dec 15, 2011 at 6:18 PM, Kevin Murphy wrote:
> > I get this error when trying to save the object:
>
> > java.io.NotSerializableException: scala.collection.immutable.MapLike
> > $ImmutableDefaultKeySet
>
> Oh, that guy.  mySetOfKeys.toList.toSet will work around it.  I could've
> sworn I filed a bug about those before, but maybe not...
>
> -0xe1a

On Fri, Dec 16, 2011 at 20:23, Kevin Murphy wrote:
> Awesome, thanks!
>
> Summarizing, the following seems to work:

I tried to answer your question, then went on vacation. Great that you
found out how to make it work! :)

You might want to look at scala-io
(https://github.com/scala-incubator/scala-io) if you want more
Scala-ish I/O. I have no experience with it, but it looks promising.

[...]
> BTW, I figured out how to read/write generically,
> following http://stackoverflow.com/questions/3442171/how-do-i-use-a-serializable-s....
> Here's the code:
>
> def writeObjectToFile(filename: String, obj: AnyRef) = { // obj must
> have serialiazable trait

Shouldn't you do something like this to make sure it's always serializable?

def writeObjectToFile(filename: String, obj: Serializable) = {}

[...]