Detecting unicode

On Sun, Feb 01, 2009 at 10:55:41AM -0800, Marcus Downing wrote:
> Is there some code out there or in the standard libaries that already that
> does this, or do I have to roll my own?

It's not scala, but since I just recently had to learn more than I cared to about charsets, this is the most
promising code I found for that purpose.

http://glaforge.free.fr/wiki/index.php?wiki=GuessEncoding

Paul Phillips wrote:
>
> It's not scala, but since I just recently had to learn more than I cared
> to about charsets, this is the most
> promising code I found for that purpose.
>
> http://glaforge.free.fr/wiki/index.php?wiki=GuessEncoding
>
That looked promising - it returned the correct encoding, and I could see
that the data was in its cache - but something was stopping BufferedReader
from working. I could probably have fixed it, but I found this instead and
got it working:

http://koti.mbnet.fi/akini/java/unicodereader/

Scala doesn't seem to like accessing Java classes with an empty package, so
I had to stick it under org.unicode and make the constructor public.

While the topic is being discussed, a while ago a made a little quiz script. The New Jersey Motor Vehicles Commission has online (Ajax) quizzes to prepare you to take the permit test. I downloaded the XML files that it uses behind the scenes and a made a little console script to test you on it. I ran into a minor problem. Apparently the XML files start with a BOM, and Scala was reading it as gibberish -- invalid XML. I ended up opening each file in Notepad++ and converting it from with BOM to without BOM.

On Sun, Feb 1, 2009 at 8:20 PM, Marcus Downing <marcus@minotaur.it> wrote:

Paul Phillips wrote:
>
> It's not scala, but since I just recently had to learn more than I cared
> to about charsets, this is the most
> promising code I found for that purpose.
>
>   http://glaforge.free.fr/wiki/index.php?wiki=GuessEncoding
>
That looked promising - it returned the correct encoding, and I could see
that the data was in its cache - but something was stopping BufferedReader
from working. I could probably have fixed it, but I found this instead and
got it working:

 http://koti.mbnet.fi/akini/java/unicodereader/

Scala doesn't seem to like accessing Java classes with an empty package, so
I had to stick it under org.unicode and make the constructor public.
--
View this message in context: http://www.nabble.com/Detecting-unicode-tp21778870p21782719.html
Sent from the Scala - User mailing list archive at Nabble.com.

Naftoli Gugenheim wrote:
> I ran into a minor problem. Apparently the XML files start with a BOM,
> and Scala was reading it as gibberish -- invalid XML. I ended up opening
> each file in Notepad++ and converting it from with BOM to without BOM.

I've seen the same issue. I had thought that the BOM would be properly
processed but it was coming through as garbage, so I ended up hacking by
explicitly skipping the BOM bytes in my input. I'd love for it to be
handled correctly, but I haven't had time to really dig into the code.

Derek

Derek Chen-Becker wrote:
> I've seen the same issue. I had thought that the BOM would be properly
> processed but it was coming through as garbage, so I ended up hacking by
> explicitly skipping the BOM bytes in my input. I'd love for it to be
> handled correctly, but I haven't had time to really dig into the code.
>
Seeing as scala.io.Source already presents a character-oriented view of
I/O, I think BOM sniffing would be a useful enhancement there. IIRC a
lot of XML parsers do it.

-0xe1a

Hi Marcus,

Thanks for introducing me to BOMs - I'd somehow never been aware of them
before!

Marcus Downing wrote:
> Scala doesn't seem to like accessing Java classes with an empty package, so
> I had to stick it under org.unicode and make the constructor public.

I've not tried this, but does importing _root_.JavaClass work?

Jon