Scala in EDF, Lee Momtahan at CUFP 09

Lee Momtahan talks at the Commercial Users of Functional Programming meeting, CUFP 2009, about his experiences at Électricité de France Trading (EDFT) as a member of a group developing applications for energy derivatives trading and pricing. He explains the benefits EDFT has gained by replacing a significant part of their 300,000 lines of Java code with Scala and their impressions of Scala. You can find a copy of his slides here and the video of his actual presentation here.

He summarizes by saying "We're using Scala for real work in a business critical situation and there is a lot of money at stake if we get it wrong. There are risks, but we have had no problems [with Scala]. For me a developer working at the coal face, the productivity improvements are massive.", about the Scala syntax "We wanted the ability to express our mathematical concepts from the domain easily and naturally and have code that is somewhat like the models." and about interfacing to Java "In fact easier to call java libraries from Scala than it is from Java. With implicit conversions available in Scala."

About Electricité de France Trading (EDFT)

EDF Trading is a part of EDF and their interface to the electricity, gas, emmissions and freight market. In the past power companies produced power to meet their consumers demand. Their problem was essentially one of forecasting demand and making sure they had enough capacity on line to meet it. When the energy markets were liberalized companies like EDF did not have to be so strict about meeting power demand, they could always buy power from somewhere else. So power became a tradable thing. They could sell future capacity to other companies or set up contracts to buy future capacity for given prices. Power companies such as EDF, and consequentially the consumers, can gain from a much lower cost and better plant utilization. Many people are surprised that power is traded using financial instruments in the same way that some people trade regular stocks, shares or bonds. For example, a power producer can agree a contract with another company to provide 100 Mega-watts-hours for a given price in 10 months time. In turn sell this on to other companies. In all these trades no actual power production has yet taken place. Only futures have been traded. By doing this power companies are in effect balancing supply and anticipated demand across many suppliers.

The trader group at EDFT takes the real assets like coal or oil based power stations, gas storage units, hydro electric dams, emissions contracts, just about everything related to energy, and try to model them as financial products or derivatives rather than as physical things. This gives a whole new dimension to power delivery. The company monitors the power market and decides when they should generate power, sell it or buy to fill shortfalls in capacity. Sometimes, for example, it is cheaper to buy power on the market and turn a power station down or off, or leave the water in dams because it will be more valuable in the future, perhaps in as little as a week or two, or even several months later. Every asset has its own unique features, so they must support a variety of exotic complex options and each one needs to be modelled and priced.

Summary of Talk

Lee explains how they developed a system called Bluebird over a period of seven years. Six developers created around 300k lines of Java code that run the complex Monte Carlo simulations on a backend grid of several hundred computers. The GUI includes a domain specific external DSL called DIESEL which allows traders to easily specify complex derivative structures.

In December 08 they decided to start using Scala. Now they write all new classes in Scala and have replaced a significant part of the Java application by Scala. He describes how important it was to be able to interface with the existing Java code in order to manage the risk of introducing the new technology. Scala runs on JVM so no other investment was needed and it gave access to JVM-level tools like the profiler and Reflection. He says "In fact easier to call java libraries from Scala than it is from Java. With implicit conversions available in Scala."

He emphasized that they wanted the ability to express their mathematical concepts from the domain easily and naturally and have code that is somewhat like the models.

Some programmers were already familiar with functional programming. However, all the developers took to Scala quickly and appreciated it. There was a gentle learning curve. Rather than have one syntax for the model and the code something else as in Java. They could concisely express mathematical concepts using Scala features such as infix operators, pattern matching and closures. However, he cautions that eco system still favors C++. That there is more to it than getting the language right.

Lee describes DIESEL, an external DSL. He shows an example of a trade contract and how this can be written in a very natural DSL for traders. It allows them to price custom derivative structures where they trade physical and financial swaps or options with strike prices not fixed as is usual but are based on an arithmetic formula combining commodity and FX indexes.

He explains that the business needs a fast turn around of analytics for these custom derivatives. Rather than writing custom code for each derivative structure simulation they decided to implement a DSL in Scala that specifies the anaysis.

The full example of the a contract and the equivalent DIESEL can be found in his slides and a more detailed description of the conract structure in his talk. A sample of that DIESEL example follows:

:DELIVERY = Oct06..Sep09;
:POWER = 500000.0 thm/day;
:P0 = 13.20EUR/MWH;
:G0 = 185.000EUR/TONNE;
:F0 = 110.000EUR/TONNE;
:G = AVE(GAS_OIL_0_2PCT_FOB_ARA_BARGES,[9,0,1,:DELIVERY],Monthly) 
       * AVE(1/ECB_USD_EUR,[1,0,1,:DELIVERY],Monthly);
:F = AVE(FUEL_OIL_1PCT_FOB_ARA_BARGES, [6,0,1,:DELIVERY],Monthly) 
       * AVE(1/ECB_USD_EUR,[1,0,1,:DELIVERY],Monthly);
:PTFG = 0.7*(:G-:G0) + (0.1*(:G-:G0))^+;
:PTFF = 0.7*(:F-:F0) + (0.1*(:F-:F0))^+;
:P1_EUR_TON = :P0 + 0.55*0.007595*10TONNE/MWH*:PTFG + 0.45*0.007825*10TONNE/MWH *:PTFF;
:P1 = :P1_EUR_TON*29.3071MWH/1000THM;
:P2 = AVE(HEREN_ZEEHUB_GAS,[1,0,1,:DELIVERY],Monthly) * AVE(1/ECB_GBP_EUR,[1,0,1,:DELIVERY],Monthly);

Lee explains that Scala has a great parser combinator library. It is very easy to write a parser that looks close to the BNF representation. The parser was written in the functional style. However, for the the macro definitions that can be substitute later, they used imperative style, a map to store variables and return them when needed. It is nice to express this structure in an imperative style.

He adds that the ability to choose OO or functional style is very convenient, for example, defining the sematics for the DSL it was easier using pattern matching.

He emphasises that "Scala is very concise particulary for writing parsers/DSLs"

On performance he reports that the Monte Carlo times are comparable to C++ code, mainly due to using CERN's Colt library for doing linear algebra. He says "In fact it is easier to use Colt from Scala than Java."

He summarised by saying "We're using Scala for real work in a business critical situation and there is a lot of money at stake if we get it wrong. There are risks, but we have had no problems [with Scala]. For me a developer working at the coal face, the productivity improvements are massive."