Configure Eclipse IDE to work for both 32-bit and 64-bit R

Sometime we need to make Eclipse compatible simultaneously for both 32-bit and 64-bit versions of R. For example, we may have the 64-bit R as the default configuration. However, if we want to run the odbcConnectExcel function which only works under 32-bit scenary, we have to switch the current active configuration to 32-bit R. This tutorial tells how to make two configurations of 32-bit and 64-bit R coexist in one Eclipse IDE.

1. When install R, install both 32-bit and 64-bit versions.
   
   
2. Install Jave Runtime Environment (JRE), both 32-bit and 64-bit. Download both versions and install them respectively, using their default installation folders, so
   • 32-bit JRE automatically goes to folder C:\Program Files (x86)\Java\jre7
   • 64-bit JRE goes to folder C:\Program Files\Java\jre7.
3. Set up two versions of JREs in Eclipse
   
   
   
4. Set up two R environments in Eclipse
   
   Menu: Run --> Preferences --> StarET --> Run/Debug --> R Environments , assigning different names for different R environments (note: the R_HOME location are same for both environments)
   
   
   
   
   
   The most important setting here is that the 'Architecture' fields for the two environments have to be set as x86_64 and x86, respectively.
   
   
5. Create two R consoles in Eclipse<>
   
   Menu: Run --> Run Configurations --> R Console, right click 'New' to create two R consoles, as shown below
   
   
   
   
   
   Make sure to set correct versions of the R environments and JRE in the corresponding R consoles
   
   
   
   
   
   
   
6. To activate one R console, Run --> Run Configurations --> R Console --> Choose one console and click 'Run' button....that's it !!!

Read More

Use Eclipse as IDE for R under Windows

Background

Eclipse is an open source Integrated Development Environment (IDE). As with Microsoft's Visual Studio product, Eclipse is programming language-agnostic and supports any language having a suitable plugin for the IDE platform. For Eclipse, the R language plugin is StatET.

The StatET plugin enables Eclipse to connect to R. You may have been using R console or a simple editor like notepad or wordpad to write and submit R programs to R before. With Eclipse and the StatET plugin for R, you will use the rich editing environment provided by Eclipse and the StatET plugin to write and submit those programs to R from within the Eclipse IDE (Integrated Development Environment). The result will be a better, less error-prone development environment for your R software development.

Above figure shows Eclipse, StatET with R, and the R debugger (bottom window) at work. The R debugger is an R package library and has its own graphical output window separate from Eclipse.

Install Eclipse

• Download the latest stable Eclipse release (I use Eclipse Classic 4.2.1, 183 MB, as of 20-Oct-2012). For 64-bit Windows be sure to download and install the 64-bit version of Eclipse AND install the 64-bit JRE (Java Runtime Environment). Once downloaded and extracted, copy folders and files under the obtained Eclipse folder (for example, eclipse-SDK-4.2.1-win32-x86_64) to your desired installation location, such as C:\eclipse. Add a desktop shortcut to the Eclipse executable (c:\eclipse\eclipse.exe) if you like.
  
  
• Your system may already have Java installed, in which case you can skip these Java installation steps. 64-bit Java runtime environments (JREs) are designated explicitly on the java.com website as 64-bit, for example 'Windows 7/XP/Vista/2003/2008 (64-bit)' or similar. If you do not see the 64-bit JRE offering for Windows then you are likely running 32-bit Windows and should use the 32-bit JRE and Eclipse.
  
  
• If you are running a 64-bit Windows and you still do not see the 64-bit offerings on java.com then switch to another browser (64-bit).
  
  

Install StatET

1. From the Eclipse menu bar click Help -> Install New Software
2. Click the 'Add' button. The "Add Site" dialog appears
3. Type in a friendly name for your remote resource, such as StatET
4. Paste the URL (http://download.walware.de/eclipse-3.8, this is the latest version available at StatET website, as of 20-Oct-2012) into the 'Location' box, make sure to check the option of Contact all update sites during install to find required software, then click 'OK'
5. Select (check) the package components that you want to install, then click 'Next'
6. A review screen showing your selection(s) displays. Click 'Next'
7. Accept the license agreement, then click 'Finish'
8. Now Eclipse will install StatET, but it will take a few minutes. During the installation you may be asked to accept a digital certificate from Eclipse.org.

Configure the StatET Eclipse plugin

1. Install R, if you need to. NOTE: as far as Eclipse is concerned, it makes no difference if you run 32 or 64-bit R. Just be sure that you identify 32 or 64-bit R to Eclipse, which we do in the next steps (among other things).
   
   
2. From the Eclipse menu bar click Help -> Cheat Sheets: StatET: R in Eclipse to put up the official help. This is a handy, built-in guideline, but some of the steps inaccurately describe the menu choices (and one reason why I wrote these instructions). It is a good reference, though, for your later work, so keep it in mind. Click the 'Cancel' button to close the Cheat Sheet Selection dialog.
   
   
3. From the Eclipse menu bar click Window -> Preferences: StatET : Run/Debug: R Interaction: Select 'New Console Inside Eclipse' for the 'Connector used to run R code'. Note that this is the default so it might already be this way. This is how I run mine. Be sure to click 'Apply' if you change the setting.
   
   
4. Back in the left pane click 'R Environments' (StatET : Run/Debug: R Environments, which is directly above the 'R Interaction' of the previous step). The 'R Environments' pane now appears on the right.
   
   



• Click the pane's 'Add' button. The 'Add R Environment Configuration' dialog appears.
  
  
• Type in a friendly name in the 'Name' box, such as R.x64 2.15.1 or whatever your R version is, for example.
  
  
• For the 'Location (R_HOME)' box, paste in the path to your R install directory, for example C:\Program Files\R\R-2.15.1. Make sure the 'Architecture' field is set to your R type (32 or 64-bit), such as x86_64.
  
  
• Click 'OK' to save your changes and close the dialog, which returns us to the 'R Environments' pane, which now shows our newly created R environment.
  
  
• Click 'Apply' and 'OK' to save and close the 'R Environments' pane.
  
  
5. From the Eclipse menu bar click Run -> Run Configurations. The 'Run Configurations' dialog box appears.
• In the left pane click 'R Console', right click and select 'New' to define a new console entry with a friendly name in the 'Name' box, such as 'MyConsole'.
  
  
• Under the 'Main' tab ensure that 'Launch Type' is set properly for your configuration. If you are uncertain, choose 'Rterm' (this selection is VERY IMPORTANT).
  
  
• For 'Working Directory' just provide a path to your projects directory, for example C:\d2\SOFTWARE_DEVELOPMENT\Research\R\Projects
  
  
• No further changes are required, but you can check to make sure that the 'R Environment' under the next tab ('R Config') is set to that environment you named previously.
  
  
• Click 'Apply' and then 'Close' to close the dialog



6. Set perspective for R project If you can not see the R-project under File --> New, switch Eclipse's perspective to StatET by clicking Window --> Open Perspective --> Other --> StatET.

Testing the Eclipse/StatET/R IDE Configuration

1. From the Eclipse menu bar click File -> New -> R-Project.
   A 'New R-Project' dialog box appears, so give the new project a name like 'test'.
   
   
2. From the Eclipse menu bar, click File -> New -> R-Script File. Name your new script and click 'Finish'.
   A new script window appears in the Eclipse IDE.
   
   
3. Paste in or type some R test code, for example:
   
   

   # TODO: Add comment
   # 
   # Author: mark qu
   ################################################################
   # some test code...
   a = 2
   b = 3
   
   c = a+b
   c

   
   
4. To run the script, click inside the R script window to give it focus. Then, from the Eclipse menu bar, click Run -> Run : R Script in R Submitting Directly -OR- R Script in R via Source. You can then execute by simply clicking Run -> Run. You can also select the Run icon (green ball with the white triangle inside) from the Eclipse toolbar.

Read More

Import date from Excel spreadsheet into R using RODBC package

Package RODBC can be used not only for ODBC database connectivity, but also for importing Excel spreadsheet data into R, as shown in following R code:

library(RODBC)
data=odbcConnectExcel(file.choose())
mydata=sqlFetch(data,"Sheet1")
odbcClose(data)
mydata

Here, the odbcConnectExcel establishes a connection to an external Excel spreadsheet.

Note:

• Function 'odbcConnectExcel' is only usable with 32-bit R. Under Eclipse IDE under 64-bit Windows, you need to activate 32-bit R console in order to run above code.
• It is said 'odbcConnectExcel2007' works for 64-bit environment, but I had no luck to make it works.

Read More

Import data from Microsoft SQL server to R using RODBC package

As preparation, first create an ODBC driver connection for SQL Server by
Control Panel >> Administrative Tools >> Data Sources (ODBC)

Then import data from SQL server with following R code

library(RODBC)
conn <- odbcConnect(dsn="ACCPAC 54 SQL",uid="my_ID",pwd="my_PWD")
#sqlTables(channel)
queryResult <- sqlQuery(conn, "SELECT * FROM TableName")
odbcClose(conn)
dim(queryResult)

Alternatively, without creating ODBC driver connection, you can use following R code to directly set up connection

library(RODBC)
conn <- odbcDriverConnect("Driver=SQL Server; Server=xxx.xx.xxx.xx; Database=test_DB; Uid=my_ID; Pwd=my_PWD;")
#sqlTables(channel)
queryResult <- sqlQuery(conn, "SELECT * FROM TableName")
odbcClose(conn)
dim(queryResult)

Read More

Import data from Microsoft SQL server to R using RJDBC package

Environment

• Windows 7, 64-bit
• R 2.15.1 (32- & 64-bit)
• Microsoft SQL Server 2008

Installation & Usage

1. Download JDBC driver from Microsoft ( Link ) and install it so that RJDBC plugin is able to interact with SQL Server database via the Java interface.
• You can install the driver to a local folder, such as C:\Program Files\Microsoft SQL Server JDBC Driver 3.0
2. The latest RJDBC release is available from CRAN, install it using following command:

   install.packages("RJDBC",dep=TRUE)

3. Usage of JDBC

   drv <-JDBC (driverClass, classPath)
   conn <- dbConnect(drv, "jdbc:sqlserver://serverName", "userID", "password")

• driverClass --- name of the Java class of the JDBC driver to load. If empty, it is assumed that corresponding JDBC drivers were loaded by other means.
  For Microsoft SQL Server, driverClass = "com.microsoft.sqlserver.jdbc.SQLServerDriver"
• classPath --- class path that needs to be appended in order to load the desired JDBC driver. Usually it is the path to the JAR file containing the driver. For example, "C:/Program Files/Microsoft SQL Server JDBC Driver 3.0/sqljdbc_3.0/enu/sqljdbc4.jar"
• When specifying database server name, I tried following two ways, and looks like both worked
  • jdbc:sqlserver://xxx.xx.xxx.xx
  • jdbc:sqlserver://xxx.xx.xxx.xx:port;database=databaseName
4. Example code:

   library(RJDBC)
   drv <- JDBC("com.microsoft.sqlserver.jdbc.SQLServerDriver", "C:/Program Files/Microsoft SQL Server JDBC Driver 3.0/sqljdbc_3.0/enu/sqljdbc4.jar")
   conn <- dbConnect(drv, "jdbc:sqlserver://xxx.xx.xxx.xx:1433;database=My_MARKET_DATA", "admin", "123456789")
   #dbListTables(conn)
   sqlText <- paste("SELECT * FROM myTable", sep="")
   queryResults <- dbGetQuery(conn, sqlText)

Read More

Truncate minute and second timestamp from date value in SQL

Suppose we retrieve a variable 'Valuation_Date' from SQL database and get following number

2012-01-22 00:00:00.000

We can use convert function to truncate minute and second timestamp using the convert function

convert(varchar(8), Valuation_Date, 112)

then we will get following reault:

20120122

Read More

Matlab tip -- finding the indices of the elements of one array in another

Suppose all elements in array1 are contained in array2, now we want to get the indices of array1's elements in array2.

Here is a simple and neat solution, using Matlab's arrayfun function, which applies a function to each element of an array.

array1=[1,6,2,3,7];
array2=[3,6];
arrayfun(@(x)find(array1==x,1),array2)

The output is:
ans =
      4    2

Read More

A beginner's tutorial for R

• Some common commands

# Quit R
q()
q(save="no")
 
# Setting the default working directory
setwd(file="c:\\AnyDirectory\\")
setwd(file="c:/AnyDirectory/")
 
# Get the default working directory
getwd()
 
# See which variables are currently in the workspace
ls()
 
# Remove a list of variables
rm(var1,var2,var3,...)
 
# Remove all objects from workspace
rm(list=ls())
 
# Note: the variables whose names begin with a dot are not listed by ls(), so they are not removed by above command. 
# Instead, you can use following command to remove them
rm(list=ls(all=TRUE))
 
# Load a package
library(PackageName)
 
# Sink R output to a file
sink(FileName)
sink(NULL) # stop the sink



• You can use either single- or double-quote for text

> cat("I said:\"Hello!\"",'then smile.')
I said:"Hello!" then smile.
> cat("This is Eric\'s book,",'and that is Mike\'s toy.')
This is Eric's book, and that is Mike's toy.

Read More

Control Excel chart position with row and column numbers in Matlab

Sometimes we want to align, in Excel spreadsheet, a chart with a cell which, for example, may contain a label for the chart. We can easily drag (with mouse) the chart to make it aligned with the cell, but it may not be so easy to do this by program. The challenge here is that in Excel spreadsheet a cell position is expressed in row and columns, while a chart position is expressed in points. Depending on the screen resolution, the number of points corresponding to one row or column may vary on different computers. When we want to insert a large amount of charts into a sheet by program, exact positioning of these charts will be a headache. This tutorial teaches you how to programmingly obtain and position the chart in terms of row/column numbers.

1. Determine the chart position in terms of rows
A VBA chart Object has properties TopLeftCell and BottomRightCell which return the cells in the upper left and lower right corners of the chart, respectively. For example, following VBA code is used to retrieve the row property of cells pertaining to the chart position:

Dim TopRow As Long
Dim BottomRow As Long
With ActiveSheet.ChartObjects(1)
    TopRow = .TopLeftCell.Row
    BottomRow = .BottomRightCell.Row
End With



2. Determine the cell position in terms of points
We can use the Left and Top properties to determine the distances (of a cell, row or column) to the left and top boundaries of the spreadsheet, in the unit of points. Here is an example using VBA code

Worksheets("Sheet1").Range("A5").Top
Worksheets("Sheet1").Range("B7").Left
Worksheets("Sheet1").Columns(6).Left
Worksheets("Sheet1").Rows(11).Top


Following Matlab code shows how to insert 10 charts into an excel spreadsheet, each having a figure number printed in a cell beside it:

ObjExcel = actxserver('Excel.Application');
TheWorkbook = ObjExcel.Workbooks.Open('example.xls');
DataSheet = TheWorkbook.Sheets.Item('Data');
RowLastChartBottom = 0;
for i = 1 : 10
    RowChartTitle = RowLastChartBottom + 2;
    DataSheet.Range('A', num2str(RowChartTitle)).Value = ['Chart ', num2str(i)];
    %---
    RowChartTop = RowChartTitle + 1;
    PointsChartTop = ChartSheet.Range(['B', num2str(RowChartTop)]).Top;
    ChartObject = ChartSheet.ChartObjects.Add(10, PointsChartTop, 1000, 250);
    ChartObject.Chart.HasTitle = 1;
    ChartObject.Chart.ChartTitle.Text = '......';
    ChartObject.Chart.Axes.Item(1).TickLabelPosition = 'xlTickLabelPositionLow';
    %---
    CurveObject = ChartObject.Chart.SeriesCollection.NewSeries;
    XRange = DataSheet.Range(......);
    YRange = DataSheet.Range(......);
    CurveObject.XValues = XRange;
    CurveObject.Values = YRange;
    CurveObject.ChartType = 'xlLineMarkers';
    CurveObject.MarkerSize = 2;
    CurveObject.Format.Line.Weight = 2;
    %---
    RowLastChartBottom = ChartObject.BottomRightCell.Row;
end

Read More

A latex template for writing an technical article

\documentclass[8pt,onecolumn,letterpaper]{article}
\usepackage[top=0.5in, bottom=0.5in, left=1.0in, right=1.0in]{geometry}
\setlength{\columnseprule}{0.25pt}
\usepackage[utf8x]{inputenc}
\usepackage{ucs}
\usepackage{amsmath}
\usepackage{amsfonts}
\usepackage{amssymb}
\usepackage{bm}

\setlength{\parindent}{0in}
\setlength\parskip{0.1in}
\linespread{1.25}

\usepackage{graphicx}
\usepackage{color}
\usepackage[colorlinks=true, pdfstartview=FitV, linkcolor=blue, citecolor=blue, urlcolor=blue]{hyperref}

%\usepackage{fancyhdr}
%\pagestyle{fancy}
%\renewcommand{\headrulewidth}{0.0pt}
%\fancyhead{}
%\fancyfoot[CF]{\thesection - \thepage}

\author{Quant Lego}
\title{This is an article template}

\begin{document}
\maketitle

\tableofcontents
\pagenumbering{roman}

\newpage
\pagenumbering{arabic}

\section{Introduction}

\section{Methods}

\section{Results}

\section{Discussions}

\section{Conclusions}

\end{document}

Read More

Southeast and Gulf Coast electricity markets

• Unlike regions with independent power grids, power in the Southern states is provided by integrated power companies. Each of these companies is linked to nearby grids and power can be traded between them. However, because these are still regulated markets, there is little opportunity to trade power within the service area of any of these companies.
• The Gulf Coast region is characterized by mild winters and prolonged hot summers. Long-lasting heat waves can start affecting the area inearly spring and may last until late October ==> Seasonal demand is quite different from the northern states --- demand is low most of the winter and peaks for an extended time in the Summer.
• From a precipitation perspective, the southeastern United States has a tropic climate.
• There is a wet season running from November to May, and a dry season running from June to October.
• There is substantial hydroelectrical generation in the Tennessee Valley area that is affected by this rainfall. During the period of high rainfall, the Tennessee Valley can export power to other parts of the Southeast, the Midwest, or the East Coast.
• Coal and natural gas are both marginal fuels for the area. There are ready supplies of both fuels locally --- the Gulf Coast is a major source of natural gas, and the Appalachian Mountains contain abundant coal deposits.

Read More

New England electricity market (ISO-NE)

• There is only a single trading hub in the New England region. The price at thsi hub are based on selected nodes around the Boston area.
• New England has to import fuel for almost all of its power generation requirements.
• The marginal fuel for the region is natural gas. The bulk of this supply comes from the U.S. Gulf Coast and eastern Canada.
• There are efforts to augment this supply by constructing liquefied natural gas terminals in Massachusetts.
• Most of the population in New England is in the southern part of the region: Connecticut and the area around the city of Boston, Massachusetts. In contrast, the northern part of the region is relatively unpopulated.

Read More

New York electricity market (NYISO)

• The New York power grid is self-contained in the state of New York.
• The southeastern part of this region the most heavily concentrated areas of demand for electricity in the country: New York City and Long Island.
• The region is characterized by the flow of low cost power from the northern and western sections of the state into the high demand regions surrounding New York City.
• Most of New York metropolitan area is located on islands ==> (1) there is limited ability to transfer power to those areas from other parts of the region; (2) high reliance on inefficient generators during periods of peak demand. ==> New York City area is one of the most expensive power market in the country.
• To assure reliable service, power plants in the New York metropolitan area are subject to much stricter rules and regulatory requirements than are common in other regions.

Read More

Midwest electricity market (MISO)

• The Midwest ISO coordinates power for the north-central United States
• The eastern part of the MISO region is heavily industrialized.
• The western section of th MISO region is more rural.
• There are several major trading locations in the region. among them Cinergy Hub is the most influential.
• From a climate perspective:
• The middle of the United States has cold winters and hot summers.
• The cold weather usually starts in December, but can arrive in early November and usually lasts through April.
• The summer months, July and August, are hot and humid, but the weather cools off quickly in September.
• Many of the power plants in the region, including a large number of marginal producers, are coal powered ==> Coal prices often dominate the price of power in the Midwest even in high demand periods. This is a unique feature of the area (In most of United States, the power price is set by natural-gas-fired plants).
• Power is often wheeled from the Midwest to adjacent regions when the price of natural gas spikes upward.
• Cool shoulder months (the spring and fall) in the Midwest ==> Power is often wheeled to southern states where hot weather is more common.
• Some hydroelectric power is available from the Great Lakes and Tennessee Valley areas. Depending on the time of the year, power may be imported from either of these two regions.

Read More

Mid-Atlantic electricity market (PJM ISO)

• The Mid-Atlantic region contains many large population centers, like Washington D.C., Baltimore, and Philadelphia, on the eastern seaboard.
• This area formed one of the first independent power grids, the PJM ISO, named after its condtituent states: Pennsylvania, New Jersey, and Maryland.PJM has since expanded to include West Virginia, Delaware, large parts of Ohio, parts on northern Illinois, and Virginia.
• It is one the largest and most liquid electricity markets. A number of active trading hubs are located within PJM.
• Geographically, the PJM service area is very diverse:
• Large reserves of coal native to Pennsylvania and West Virginia serve as the marginal fuel for western part of the PJM service area.
• Power on the East Coast and Virginia is commonly determined by gas-fired power plants.
• Climate:
• The climate in the northern half of the region ranges from cold winters to hot summers.
• The climate in the southern half of the region is warmer --- mild winters and extended hot summers.
• A large number of power plants in the less populated areas of the PJM service area export power into the more populated regions ==> the power lines between these areas are often congested.

Read More

Pacific northwest electricity market

• In most years, the Pacific Northwest sells surplus power into California and the Southwest.
• About two-thirds of all the electricity in this region comes from hydroelectric production ==> the quantity of surplus power depends heavily on precipitation ==> Water flow in this region directly affects the price of power in California.
• Melting snow from the Cascade Mountain range, which runs through central Washington and Oregon, is a major source of river water.
• To the west of the mountains are several major population centers: Seatle, Washington, and Portland, Oregon. The climate in these regions is fairly temperate all year.
• To the east of the mountains is the start of the northern plains state. This area is characterized by cold winters and hot summers.
• Mid Columbia (MIDC): is a delivery hub for a number of hydroelectric plants on the Columbia River.
• California-Oregon Border (COB): is a major switching station in southern Oregon, adjacent to the California border, that interconnects several major transmission lines.

Read More

Desert southwest electricity market

• The desert southwest has moderate winter and prolonged hot summers
• Mush of the year, the area has a surplus of generation capacity and is a major exporter of power to South California.
• Natural gas is the most common marginal fuel due to the abundant natural gas reserves, but coal and hydropower are relatively common too.
• The area also receives a great deal of sunlight, making it an attractive location for solar power.
• There are three major hubs for electricity trading in the area, and major power generation facilities are located near each hub.
• Mead: is located on the outskirts of Las Vegas. Lake Mead is the artificial lake formed when the Hoover Dam was built on the Colorado River. The Hoover Dam is a major source of hydroelectric power and the primary delivery point for power in southern Nevada.
• Palo Verde: is located at a major nuclear power plant and switching yard about 45 miles west of Phoenix, Arizona. Many high power lines to California are located at this switching yard.
• Four Corners: is located in northwestern New Mexico at the interconnection of several major transmission lines.

Read More

California electricity market (CAISO)

• California is a heavily populated state that is divided into two major zones: NP-15 and SP-15.
• Most of the year, the daily temperatures are moderate to warm. However, the dense population and hot summers can cause spikes in the electrical loads due to a demand for air conditioning.
• One of the major transmission lines, Path-15, bisects the state into two approximately equal parts and has given its name to the two major zones: NP-15 and SP-15.
• NP-15 (North Path 15)
• Covers the northern part of the state (San Francisco Bay area), north of Path 15.
• When rainfall or melting snow allows, this region imports low cost electricity from the Columbia River Basin in Oregon.
• For the remainder of the year, natural gas plants, supplied primarily from western Canada, set the price of power in NP-15.
• SP-15 (South Path 15)
• South of Path 15, the pricing zone for most of Southern California.
• A major importer of power from Arizona (natural gas and solar power) and Nevada (hydropower from the Colorado River).
• Natural-gas-fired plants in the south are fed from supplies in the rocky Mountain and West Texas regions.
• ZP-26
• A large but less important pricing zone that has a relatively low population and excess generation capacity.
• It connects to both the NP-15 and SP-15 zones.

Read More

Regional power markets in United States

There are three major integrated power grids in the United States:

• Eastern Interconnect --- east of the Rocky Mountains
• Western Interconnect --- west of the Rocky Mountains
• Texas Interconnect --- Texas

Inside each interconnection, all of the transmission lines are synchronized ==> allows power to be transported across long distances within these interconnections.

Electrical prices vary regionally with an interconnection depending on regional population, industries, and weather conditions ==> Long distance transmission of power, called wheeling, between fundamentally different markets is the source of large number of trading opportunities.

Texas Interconnect

• Smallest interconnection in the United States.
• Entirely located within the state of Texas.
• Coordinated by a single ISO, the Electric Reliability Council of Texas (ERCOT).
• There are a number of DC power lines, of limited capacity, connecting the Texas power grid to the Eastern Interconnect and the Mexican power grids.

Western Interconnect

• The West Interconnect is a long line of regional grids stretching from the Pacific Northwest into the Desert Southwest, with California in the middle ==> Much of the trading in the Western Interconnect revolves around the flow of power into or out of the highly populated areas of California.
• The pacific Northwest (MIDC) and Colorado River (Mead) near las Vegas are major sources of hydroelectric power.
• 70% power supplies in Washington and Oregon come from hydroelectricity ==> small changes in water flow can result in substantial changes to regional power supply ==> Snow melts during Spring lead to more power supply than demand in Pacific Northwest, so some power is transmitted to areas of higher demand (usually California).
• California is a densely populated area: in summer it needs import of power from Pacific Northwest and Desert Southwest; in Fall mild temperature makes it possible to export power to still very hot areas of Las Vegas and Phoenix.
• The marginal power producers in California are usually natural-gas-fired plants ==> a steady supply of natural gas is required: southern California imports natural gas from the Rocky Mountains and West Texas, and northern California import natural gas from west Canada.

Eastern Interconnect

• It includes the two most important RTO/ISOs: PJM (originally named the Pensylvania, New jersey, and Maryland Interconnection) and MISO (Midwest ISO)
• The two most liquid trading locations in the Eastern Interconnect are Cinergy Hub (abbreviated MISO-Cin Hub) and PJM Western hub (PJM-W).
• Varieties in fuels and regional weather conditions provide many trading opportunities in the East Interconnect:
• Some important trades are between regions using col-fired plants and those using natural-gas-fired plants
• Other trades are weather related.
• The southern part of Eastern Interconnect has extended summer and mild winter, while the northern part has colder winter and shorter summer air-conditioning season.

Read More

Heat rate and spark spreads

Heat rates

• Heat rate represent the efficiency of a power plant to convert fuel into electricity ==> Lower heat rate implies a more efficient power plant.
\[\text{Heat Rate}=\frac{\text{Quantity of fuel used}}{\text{Quantity of power produced}}\]
• The unit of heat rate is Btu/KWh or MMBtu/MWh, with 1 MMBtu/MWh = 1000 Btu/KWh
• For example, heat rate of 8 MMBtu/MWh means a natural-gas-fired power plant will sell power profitably when the power price is 8 times of the natural gas price
• Market implied heat rate
\[\text{Market implied heat rate}=\frac{\text{Power price}}{\text{Fuel price}}\] with the unit also being Btu/KWh or MMBtu/MWh.
• As a rule of thumb, natural gas power plants commonly have heat rates between 7 ~ 10 MMBtu/MWh.
• Heat rate extremely efficient
• Heat rate > 10 MMBtu/MWh ==> less efficient

Spark spreads

• Spark spread is the theoretical profit that a natural-gas-fired power plant can make from buying fuel and selling power at current market prices, ignoring any charges for operational cost.

Spark Spread = Power Price - (Gas Price × Heat Rate)

• Below tables shows an example of heat rates and spark spreads that are used by traders to approximate the behavior of electricity generators.



• Note: for coal-burning power generator, the corresponding spread is called dark spread.

Read More

Design of electrical trading markets

Spot vs forward power markets

• The forward power market is where the bulk of speculative tradings occur, while the spot power market is where daily auctions occur.
• The forward market is open to anyone with sufficient money to meet trading requirements, while the spot market is open only to power providers with the ability to generate power and place it to the transmission grid.
• In forward market both physical contracts (requiring delivery of power) and financial contracts (settle in cash) are traded, while spot market involves only physical contracts.
• In spot market it is possible to buy spot power in arbitrary small sizes for immediate use anywhere in the country, while in the forward market it is only possible to trade large (month) blocks at a limited number (~20) of locations around the country.
• Trading power in large units at a limited number of locations has two consequences:
• Pro: it is easier to standardize contracts and find trading partners
• Con: trading prices at extended geographic areas around a major hub may not represent the exact price of power anywhere in the region ==> Power is often traded in two pieces: (1) standardized forward trades made at major hub to get the desired regional exposure approximately correct; (2) smaller fine-tuning trades to lock in an exact price at the specific location.

Market design under LMP methodology

• Congestion charge: In periods of heavy demand, power lines can become overloaded ==> The primary way of rerouting power around congestion is to activate power plants close to the area of high demand even though their costs may not be low enough to enter winning bids of auction (out-of-merit-bid) ==> Under the FERC's Standard Market Design (SMD), the cost line congestion are paid by the affected parties (in a single location), rather than being shared by every user of the entire grid.
Congestion costs aren't just paid by consumers, power producers pay them too. There is a charge for routing power into a high load area over congested power line, and a credit for producing power that bypasses the congestion.
• Line loss charge: Under the FERC's SMD, power produced only get paid for deliverable power, not on the gross power placed into a power grid.
• Locational Marginal Price (LMP) = clearing price (same everywhere in a power grid) + congestion charge (location specific) + Line loss charge (location specific) ==> Different prices are assigned to different locations in a power grid ==> location has become a major factor determining the profitability of a power plant, in addition to fuel cost and efficiency ==> There is economic incentive for power producers to relieve both line loss in transmission and line congestion ==> Power grid reliability increased significantly (compared with unregulated markets).
• There are three types of locations for calculating prices under SMD:
• Node prices: corresponds directly to the power price at an 'electrical bus', where power enters or leaves the transmission grid. Power generators are paid the nodal price of the electrical bus where they deliver power into the transmission grid.
• Zone price: is the average of all nodal prices within a limited geographical area. This is the price that customers pay for the power they receive.
• Hub price: is an average of selected nodal prices across several zones, and it serves as the benchmark price for a power grid ==> Hub prices are used extensively in forward market tradings. In most ISO/RTO regions, the clearing price for power and the hub price are synonymous.
• Financial Transmission Right (FTR)
• Here is a nice introduction about FTR.
• FTR is a financial instrument closely linked to Locational Marginal Price (LMP). It helps customers manage the price risk of having purchased or sold power at a major hub and then being forced to pay a different price when they deliver or receive power at a specific node.
• FTRs are tradable contracts made between two parties, which take opposite sides of an obligation to pay or receive the difference in price between two nodes. One party need to pay the other when there is a congestion.
• Sometimes FTRs can be structured as options, acting as insurance against adverse price change due to congestion.
• Trading products of power in terms of hours: weekdays x hours
• Peak: daytime hours, 7 am ~ 11 pm, examples are 5x16, 7x16
• Off-peak: nighttime hours, 11 pm ~ 7 am, examples are 5x8, 7x8

Read More

Overview of electricity markets

• It is hard to store electricity and expensive to transmit electricity over long distance ==> No unified national electricity market, instead just a collection of small regional markets with their own unique characteristics and regulations.In each market, supply and demand must constantly be matched, resulting in highly volatile prices.
• Regulations of regional markets
• Each regional is coordinated by its own Transmission Service Operator(TSO).
• In regulated markets, TSOs are government-sponsored monopolies
• In deregulated markets, TSOs are either Independent Service Operators (ISOs) or Regional Transmission Organizations (RTOs)
• ISOs are limited to doing services in a single state and are exempt from federal jurisdiction.
• RTOs are do business across several states and fall under federal jurisdiction.
• Some RTOs began in a single state as ISOs and became RTOs when they expanded across state boundaries, but they still keep ISO as part of their names.
• Deregulated markets use economic incentives to effect changes, while regulated markets use legislative mandates.



• In deregulated markets, the power price for a transmission grid is set by daily auctions
• Power producers submit the price at which they are willing to supply power, and are activated in order from the lowest to highest bid.
• All winning bidders get paid the same price regardless of their bids ==> non-discriminatory auctions
• The power price for every producer and every wholesale consumer is set to a single price ==> clearing price or wholesale price.
• Smaller customers pay a slightly higher power price ==> retail price
• The cost of bringing the last unit of electricity into the market is called the marginal price, and the most recently activated plant is the marginal producer ==> In deregulated markets, the clearing price is set by the marginal price of power.
• ISOs coordinate two types of auctions
1. Day-ahead auction (DA)
• sets the power price for the following day in one-hour increments
• commonly completed in the early afternoon on the day before delivery ==> allows power producers have sufficient time to arrange fuel and operating schedules for the delivery day
• this auction is based on a prediction of next days' demands
2. Real-time auction (RT)
• runs continuously throughout the actual delivery day
• balances the actual demand against the prediction of demand made in the previous day
• typically bid in five-minute increments
• Power plants participating in real-time auctions must have the capability of turning on and off quickly

Read More

Factors affecting natural gas prices

Natural gas prices are a function of market supply and demand. Because of limited alternatives for natural gas consumption or production in the short run, even small changes in supply or demand over a short period can result in large price movements to bring supply and demand back into balance. See Today in Energy, June 11, 2012 for recent examples.

Factors on the supply side that may affect prices include:

• Variations in the amount of natural gas being produced
• The volume of gas being imported and/or exported
• The amount of gas in storage facilities (referred to as storage levels)

Increases in supply result in lower prices, and decreases in supply tend to increase prices.

Factors on the demand side that may affect prices include:

• The level of economic growth
• Variations in winter and summer weather
• Oil prices

Higher demand tends to lead to higher prices, while lower demand tends to lead to lower prices.

Domestic Supply and Prices Can Be Cyclical

Most of the natural gas consumed in the United States comes from domestic production. U.S. production increased from 2006 through 2011, when it reached the highest recorded annual total since 1973. The increases in production were the result of more efficient, cost-effective drilling techniques, notably in the production of natural gas from shale formations.

Increased natural gas supply tends to dampen prices. In turn, lower prices can erode incentive for drilling, which eventually results in decreased production.

Severe Weather Can Disrupt Production

Hurricanes and other severe weather can affect the supply of natural gas. For example, in the summer of 2005, hurricanes along the U.S. Gulf Coast caused the equivalent of about 4% of U.S. total production to be shut in between August 2005 and June 2006.

Imports Contribute to our Supply of Natural Gas

In 2011, U.S. pipeline imports amounted to almost 13% of total natural gas consumption. More than 99% of the pipeline-imported natural gas came from Canada, with the remainder from Mexico. U.S. pipeline imports are expected to decline in the future because of robust U.S. production.

Economic Growth Can Affect Natural Gas Demand and Prices

The strength of the economy is a major factor influencing natural gas markets. During periods of economic growth, the increased demand for goods and services from the commercial and industrial sectors generates an increase in natural gas demand. This is particularly true in the industrial sector, which is the leading consumer of natural gas as both a plant fuel and as a feedstock for many products such as fertilizer and pharmaceuticals. The increased demand can lead to increased production, and, in general, higher prices. Declining or weak economic growth tends to have the opposite effect.

Winter Weather Strongly Influences Residential and Commercial Demand

During cold months, residential and commercial end users consume natural gas for heating, which places upward pressure on prices as demand increases. If unexpected or severe weather occurs, the effect on prices intensifies because supply is often unable to react quickly to the short-term increased level of demand. These effects of weather on natural gas prices may be exacerbated if the natural gas transportation system is already operating at full capacity. Under these conditions, prices tend to increase, which reduces overall demand for natural gas and brings the market into balance. Natural gas supplies that were placed in storage during periods of lesser demand may be used to cushion the impact of high demand during cold weather.

Hot Summer Weather Can Increase Power Plant Demand for Gas

Temperatures also can have an effect on prices in the cooling, season as many electric power plants that are operated to meet air-conditioning needs in the summer are fueled by natural gas. Hotter-than-normal temperatures can increase natural gas demand and push up prices.

Natural Gas Supplies Held in Storage Play a Key Role in Meeting Peak Demand

The overall supply picture is also influenced by the level of gas held in underground storage fields. Natural gas in storage is a critical supply component during the heating season that helps satisfy sudden shifts in supply and demand, accommodates stable production rates, and supports pipeline operations and hub services. Levels of natural gas in storage typically increase during the refill season (April through October), when demand for natural gas is low, and decrease during the heating season (November through March), when space heating demand for natural gas is high. Natural gas in storage represents an incremental source of supply immediately available to the market. This can counteract the effects of sudden increases in demand for natural gas or supply disruptions such as pipeline outages that cause demand to exceed supply and thus lead to higher prices.

Competition With Other Fuels Can Influence Natural Gas Prices

Large-volume gas consumers (primarily industrial consumers and electricity generators) can switch between natural gas, coal, and oil, depending on the prices of each fuel. Because of the interrelationship between these fuel markets, when prices of the other fuels fall, any resultant shift in demand from natural gas to coal or oil reduces gas demand and pulls gas prices downward. When prices of the competing fuels rise relative to natural gas prices, there may be switching from the competing fuels to natural gas, increasing the demand for natural gas and pushing gas prices upward.

Although most of the switching occurs between natural gas and oil, natural gas and coal markets can also interact when the price of natural gas falls significantly. Electricity generation using natural gas can become attractive relative to coal-fired electricity generation in some areas of the country when the price of gas on an energy equivalent basis becomes less than the price of coal.

Read More

The price of natural gas

The price that residential consumers pay for natural gas has two main parts (all cost components include a number of taxes):

1. Commodity costs — the cost of the natural gas itself, known as the wellhead cost.
2. Transmission and distribution costs — the cost to move the natural gas by pipeline from where it is produced to the customer's local gas company, and to bring the natural gas from the local gas company to the customer's house. These transmission and distribution costs account for the difference between the price of natural gas at the wellhead and the price paid by residential customers.

The shares of these two cost components vary according to natural gas market conditions.

The low wellhead price in recent years reflects several market conditions:

• Record levels of production due to the development of gas from shale formations (plays)
• Growth in supply exceeding growth in demand
• Few supply disruptions as a result of relatively calm hurricane seasons
Residential prices also reflect market forces including:
• Pipeline constraints to high-demand areas, particularly in the Northeast
• Net producing regions tend to be geographically removed from net consuming regions

Average Natural Gas Prices in the United States Although national average prices for residential natural gas have declined in recent years, the prices in individual states can differ greatly. These differences are related to a number of factors:

• the market's proximity to the producing areas
• the number of pipelines in the state
• average consumption per residence receiving service
• transportation charges associated with delivery
• state regulations
• degree of competition

How Can Residential Customers Reduce Their Natural Gas Bills?

To help reduce their natural gas bills, residential customers can:

• Shop for lower-priced natural gas, if their state has customer choice programs. Find more information on the status of natural gas residential choice programs in each state.
• Participate in their local gas company's yearly budget plan to spread gas costs evenly throughout the year, thereby lessening the impact of higher prices.
• Check natural gas appliances and space-heating equipment for efficient operation.
• Obtain a home energy audit to identify ways to conserve energy.
• Reduce thermostat settings, especially when they are not at home.

In addition, both federal and state energy assistance programs are available to natural gas customers who have a limited budget. For example, the Low Income Home Energy Assistance Program (LIHEAP) is a federal program that distributes funds to states to help low-income households pay heating bills.

Additional state energy assistance and fuel fund programs may be available to help households pay energy bills during a winter emergency. To find out if you qualify for assistance in your state, contact your state public utility commission or your local gas company.

Read More

Natural gas consumption in United States

Natural Gas Is a Major Energy Source for the United States

About 25% of energy used in the United States came from natural gas in 2011. The United States used 24.37 trillion cubic feet (Tcf) of natural gas in 2011.

How Natural Gas Is Used

Natural gas is used to produce steel, glass, paper, clothing, brick, electricity, and as an essential raw material for many common products. Some products that use natural gas as a raw material are: paints, fertilizer, plastics, antifreeze, dyes, photographic film, medicines, and explosives.

Slightly more than half of the homes in the United States use natural gas as their main heating fuel. Natural gas is also used in homes to fuel stoves, water heaters, clothes dryers, and other household appliances.

The major consumers of natural gas in the United States in 2011 included:

• Electric power sector — 7.6 trillion cubic feet (Tcf)
• Industrial sector — 6.8 Tcf
• Residential sector — 4.7 Tcf
• Commercial sector — 3.2 Tcf

Where Natural Gas Is Used

Natural gas is used throughout the United States, but the top natural gas consuming states in 2010 were:

• Texas
• California
• Louisiana
• New York
• Florida
• Illinois

Read More