Residual Analysis

Sprawling Cities Getting Hotter Faster

2010-06-26T11:26:00.000-07:00

An interesting new paper has been published (ahead of print) in Environmental Health Perspectives: Stone et al. (2010). It's being widely reported in the media (e.g Yahoo! News.) It finds that:

Our results find the rate of increase in the annual number of extreme heat events between 1956 and 2005 in the most sprawling metropolitan regions to be more than double the rate of increase observed in the most compact metropolitan regions.

The primary author, Brian Stone, is also quoted by OurAmazingPlanet as follows:

"These findings show that the pace of climate change is greater in sprawling cities than in others, which has not been shown before."

The study uses "urban sprawl" as the independent variable, which is more sophisticated than simply using a city's population size. That said, these findings essentially confirm what I stated in my post titled Urban Heat Island Effect - A Model. Briefly, I had determined that the UHI effect on the 130-year temperature trend depends logarithmically on the size of a station's associated town, but only if the town's population is greater than about a million people.

Intensity or Frequency?

2010-04-07T17:36:00.000-07:00

I have previously argued that – in my estimation – there's a strong causal association between sea-surface temperatures and the number of named storms (or tropical cyclones) in the Atlantic Basin. Statistically, the association is quite significant, and graphically, it is evident once you apply very simple smoothing filters.

This is not the prevailing scientific view, which essentially says that the intensity of storms should increase with global warming, and the frequency of storms should actually decline. This prevailing view, largely based on computer modeling and not observations, is best summarized by the IPCC in 4AR WGI 10.3.6.3.

I was wondering if both could be true: global warming increases the frequency and intensity of tropical cyclones. How can we test this idea using available observations? You can't just look at, say, Accumulated Cyclone Energy (ACE.) If the frequency of storms increases, ACE should also increase, even if the average intensity of each storm doesn't change.

It occurred to me that a much better test would be to look at the ratio of hurricanes to all named storms, and the ratio of major hurricanes to storms. I've done this, but I'll leave it as an exercise for the reader. It's a very easy analysis. You can use the named storm count data from the Hurricane Research Division of NOAA. If you have concerns that tropical storms were under-counted in the past relative to hurricanes (a reasonable assumption), you can use data starting in 1944, which is when systematic aircraft recognizance started. But remember, causality matters more than the trend in this case.

To make a long story short, observations do not appear to support the view that global warming will cause storm intensity to increase. The historical data is telling me the opposite of what the IPCC claims. What, if anything, am I missing? Could it be that things will work differently in the future?

Urban Heat Island Effect - A Model

2010-04-06T15:01:00.001-07:00

In the addendum of my last post on the Urban Heat Island (UHI) Effect, I noted that GHCN v2 apparently does contain data that we can use to verify the existence of the effect, even though UHI doesn't seem to have a discernible impact on global temperature trends. This is interesting because it's at odds with some well known findings from the literature, such as Peterson et al. (2003), and it addresses a "mystery" of sorts about the instrumental temperature record.

I wrote some code in order to carry out a more thorough analysis of a possibly systematic effect in the raw data, hypothesizing the effect depends on the size of the station's associated town. Basically, I divided stations in population size groups, using 1.25-fold increments. That is, the first group consists of towns whose population is between 10,000 and 12,500. The second group has between 12,500 and 15,625 people, and so on. The last group consists of towns with populations between 15.8 million and 19.7 million. For each group, I got a global temperature series, in a way equivalent to how GHCN Processor would produce them. This is what I came up with:

This is a highly significant effect. It doesn't even make sense to post a confidence level, because it's exceedingly close to 100%.

It is obvious from the graph, nonetheless, that the number of cities declines rapidly with population size. It's a good idea in these cases to look at a logarithmic scale of the X axis.

This logarithmic relationship is clearly a good candidate for segmented regression. When the population is less than about 1.04 million, there is no discernible effect. A linear regression of the left-hand "segment" has a slight downward slope, which is not statistically significant. The average temperature slope between 1880 and 2009 is 0.0056°C/year (which is what the red line represents.)

We can thus derive a straightforward model for UHI, applicable to the GHCN v2 raw data file, which follows.

C = -0.0039·[ln(P + 1) - ln(1042)]
Where:

C is a correction (in °C/year) that should be added to the temperature slope of a station only if the population of its associated town is greater than 1.04 million.
P is the population of the town associated with the station, in thousands.

Urban Heat Island Effect - Probably Negligible

2010-04-05T07:40:00.000-07:00

Previously I had discussed the difference between rural and urban temperature stations in the U.S. Commenter steven argued that population assessments (R, S and U) in GHCN v2 might be outdated and – in general – not very good proxies of what we really want to measure.

I then compared rural stations in the Mid-West (a low-population-density region of the U.S.) with all rural stations. There wasn't a major difference between these two sets of stations either. Commenter steven was not convinced, however. He posted some satellite pictures of rural stations that are located in what appear to be sub-urban areas.

How could we measure the impact of human populations on station temperature with the data available to us? It's clearly not enough to express doubt and speculate about what might be going on.

Here's what I came up with. There's a vegetation property in the station metadata. If you look at stations in regions that are forested (FO), marshes (MA) or deserts (DE), they appear to be actually rural. I looked at a subset of such stations in Google Maps, and they are not close to human settlements, with few exceptions. The GHCN Processor command I used to obtain a temperature series is the following.

ghcnp -dt mean -include "population_type=='R' && (vegetation=='FO' || vegetation=='MA' || vegetation=='DE')" -o /tmp/global-rural-plus.csv

575 stations fit these characteristics. For comparison, I got temperature series for big cities (population > 0.5 million), and small towns and cities (population <= 0.5 million.) I calculated 12-year moving averages in each case, which is what you see in the figure below.

There might be some differences, but they are always small, and we've compared several different stations sets now, globally and at the U.S. level.

An argument could also be made that small human settlements increase the albedo of an area, so they might have a cooling effect.

Addendum (4/5/2010)

Here's an actual UHI finding of interest. I compared cities of population over 2 million with towns whose population is between 10,000 and 15,000. The difference is more pronounced in this case.

The overall effect is still negligible, nevertheless. The number of cities decreases exponentially with population size.

US Rural vs. Urban Temperature Stations

2010-03-30T10:12:00.001-07:00

There's an article by Edward R. Long titled "Contiguous U.S. Temperature Trends Using NCDC Raw and Adjusted Data For One-Per-State Rural and Urban Station Sets." It claims to show that in the raw/unadjusted NCDC data, urban U.S. stations have a warming trend that diverges from that of rural stations, whereas in the adjusted data, the rural trend has been adjusted to "take on the time-line characteristics of urban data." In not so many words, it claims that NCDC data has been fudged. Not surprisingly, Long's article appears to be quite popular in "sceptic" circles.

The methodology of the article is peculiar. First, why only analyze the U.S.? Even though the U.S. has more stations than any other single country, its surface area is only 2% that of Earth.

More importantly, why pick only one rural and one urban station from each state? What was the criteria used to pick each state's stations? Was it random? The article does not clarify, so it lends itself to accusations of cherry-picking.

It's fairly easy to verify Long's claims with GHCN Processor. A quick verification takes perhaps 10 minutes if you're familiar with the tool's options.

First, we can get rural and urban temperature anomaly series for the U.S. from the adjusted data, with the following commands, respectively:

ghcnp -include "country eq 'UNITED STATES OF AMERICA' && population_type eq 'R'" -reg -o /tmp/us-rural.csv

ghcnp -include "country eq 'UNITED STATES OF AMERICA' && population_type eq 'U'" -reg -o /tmp/us-urban.csv

GHCN Processor informs us that it processed 867 rural stations, and 117 urban stations. That's plenty more than the article analyzes. Let's take a look at a graph of both series.

This is still consistent with Long's claims. What we want to confirm is whether the rural trend is significantly less steep in the raw unadjusted data. To get rural and urban series from the unadjusted data file, I used the following commands, respectively:

ghcnp -dt mean -include "country eq 'UNITED STATES OF AMERICA' && population_type eq 'R'" -reg -o /tmp/us-rural-raw.csv

ghcnp -dt mean -include "country eq 'UNITED STATES OF AMERICA' && population_type eq 'U'" -reg -o /tmp/us-urban-raw.csv

In this case, the tool informs us that 1046 rural stations have been analyzed, compared to 392 urban stations. A graph of the series follows.

To be thorough, let's also get 12-year running averages of the unadjusted series. That's what the article does.

This graph is very different to Figure 6 from Long's article, and it doesn't support Long's conclusions by any stretch of the imagination. It's also clear that while Long's urban trend is roughly correct, the 48 rural stations he picked are not representative of the 1046 stations GHCN Processor retrieves out of the raw data file. Why they are not representative can only be speculated upon, but I have some ideas.

The divergence between urban and rural stations that exists prior to the reference period (1950 to 1981) might be something that needs to be looked into further, but it's not too surprising. The farther back you go, the fewer the number of stations that report. There's simply more error in older data.

Addendum (3/31/2010)

In comments, steven suggests that GHCN v2 population assessments are old and may no longer be applicable. For example, a station might be near a town that used to have less than 10,000 people, and classified as 'R', but then the town grew.

Intuitively, it doesn't seem likely that this would be sufficient to explain away the findings, and it certainly doesn't address Dr. Long's choice of only 48 rural stations in the U.S. But I try to keep an open mind about these types of arguments, within reason.

Fully addressing steven's objection would take substantial work, but we can do the next best thing. Steven indicates that population density is what really matters. Let's take a look at a population density map of the United States (from Wikimedia):

Here's another such map. A portion of the U.S. (basically, the mid-west) has considerably lower population density than the rest of the country. Let's define this low-density region as that bounded by longitudes -95° and -115°, which excludes California. A longitude map of the US can be found here.

A typical rural station in the low-density region should not be as likely to be near an urban area as a rural station in high-density areas of the U.S. Additionally, the population density around the station should be lower for rural stations in the low-density region, in average. Does that make sense?

With GHCN Processor we can easily obtain an unadjusted temperature anomaly series only for rural stations in the low-density region, as follows.

ghcnp -dt mean -include "country eq 'UNITED STATES OF AMERICA' && population_type eq 'R' && longitude < -95 && longitude > -115" -reg -o /tmp/us-really-rural-raw.csv

I've calculated 12-year running averages of the rural low-density series, and plotted it along with the full-rural series.

Things haven't really changed, have they?

There seems to be somewhat of an offset between both station sets, which is interesting to some extent. Apparently, "really rural" stations were warm relative to all rural stations during the baseline period (1950 to 1981.)

BTW, there are 432 "really rural" stations in the unadjusted data file.

The Average Temperature of Earth

2010-03-28T17:00:00.000-07:00

If you Google the average temperature of Earth, you'll find a couple of frequent estimates: 13°C and 15°C. GISTemp data files carry a note that says:

Best estimate for absolute global mean for 1951-1980 is 14.0 deg-C or 57.2 deg-F...

GHCN Processor 1.1 has a -abs option that causes the tool to write out "absolute measurement" averages, as opposed to temperature anomalies. Additionally, simulations I've run indicate that the tool's default cell and station combination method (the linear-equations-based method) is adequate for this sort of application.

You can get a global absolute measurement series with the following command.

ghchp -gt seg -abs -o /tmp/ghcn-global-abs.csv

In this case the tool says 4771 stations have been analyzed. GHCN v2 has considerably more stations, however. A lot of them are dropped in the adjusted data set because they don't have enough data points. To get a series based on the raw data, you can run:

ghcnp -dt mean -gt seg -abs -o /tmp/ghcn-global-abs-noadj.csv

The tool now analyzes 7067 out of 7280 stations. (GHCN Processor 1.1 has an implicit filter that drops any station without at least 10 data points in any given month.)

It probably shouldn't come as a surprise that there are differences in the results we obtain with each data set. With the adjusted data, the average for the baseline period 1950 to 1981 is 14.7°C. With the raw data, the average is 14.1°C. The reason for the discrepancy probably has to do with the sorts of stations that don't make it into the adjusted data set, typically because they haven't reported long enough. They might be cold stations, like stations near Antarctica, a region with a substantial scarcity of stations in the adjusted data set.

Let me post a graph of both series, just so there's no confusion.

So it's basically an offset difference between the two. If I'm correct about it being due to stations near Antarctica, one caveat would be elevation. It appears that stations in Antarctica are high up, and this is a problem. We could filter stations by elevation, but then we basically drop Antarctica, like the adjusted data set does.

I believe 14.1°C is probably a low estimate. It's also pretty clear that GHCN v2 is land-biased.

There does seem to be a slight slope difference (0.15°C/century) between the adjusted and raw series. This can't be anywhere near significance, and again, it probably has to do with the sorts of stations that don't make it into the adjusted data set. I wouldn't be surprised if "sceptics" manage to make a big deal of it, though.

GHCN Processor 1.1

2010-03-28T05:40:00.000-07:00

Version 1.1 of GHCN Processor is now available for download. Relative to Version 1.0, the highlights are:

Implemented the first difference station combination method (Peterson et al. 1998.)
Added -abs option, which causes the tool to produce an "absolute measurement" series, as opposed to an "anomaly" series.
Added a -ccm option (similar to -scm) that sets the grid cell/box combination method.
The default station and grid cell combination method is the linear-equation-based method (olem.) This decision was based on simulations I ran comparing all 3 supported methods, involving series with equal and different underlying slopes.
Added -og option, which lets you write grid partitioning information to a CSV-format file. This feature is informative if you use the -gt seg option. (Zeke suggested producing a map. This is the next best thing.)

See the release-notes.txt and readme.txt files for additional details.

How GHCN Processor Combines Stations

2010-03-22T06:13:00.000-07:00

To combine temperature series from stations that occupy the same grid cell/box, it's normally not a good idea to simply calculate an average of the measurements. Actually, this would work just fine if all stations had complete data for the entire period of interest (e.g. 1880 to 2009.) They typically do not, however. Moreover, the absolute temperature measurements at one station can be systematically higher or lower than those of another station, even if the stations are close to one another. Because of this, if you just average stations, you will probably end up with "step" changes that do not accurately reflect reality.

Tamino proposed an optimal solution to the problem not long ago. Then RomanM wrote about a "more optimal" method, which he later revised in his Plan C post.

The problem both Tamino and RomanM took on involves finding an offset for each station-month (μ_i for station i.) Once you find the right offsets, you can add each temperature series to their corresponding offsets in order to "normalize" the stations, so to speak. Then you can average out the "normalized" series.

I believe that's the right problem to solve. My approach to solving it was different, nonetheless, and rather simple, in my view. The approach is already implemented and shipped with GHCN Processor 1.0. You can browse the source code of the implementation in CVS. (You need to pass a -scm olem option to the tool in order to use the method, otherwise it just uses a simple "temperature anomaly" method.)

Let's take as a given that you can always find a difference or delta between two temperature stations. Between stations i and j, let's call the difference ΔT_i,j.

Note that the difference between offsets μ_i and μ_j is equal to -ΔT_i,j. That's obvious enough, isn't it? The solution is easier to explain with an example. Let's take a grid cell with only 3 stations. The following two equations hold for such a set of stations:


(1) μ₀ - μ₁ = -ΔT_0,1
(2) μ₀ - μ₂ = -ΔT_0,2

Additionally, the sum of all offsets should be zero. This way, if you have data from all stations in a given year, the sum of the "normalized" measurements will be the same as the sum of the original absolute measurements. Hence, we have a third equation:


(3) μ₀ + μ₁ + μ₂ = 0

Equations 1, 2 and 3 are nothing more than a system of linear equations: 3 equations with 3 variables. It's actually simpler that a typical system of linear equations. Notice that if you simply add the equations, you can easily solve for μ₀:


μ₀ = -(ΔT_0,1 + ΔT_0,2) / 3

Once you have μ₀, you can easily solve for μ₁ (with equation 1) and μ₂ (with equation 2).

Unbiasing

You might have noticed that there's one more equation we could have used: one involving μ₁ and μ₂. In fact, a valid alternative system of 3 equations is the following:


(1) μ₁ - μ₂ = -ΔT_1,2
(2) μ₁ - μ₀ = -ΔT_1,0
(3) μ₀ + μ₁ + μ₂ = 0

While both equation systems are valid, the first equation system was biased toward station 0, whereas the second equation system is biased toward station 1. If you only solve one system of n equations biased toward station i, the solution is similar to the reference station method (Hansen & Lebedeff 1987.) The main limitation of this method is that it's dependent on the reliability of a single station: the reference station.

The full algorithm solves n systems of n equations each, with station i (from 0 to n-1) being the reference station in each case. Then it averages out the offsets that result from solving each of the equation systems. A straight average would be completely unbiased. The implementation uses a weighted average, where the weight of a reference station is the number of years with available data. The solution is thus biased, but only to the extent that some stations have more complete data than other stations.

Simulation

I thought that was a sound plan, but I wondered if it worked in practice. I wrote a simulation to find out.

The simulation consists of 15 stations with a common non-linear trend, and some common noise. On top of that, each station has noise and seasonality unique to it. Stations start out with complete data, and we calculate a "expected" temperature series with these data. Then we remove data points at random from each station – anywhere from 0% to 80% of them. We then apply a station combination algorithm and come up with a "estimated" temperature series. I have uploaded the source code of the simulation into CVS, where you will find all other details.

Let's start by evaluating how a simple "temperature anomaly" method does with this simulation. I'm interested in how well it predicts the absolute temperature measurements, not just the anomalies. It's not too bad. A linear regression of "expected" vs. "estimated" temperatures has a correlation coefficient R² of 0.987, with a slope of 0.99. Ideally, the slope should be 1. There's also an intercept of -0.251. This should ideally be zero.

Now let's see how the linear equation-based method does. The correlation coefficient R² is 0.997 in this case. The slope is basically 1, and the intercept is basically zero. It's nearly an ideal solution.

GHCN Processor 1.0

2010-03-20T10:25:00.000-07:00

As widely reported, Tamino debunked a key "sceptic" claim to the effect that the drop-out of stations in GHCN around 1990 introduced a false warming trend. Tamino's result has been replicated multiple times.

There's probably no point in producing yet another replication of Tamino's work. The theoretical problem itself did peak my interest in an academic sense, nevertheless. In particular, Tamino had mentioned in his preliminary results post that he was making all grid rows 10° tall, except for the northernmost row, which was 20° tall. However, he was counting its surface area as if it were 10° tall. Evidently, this has to do with the fact that there are relatively few temperature stations near the Arctic. That row is not equivalent, in a statistical sense, to other rows.

It occurred to me that an alternative way to partition a grid would be to create grid cells (or grid boxes, if you prefer) that have the same number of stations in them. I call them statistically equivalent grid cells. So I started to write some code.

I ended up writing not just a script, but a full-fledged command-line tool that can produce a temperature record for a pretty-much arbitrary set of stations. For example, it can produce a temperature series for urban areas of Brazil, or hilly regions of the Northern Hemisphere. All you do is pass a simple expression to the tool. Additionally, it is written in a modular manner, so it's relatively easy for others to contribute new grid partitioning methods, new adjustments, new station combination methods, station filters, and output formats. At least a couple alternatives are already supported in each case.

I have set up a SourceForge project page for this project, and other projects and code that I might release through my blogs in the future.

License

GHCN Processor is open source software. It is released under the terms of the Apache License 2.0.

Requirements

You need to have a Java Runtime Environment 1.5+ installed and in your PATH. It's not uncommon for PCs to already have it. You can check with:

java -version

Download

GHCN Processor can be downloaded from the Files Section of the project page.

Installation

You can unzip the ZIP file in a directory of your choice. Then you need to add the bin directory of the distribution to your PATH environment variable. In Windows, you need to right-click My Computer, then select Properties / Advanced / Environment Variables.

Documentation

Documentation on tool options can be found in the readme.txt file shipped with the GHCN Processor distribution. Any other posts I write about GHCN Processor should be available via the ghcn processor tag.

Global Examples

A default global temperature record can be created with:

ghcnp -o /tmp/ghcn-global.csv

The first time you run the tool, it will download inventory and data files from the GHCN v2 FTP site. Note that if you happen to interrupt the download, you could end up with incomplete files. In this case, run the tool again with the -f option.

The output is in CSV format, readable by spreadsheet software. By default it will contain one year per row, with 14 columns each. You can also produce a "monthly format" file with one month per row, as follows.

ghcnp -of monthly -o /tmp/ghcn-global-monthly.csv

By default, the tool uses the adjusted data set from GHCN v2. To use the raw, unadjusted data, try:

ghcnp -dt mean -o /tmp/ghcn-global-raw.csv

To use what I call "statistically equivalent grid cells", try:

ghcnp -gt seg -o /tmp/ghcn-global-seg.csv

The default combination method for stations that are in the same grid cell is a simple "temperature anomaly" method, with a base period if 1950 to 1981 by default. I've implemented a experimental "optimal" method that you can use as follows.

ghcnp -scm olem -o /tmp/ghcn-global-olem.csv

I'll discuss this in more detail on another occasion.

Regional Examples

To produce a regional temperature record, you should always pass a -reg option. The grid partitioning algorithm works differently when it's not attempting to produce a global grid. The following command produces a temperature series for airports in the United Kingdom.

ghcnp -reg -include " country=='UNITED KINGDOM' && is_airport " -o /tmp/uk-airports.csv

This alternative syntax also works:

ghcnp -reg -include " country eq 'UNITED KINGDOM' && is_airport " -o /tmp/uk-airports.csv

The expression after the -include option should be enclosed in double-quotes, and its syntax is that of EL (basically the same as Java or C expression syntax.) String literals in the expression must be enclosed in single quotes. The names of countries must be written exactly as they appear in the GHCN v2 country codes file.

As another example, let's get a temperature series for stations in the southern hemisphere that are in forested areas.

ghcnp -reg -include "latitude < 0 && vegetation eq 'FO' " -o /tmp/sh-forested.csv

GHCN Processor in this case tells us that there are only 7 such stations with temperature data in the adjusted data set.

Tamino Reproduction

In addition to station properties you find in the inventory file of GHCN, there are a number of properties added by GHCN Processor: slope, variance, first_year, last_year, and mean_temperature. The properties first_year and last_year tell you the years when the station started and finished reporting, respectively.

To reproduce Tamino's work, we can simply run the following commands:

ghcnp -include "last_year <= 1991" -o /tmp/wattergate-dropped.csv

ghcnp -include "last_year > 1991" -o /tmp/wattergate-kept.csv

GHCN Processor tells us that 2899 stations reported only until 1991 or earlier, whereas 1872 stations were reporting after 1991. That's in the adjusted data set. Notice that they are completely disjoint sets of stations. I can assure you; any half-decent grid partitioning method and station combination method will produce essentially the same results as Tamino's.

Acknowledgments

GHCN Processor relies on the following third-party libraries and environments:

Apache Commons Math 2.0 (Apache License 2.0)
JUEL - Java Unified Expression Language (Apache License 2.0)
Java

Of course, this tool wouldn't be possible without the GHCN v2 database.

Essentially The Same Graph, Different Smoothing

2010-03-05T05:34:00.001-08:00

This is probably the last post I will write on the topic of tropical cyclones in a long time. I don't intend to make this an all-tropical-cyclones-all-the-time blog. I promise.

I thought I should mention a paper I found – Mann & Emanuel (2006) – that contains a graph of storm counts and sea surface temperatures, shown below.

The smoothing is somewhat different to that of the graphs I've produced. It's decadal smoothing. They apparently use SST data for the Atlantic averaged over August-September-October. (I did something like this in my first analysis, but I do not believe it makes sense anymore.)

Mann & Emanuel then compare the decadally smoothed series, and find a correlation coefficient R of 0.73 (p < 0.001 one-tailed.) Based on these results, the authors concluded that:

There is a strong historical relationship between tropical Atlantic SST and tropical cyclone activity extending back through the late nineteenth century.

The way I see correlation coefficients, 0.5 is "not bad," 0.75 is "pretty good," 0.9 is "very good," and 0.99+ is "law of physics."

The confidence level is what really matters when it comes to causality, in my view. In my statistical analysis, where data had all the original noise, I found a detrended association with 99.993% confidence (equivalent to p < 0.00007) – allowing for a lag of 1 year.

I wondered, nevertheless: What would the correlation coefficient be like if we compare the 15-year central moving averages of the data I've been using? Let's do this only considering data since 1870, like Mann & Emanuel do. Let's also allow for a lag, given that such is evident in the graph.

I found something surprising. The optimal lag is 7 years, not a few years like I had presumed. The correlation coefficient R at this lag is 0.924 – well within "very good" territory.

You know what's interesting about the 7-year lag? 2005 minus 1998 is 7. I might come back to that some other time.

I conclude that Mann & Emanuel were on the right track, but they didn't make a strong enough case. There's a presentation by Tom Knutson of NOAA where he mentions the graph from Mann & Emanuel (2006), apparently thinking it's interesting, but wonders if the storm record is reliable enough to produce a graph like that. Then he goes on to discuss the long-term trend, which may or may not be statistically significant (as I already explained, causality and trend significance are different things), and the projections of computer simulations.

Literature on the Number of Tropical Cyclones

2010-03-04T08:59:00.000-08:00

It's not exactly true that my claims about the number of tropical cyclones tracking SSTs are at odds with observational science. They might be at odds with some models, as discussed in 4AR WGI 10.3.6.3.

See, for example, Webster et al. (2005). This paper has some interesting graphs too, but the time span is short.

We conclude that global data indicate a 30-year trend toward more frequent and intense hurricanes, corroborated by the results of the recent regional assessment.

I should also note that determining causality and determining whether there's a statistically significant upward trend are technically different endeavors. You can have one without the other. This applies to other effects of AGW as well, like sea level rise. Does that make sense?

Recent Storm Seasons Have Been Mild, Right?

2010-03-03T15:22:00.001-08:00

OK. The graph with the 15-year (or 17-year, or 21-year) smooth NH SST and Atlantic storm series looks good, almost too good. I was reminded of that fact recently. But then, you might ask, how do we reconcile this with the fact that Atlantic storm seasons after 2005 have been mild? Should I believe my lying eyes, or... my lying eyes?

Mild compared to what? Anything is mild when you compare it to the Katrina season.

You can search Wikipedia for each year's count, e.g. google "2006 Atlantic Hurricane Season." These are the figures:

The average is 12.25. The average between 1944 and 1990 was 9.8. I'm not saying recent seasons are significantly more busy, but you can't claim they are milder than the 1944-1990 average either.

Clearly, 2007 and 2008 were both considerably more busy than normal. They probably went unnoticed because, again, what could possibly compare to the 2005 season?

Note that I never claimed storms resulting from global warming were going to be the end of us all, or anything of the sort. I haven't even attempted to model it yet. Any projection I made carried a caveat: that the association would have to be linear. But it can't really be linear, can it? I can't say, for example, if the number of storms saturates after a certain temperature is reached.

I never even claimed that there has been a statistically significant rise in the number of storms. It seems reasonable this is the case, but I just haven't done this analysis. What I did is determine whether storms track sea-surface temperatures, and they do.

I will say one thing, though. Do not be surprised if the 2011 or 2012 seasons are quite busy. I'm saying that because 2010 is an El Niño year. It's not the case that busy seasons always follow El Niño years, or that all El Niño years end up producing busy seasons. But based on a quick comparison, it often does happen that way.

Too Easy To Be True?

2010-03-03T05:22:00.000-08:00

Back in June of 2008, I wrote a statistical analysis of the association between northern hemisphere sea surface temperatures and the number of named storms in the Atlantic basin. I determined, with 99.993% confidence, that indeed there was such an association. I had controlled for coincidental trends (otherwise known as a spurious correlation) by means of detrending.

A commenter over at Climate Audit tacitly admitted reproducing my analysis, but pointed out that if he detrended the series using 6th-order polynomial trendlines, the association no longer held. I noted that if you allow for a lag of 1 year between the series, even the 6th-order detrending resulted in a statistically significant association, despite the loss of information that necessarily results from such a detrending. The existence of the lag between temperatures and named storms would soon become crystal clear to me and my (not very many) readers.

During the back and forth with the Climate Audit commenter, I realized that if you simply smooth out the noise from both series, the association becomes graphically evident, and a lot more convincing – I thought – than a statistical analysis. The first graph I posted used nothing more than a 21-year central moving average for smoothing. The results were remarkable and the graph was remarkably easy to produce.

As it turns out, it was also remarkably difficult to believe. A few months went by, and a reader (paulm) posted a link to my analysis over at the Accuweather.com GW blog. The graph was met with a got-to-be-fake type of response.

When I found out, this obviously made me upset. You can tell I was upset as I was explaining my very lenient comment policy (see sidebar) in my response to the Accuweather.com incident. I even posted the spreadsheet for verification. There were no further falsification accusations after I did this, and I thought that was the end of it.

Fast forward a year and a half. Deltoid has a recent post on the topic, quoting various IPCC statements, and I basically commented that the IPCC was wrong, in my view, in regards to the number of tropical cyclones not changing in response to global warming. The data told me otherwise.

I guess questioning an IPCC claim was a mistake, wasn't it? I might have also broken some social norm I'm not aware of or something. Some of the regulars started to talk to me as if I were one of the resident trolls, like El Gordo or Spangled Drongo. They basically accused me of fraud and trying to deceive, in a way that is not dissimilar to how the CRU team are accused of fraud and so forth.

Things calmed down after I, once again, posted the spreadsheet. I appreciate Bernard J's semi-apology.

For reference, below are the links to the data and the spreadsheet. I posted the spreadsheet at a more permanent location.

What else can I do? You've seen my comment policy. Should I post screenshots too?

I'd hate to think that my most interesting graphs are assumed to be fake a priori. Is my graph of Red Sea sea level and Vostok temperatures a fake? What about my graph of the natural spline interpolation of the Law Dome CO2 data? What about the one with the Mann et al. (2008) reconstruction and CO2 at the time of the industrial revolution? What about the graph with the Greenland temperature reconstruction?

I realize I often post claims that you can't just Google to confirm, and I realize that people are sometimes paranoid. That's why I have the comment policy I have.

Sea Level Rise - Part 3

2010-02-28T11:32:00.000-08:00

Thermal Expansion

In Sea Level Rise - Part 2, I had estimated the magnitude of sea level rise (SLR) due to ice melt alone, since the last glacial maximum (LGM). It was 97 meters. This was based on an ice coverage and ice topography reconstruction made available by Peltier (1993).

Total SLR since the LGM is about 130 meters according to Fleming et al. (1998). The IPCC figure is 120 meters (4AR WGI FAQ 5.1.) The Red Sea reconstruction provided by Siddall et al. (2003) also indicates it's in the order of 120 meters.

So we're missing at least 23 meters of SLR. Could this difference be the result of thermal expansion? As it turns out, no.

I will estimate a ballpark figure for SLR since the LGM due to thermal expansion alone. For this, I will use a simplified model of the ocean: A rectangular pool of water with 3.5% salinity, a surface area of 361·10⁶ km², and a depth of 3,600 meters. The temperature at the surface of the pool has increased from 12C to 16C. The temperature at a given depth is calculated using the ocean water temperature profile model I derived in the last post. I will also assume that water pressure does not affect our calculations significantly.

In order to carry out the calculation, I divided our simplified ocean in 36 layers of 100 meters each. For each layer, I calculated current temperature and LGM temperature.

You can also calculate the density of water given its temperature and salinity. The following graph illustrates the relationship between temperature and the inverse of density (or volume of 1 kg of water in liters.)

The density of water with 3.5% salinity is, thus, well approximated by:

T is the temperature in degrees Celsius. The units for density are kilograms per liter.

With this equation, we can calculate the density of each layer of our simplified ocean, at present and during the LGM. The volume of each layer at present is 3.61·10⁷ km³. The volume of each layer in the LGM can be calculated by multiplying 3.61·10⁷ km³ by the current density and dividing it by the LGM density.

The total volumetric difference turns out to be 9.5·10⁴ km³. This translates to 0.26 meters of SLR, or 26 cm.

This should not be taken as a precise figure. The point is that it's nowhere near 23 meters. Clearly, the change in ice volume I calculated from the Peltier (1993) reconstruction must be an underestimate.

Ice melt is obviously a much more significant problem, in the long term – meaning thousands of years – than thermal expansion. However, if the temperature is rising rapidly, thermal expansion can temporarily contribute more SLR than ice melt. Presumably, the effects of thermal expansion are immediate. Ice melt is slow.

The Temperature of Ocean Water at a Given Depth

2010-02-26T10:05:00.000-08:00

In order to corroborate the results of the last post on sea level rise, I would like to estimate the impact of thermal expansion alone since the last glacial maximum. (I anticipate I will need to post a correction, but first things first.) In order to do this properly, I need to know how the temperature of ocean water varies with depth. I've tried to look for a straightforward formula that solves this problem, to no avail. (I did find a graph that was illustrative.)

So I went ahead and derived such a formula, based on ocean water characteristic data from the Argo database. More specifically, I used the ensemble mean grid made available by Asia-Pacific Data-Research Center of the University of Hawaii.

For each depth file, I calculated temperature means for 3 different latitude groups, which I've called 0S, 30S, and 60S. The 0S group includes latitudes -15S to 15N. The 30S group includes -45S to -15S. The last group includes -75S to -45S. Data is not available for all latitudes, so the mean values obtained should not be considered latitudinal averages; that's not the purpose of this exercise.

Let's start by looking at a graph of mean temperature at a given depth for each of the groups.

It doesn't look very easy, does it? Clearly, the temperature of ocean water must depend on variables other than the sea surface temperature and depth. But I was able to come up with a model that fits the data quite well (R²=0.988.) The formulas follow.

Where:

S is the sea surface temperature plus 0.338, in degrees Celsius.
D is the depth in meters.
T(D) is the temperature at a given depth D, in degrees Celsius.

Derivation

The model was empirically derived. The first thing I noticed is that the depth D is approximately inversely proportional to T(D). In fact, the following formula is a rough approximation of temperature at a given depth.

The problem with this formula is that it doesn't work very well at shallow depths. You might have noticed in the figure that the temperature is roughly stable at a depth of 30 meters or less. In order to fix this problem, the approach I came up with is to transform D into D', where D' tends to zero when D is small, but tends to D when D is big.

If we multiply D by a logistic function, we obtain something close to the desired result for D'.

Once I had the general form of the equation, all I had to do is figure out the 4 coefficients involved. I used genetic programming to solve this.

The Greenland Canard

2010-02-22T06:30:00.000-08:00

A favorite argument of AGW "sceptics" has to do with Greenland and what is known as the medieval warm period (MWP). The idea is that if the Earth was warmer some time in the past, this would undermine AGW theory. I don't find this line of argumentation to be robust either way, but let's examine it, shall we?

The Vikings colonized Greenland from 986 AD. They were able to farm, fish and raise cattle. The settlements disappeared by the 15th century, presumably because of the little ice age (LIA).

The first thing that needs to be pointed out is that Greenland was still a very cold place during the MWP. It was not a green paradise of any sort. Skeptical Science and A Few Things Ill Considered have the details on that.

Another counter-argument I've encountered is that Greenland might have been considerably warmer than it is now during the MWP, but this was most likely a local or North-Atlantic phenomenon, not a global one. Nearly all climate reconstructions that cover the MWP (and there are many of them, by many different authors, using several different methods) do not show it to be warmer than today.

Is there a way to corroborate that Greenland was indeed unusually warm during the MWP, unlike the rest of the northern hemisphere? Absolutely.

Alley (2000) provides a 50,000 year temperature reconstruction from Central Greenland whose resolution is not bad at all. The reconstruction is ice-core based, and it provides temperatures as absolute values. In order to calculate a "temperature anomaly" for purposes of graphical comparison, I added 31.29 to the temperature values provided by Alley (2000).

Let's start by looking at data from 200 AD to 1850 AD. I will use the CPS Northern-Hemisphere reconstruction from Mann et al. (2008) for comparison.

The difference between the MWP and the LIA in Greenland was around 1.6°C. Interestingly, the MWP temperature peak in Greenland occurs almost exactly at the time of the Viking colonization. Presumably, that's not a coincidence. Additionally, there are already some indications in this graph that the climate of Greenland experiences abrupt changes from time to time.

Not convinced? Let's look at 50,000 years of data. For comparison, I will use the temperature reconstruction from Vostok station, Antarctica made available by Petit et al. (1999). This is also an ice-core based reconstruction.

Here we see that the temperature of Greenland fluctuates in a manner that is not matched by the temperature of Antarctica. In particular, notice the effect of the Younger Dryas stadial on each of the series.

In synthesis, the climate of Greenland is quite peculiar, and as a result, it should not be thought of as a proxy of global or even hemispheric climate.

Sea Level Rise - Part 2

2010-02-17T06:28:00.000-08:00

Ice Melt

In the previous post on Sea Level Rise, I argued there is a clear association between the Red Sea sea level reconstruction from Siddall et al. (2003) and the Vostok temperature reconstruction from Petit et al. (2000), with sea level lagging temperatures by about 4,700 years, at least for the last 50,000 years. This is corroborated by other reconstructions, such as the SL reconstruction from Arz et al. (2007) for the time span 83,000 years to 13,000 years before present.

The magnitude of SLR since the last glacial maximum 20,000 years ago is about 130 meters (427 feet), which again, is non-trivial. If we had similar SLR today, essentially all coastal cities around the world would be under water.

Much of that SLR has to be the result of ice melt. The rest, if any, would have to be the result of thermal expansion. Ice that is floating in the ocean should not cause SLR when it melts. What matters in this sense is ice over land.

In this post I'd like to estimate the magnitude of sea level rise due to ice melt alone.

NOAA provides a reconstruction contributed by Peltier (1993), along with applets that let you visualize ice coverage and ice topography in a world map. The resolution is 1,000 years, and the data goes all the way back to the last glacial maximum.

Fortunately, most of the ice coverage is over land, as you can see. So I took the 21K-year-old ice coverage data, and used it as a baseline for the calculation of changes in ice volume. The topography data includes land topography, evidently, but we're interested in ice volume change (not so much absolute ice volume) so it will do for now. In a footnote I will provide the processed data, along with an explanation of how it is obtained. The following graph shows the ice volume data along with the Vostok temperature reconstruction.

So we're talking about a change in ice volume in the order of 3.9·10⁷ km³ in the last 21K years. The density of glacial ice is about 90% that of ocean water. (The tip of an iceberg is about 10% of the iceberg.) So the change in ice volume translates to 3.5·10⁷ km³ of additional ocean water.

Now, the surface area of the ocean is 3.61·10⁸ km². If sea level rises, this surface area will change in a manner that is negligible, even if some areas are flooded. Does that make sense? Therefore, a good approximation for the change in ocean volume (ΔV) that results from a rise (ΔL) in sea level, is:

ΔV = 3.61·10⁸·ΔL

Solving for ΔL, we have that:

ΔL = 0.097 km

That is 97 meters. It follows that thermal expansion must be responsible for the remaining 33 meters of SLR since the LGM.

[Correction 2/28/2010: As it turns out, SLR due to thermal expansion can't be anywhere near 33 meters. See Part 3 of the series.]

What if we lost all ice?

Let's assume, conservatively, that only 2.5·10⁷ km³ of ice remain over land. If we managed to melt all of it, the ocean would get an extra 2.24·10⁷ km³ of water. Dividing this volume by the surface area of the ocean, we get 62 meters of SLR.

That scenario is, of course, not something we'll see in our lifetimes, by a long shot. Even if we raised the temperature of Earth enough, it would take thousands of years to be realized, no doubt.

It's not an impossible scenario, however. About 50 million years ago, during the Paleocene-Eocene Thermal Maximum, global sea level was about 200 meters higher than today.

Data

Summarized Ice Coverage and Ice Volume from Peltier (1993)
This is obtained by processing all ice*.asc and top*.asc files in "ASCII special" format. The area of a grid cell is calculated by dividing the area of a spherical ring 1-degree wide (corresponding to each latitude in the data) by 360. Only grid cells with ice coverage in the 21K-year-old data are considered when calculating land+ice volume.

Sea Level Rise - Part 1

2010-02-11T09:30:00.000-08:00

I have begun to take a closer look at available sea level data. Frankly, I'm a bit surprised. Of course I've heard of sea level rise (SLR) before, but it was like a fuzzy abstraction. I didn't have specific figures to ponder.

In this particular post I will only discuss paleoclimate data. There's some data that I imagine is pretty well known. The figure to your right, for example, comes from Wikipedia. It's a sea level reconstruction from Fleming et al. (1998) of the last 20 thousand years or so. That's roughly the time span since the last glacial maximum (LGM), when the global mean temperature was 5°C lower than today, give or take. As the LGM ended, and the planet entered the current interglacial period, sea level rose by about 130 meters (427 feet.) Of course, much of that is probably the result of ice melt, and there was a lot of ice in the LGM.

Then there's data that is not well known as far as I can tell. For example, there's a 380,000-year reconstruction contributed by Siddall et al. (2003) based on oxygen isotope records from Red Sea sediment cores. The following graph shows this sea level reconstruction along with the Vostok station temperature reconstruction provided by Petit et al. (2000).

If there were any doubts that temperature drives sea level, I believe the graph above is enough to dispel them. This is the basic premise I wanted to establish with this first post.

The data has some features that I wish to explore further, but not today. In particular, notice that sea level lags temperature by several thousand years. (This is not so clear with sea level data older than about 250,000 years. I'm guessing there's some sort of dating error either in the Red Sea data or in the Vostok data once you go that deep.) If I only consider data for the last 50,000 years, I'm estimating that the best lag is about 4,700 years. I'm not sure I can emphasize enough how important this lag is, but I'll certainly try.

[In Part 2 I estimate SLR due to ice melt alone since the LGM.]

The Twin GHGs Paradox

2010-02-05T10:51:00.000-08:00

The means by which a greenhouse gas (GHG) forces climate change is sometimes called radiative forcing. You can think of it as the additional irradiance necessary to bring the system back into balance after a change in the concentration of the greenhouse gas.

Apparently, the radiative forcing contribution of different greenhouse gases is typically added to come up with the total contribution. For example, if radiative forcing between 1750 and 1998 is 1.46 W/m² for CO2 and 0.48 W/m² for methane, then the total for the two gases is 1.94 W/m². This makes complete sense on the surface, does it not?

But if the effect of a change in the concentration of a GHG is non-linear, is it really correct to linearly add such effects? I was thinking this would be similar to trying to add decibels.

After some digging around, my impression was that the IPCC had addressed this issue in section 2.8.4 of 4AR WGI. If you read the papers cited, nevertheless, it appears that they refer to the equivalence of the addition of forcings and the addition of temperature changes. I don't doubt these operations are roughly equivalent for relatively small effects. But that's not what I'm talking about at all. I'm questioning the validity of either operation, considering that both forcings and temperature shifts are non-linear responses.

In order to illustrate the problem, I've come up with a thought experiment that I've dubbed the twin GHGs paradox.

Imagine there are two GHGs that happen to be identical in their effects. If you have trouble picturing it, suppose one gas is industrially-produced CO2 and the other gas is naturally-produced CO2. Their effects are logarithmic.

Let's say the concentration of both gases has changed from 100 ppmv to 200 ppmv. Then their combined radiative forcing would be given by:

ΔF = k·ln(200/100) + k·ln(200/100) = 2·k·ln(2)

Since the gases are identical, the change just described should be equivalent to, say, that of one gas going from 190 ppmv to 390ppmv and the other gas remaining at 10 ppmv, not changing at all (i.e. the combined total is 200 ppmv initially and 400 ppmv later.) Therefore,

ΔF = k·ln(390/190) + k·ln(10/10) = k·ln(2.05)

Combining both equations, we have that:

ln(2.05) = 2·ln(2)

Evidently, we end up with reductio ad absurdum.

To get more technical...

When you're looking at the absorption bands of two GHGs, it seems clear that it matters whether the bands overlap or not. The net absorption calculation will depend on this.

Smoothing Splines and Law Dome CO2 Data

2010-01-06T16:44:00.000-08:00

I've been reading about a paper by Ernst-Georg Beck where it is claimed that CO2 levels in the past 150 years have fluctuated widely. Apparently, when Mauna Loa measurements started, CO2 levels magically became more stable. It's not surprising these claims have been made fun of. But that's not what I really want to discuss.

Beck's paper got me thinking about the Law Dome ice-core data I've been using to associate CO2 with temperatures. I suspected it might actually be too smooth. You see, Etheridge et al. (1998) provides two convenience data sets: One is a 20-year smooth series spanning 1832-1978. The other is a 75-year smooth series spanning 1010-1975. They are obtained using smoothing splines.

You can actually see that the Etheridge et al. 20-year smooth data is even more smooth than Mauna Loa data (e.g. in this figure, before and after 1978.) The problem with smoothing out noise is that you can easily lose information.

I went ahead and calculated the natural spline interpolation of the raw data from Etheridge et al. (which I'm making available HERE.) The natural interpolation only uses the spline function to estimate data for years that are missing. It does not smooth out the raw data that is known. In the Etheridge et al. data set, what this will mean is that recent data will have most of the original noise, while old data will be smooth, but not 75-year smooth, hopefully.

Let's take a look at the 1850-1978 20-year smooth series provided by Etheridge et al. (1998), along with the natural interpolation of the raw data.

Interesting, isn't it? I think it makes a lot of sense. In particular, notice the flat CO2 trend between the years 1933 and 1952. That's 20 years without an increase in atmospheric CO2. With the smooth series, this is not so evident. You see perhaps 12 years of pause or so.

Is this important? Let's take a look at the HadCRUT3 temperature series along with the logarithm of the new CO2 series, from 1850 to 2008. (Mauna Loa data is added after 1978, minus a small offset.)

It would appear that the climate reacts rapidly to CO2 fluctuations, which again, argues for a small amount of "warming in the pipeline."

There are other things you can't see very well with the overly smooth data. Let's take a look at the time span between 1700 and 1900. For temperature, I will use the NH-SH average from the CPS temperature reconstruction of Mann et al. (2008).

Keeping in mind that both the temperature and CO2 data in the figure are reconstructed, I think this is pretty interesting. It suggests there might have been slight warming of about 0.1C right at the beginning of the industrial revolution. This is consistent with estimates of climate sensitivity.

Warming in the Pipeline

2010-01-05T08:12:00.000-08:00

How much "warming in the pipeline" is there? Is there a way to be sure? NASA GISS findings suggest it's about 0.6C.

I was contemplating a method of estimating "warming in the pipeline" from available temperature and CO2 data. It's sort of a heuristic method, with all this implies, but it's interesting that I get somewhat different results.

CO2 appears to be the main causal agent of recent temperature shifts. I've demonstrated that temperature fluctuations lag CO2 fluctuations by about 10 years. It comes to reason that observed temperature fluctuations lag equilibrium temperature fluctuations by about 10 years as well.

Imagine you have an equilibrium temperature series that looks like a sinusoid. Observed temperature will lag the hypothetical series, also looking like a sinusoid, by some number of years. If we use Newtonian cooling as an approximation, the expected rate of temperature change (R) will be given by:

R = r·(T' - T)

T' is the equilibrium temperature and T is the observed temperature. The constant r is something I will call the rate coefficient.

The question is: If you know the lag between the sinusoid series, can you estimate the value of the rate coefficient r? Then, if you know r, can you estimate "warming in the pipeline"? I think the answer is yes.

I gave up trying to solve it with calculus. Perhaps a reader can give it a shot. I instead solved it by means of a Monte Carlo simulation.

It turns out that the rate coefficient r depends on the period of the sinusoid and the lag between the series, but not the amplitude of the sinusoid.

The period of the CO2 sinusoid that results from the 3rd-order detrending of the CO2 series is about 85 years (pulsation is 0.074.) You can see that in the figure above. For this period, and a lag of 10 years, my simulations indicate that the rate coefficient r should be just about 0.08 in units of year^-1.

Let's assume that the current rate of temperature change (without weather noise) is about 0.018 degrees Celcius per year. Then we have that:

0.018 = 0.08·(T' - T)

So the temperature imbalance is:

ΔT = T' - T = 0.018 / 0.08 = 0.23°C

This is not too bad. If correct, I'd take it as good news.

Climate Sensitivity

I'm essentially claiming that the global equilibrium temperature is knowable and that it's perhaps 0.7°C relative to the HadCRUT3 baseline. It also appears (based on various reconstructions) that the temperature in the 18th century was fairly stable at about -0.4°C. We can probably assume that's an equilibrium temperature. The difference is 1.1°C.

The concentration of CO2 in the 18th century was about 277 ppmv. The concentration as of 2008 is more like 385 ppmv. This means that:

1.1 = k·ln(385/277)

Therefore:

k = 3.34

We can estimate the climate sensitivity to CO2 doubling as follows:

λ = k·ln(2) = 2.32°C

This is actually a tad lower than the best estimates available. There are some uncertainties in the estimate, to be sure. For example, were temperatures really stable at -0.4°C in the 18th century? Then there are some errors that are immediately obvious. Some of the warming could be due to methane and other greenhouse gases. You also have cooling due to aerosols, which would confound the estimate in a manner whose magnitude is not well understood.

On the Sun, Planets, CO2 and Models

2009-12-23T10:51:00.000-08:00

Climate forcing due to solar irradiance is not that difficult to figure out. Per Stefan–Boltzmann law, we know that the equilibrium temperature (in degrees Kelvin) of a black body in the solar system is given by:

T' = k·S^1/4

S is the solar irradiance. The value of S for planet Earth is often quoted as 1367.6 W/m².

It turns out that the temperature formula above is generally applicable to almost all the planets in our solar system. Figure 1 is a scatter chart of solar irradiance (to the 1/4th power) vs. the mean temperature of planets in our solar system.

Except for the planet Venus, it's clear we have a correlation so strong that it could only be the reflection of a straightforward law of Physics. Planet Earth actually falls a little bit outside the model. It's about 8 degrees Celsius warmer than the formula would predict. (I understand this is called the natural greenhouse effect.) If we remove Earth from the analysis, the correlation coefficient improves to 0.9995 and the slope of the linear trend becomes 46.163. Hence, the formula for the equilibrium temperature of any arbitrary planet, provided nothing unusual is going on with it, is the following.

T' = 46.163·S^1/4

Per Figure 1, again, this appears to be quite precise. With this formula we can begin to estimate what might happen if solar irradiance were to increase from, say, 1363.4 W/m² to 1366.7 W/m². This is the range of variation reported in the Lean (2000) solar irradiance reconstruction between the years 1610 and 2000.

With an increase of that magnitude, I estimate that equilibrium temperature would rise by only 0.2 degrees. It can't possibly explain a 0.8- to 1.6-degree anomaly by itself.

CO2 Model

When I first started to look at data and claims related to climate science, and noticed the concept of temperature sensitivity to CO2 doubling, my impression was that equilibrium temperature could be approximated by a formula such as this one:

T' = baseline + a·ln(C)

C is the concentration of CO2, most likely in ppmv, and a an unknown constant. Climate sensitivity would be given by a·ln(2).

There are other types of formulas that model how CO2 would affect radiative forcing, as opposed to equilibrium temperature. These work in a different manner, but in general these formulas can only be approximations of the real world. They may be roughly applicable to our planetary reality, but they won't work in general.

The first problem with the conventional doubling model is that it can't work for really small concentrations of the greenhouse gas. Imagine there are only 100 molecules of CO2 in the entire atmosphere, and we subsequently double this concentration to 200 molecules. Should we expect equilibrium temperature to increase by about 3 degrees as a result?

The second problem is that greenhouse forcing is temperature dependent. In particular, it can't work for really low temperatures. It's clear from Figure 1 that a planet receiving zero solar irradiance would have a temperature of absolute zero. It would not irradiate out any energy due to its temperature, and no (theoretical) greenhouse gas would change this.

Let me propose a model that subsumes the previous models and addresses the two problems outlined.

T' = 46.163·(S·(1 + a·ln(b·C + 1)))^1/4

When b·C is sufficiently large, the extra greenhouse forcing is proportional to the logarithm of C. When C is zero, the formula is reduced to the one derived from Figure 1. When solar irradiance is zero, temperature is still zero, regardless of C. When C increases from 100 to 200 molecules, the effect is negligible.

That's still an approximation in other ways, no doubt, but I expect to use it in the future.

A trick to "hide the decline" in autism data

2009-12-07T05:39:00.000-08:00

The SwiftHack controversy reminded me of something that is done routinely in epidemiology when considering prevalence by birth year data.

I was thinking of what is perhaps the most cited email of SwiftHack; this one by Phil Jones:

From: Phil Jones To: ray bradley ,mann@xxxxx.xxx, mhughes@xxxx.xxx Subject: Diagram for WMO Statement Date: Tue, 16 Nov 1999 13:31:15 +0000 Cc: k.briffa@xxx.xx.xx,t.osborn@xxxx.xxx Dear Ray, Mike and Malcolm, Once Tim’s got a diagram here we’ll send that either later today or first thing tomorrow. I’ve just completed Mike’s Nature trick of adding in the real temps to each series for the last 20 years (ie from 1981 onwards) amd from 1961 for Keith’s to hide the decline. Mike’s series got the annual land and marine values while the other two got April-Sept for NH land N of 20N. The latter two are real for 1999, while the estimate for 1999 for NH combined is +0.44C wrt 61-90. The Global estimate for 1999 with data through Oct is +0.35C cf. 0.57 for 1998. Thanks for the comments, Ray. Cheers Phil

Some people seem to be under the impression that the "decline" refers to the decline in temperatures shown in some databases when you cherry-pick 1998 as the starting year in temperature slope calculations. The email was written in 1999, so that can't be it, first of all.

Second, how exactly do you fudge data by adding in "real" data? That doesn't make any sense on the surface.

Once you parse the statement further, you realize that "Mike's Nature trick" is a reference to Figure 5b of Mann, Bradley & Hughes (1998), a paper published in Nature, and whose first author is Mike Mann. The figure follows.

The figure is not very clear, unfortunately, because it's in black and white, but you can see it's simply a graph of multiple time series: (1) A historical temperature reconstruction up to 1980, (2) The instrumental record (referred to as "actual data" in the figure, and "real temps" by Phil Jones), and (3) a 50-year low-pass filter of the reconstructed series. Really, there's nothing underhanded or nefarious about plotting multiple series in one graph. It's not much of a "trick" at all.

In addition to this, Phil Jones says he added the real temps to Keith Briffa's series, starting in 1961, in order to "hide the decline." That's the interesting part, isn't it? After some digging (hat tip Deltoid) it's clear the "decline" after 1961 refers to something related to tree-ring data that was documented in Briffa et al. (1998). Figure 6 of this paper follows.

It's pretty clear why you'd want to add the real temps starting at about 1961. Again, I don't think there's anything wrong with this.

Some people might insist, however, that throwing away the last part of a series because it doesn't show what you think it should show is just wrong. It's not valid scientific methodology, and so on and so forth, ad nauseum.

Is it wrong? Are you sure? Let me show you something that I bet most readers of this blog have not seen.

This is a graph of the administrative prevalence of autism by birth year. The X axis is the year of birth. For example, the administrative prevalence of autism for children born in 1992, according to special education counts, was 34.5 in 10,000 as reported in 2003. The prevalence is much lower for children born in the year 2000: 12 in 10,000.

Is it actually true that the prevalence of autism is that low for children born in 2000? Absolutely not. That's what the 2003 report said. The 2009 report will tell you something very different. You see, it takes time for children to get diagnosed with autism. If they are too young, they might not have any label at all, or they might be classified under a different label.

Autism prevalence by birth year series will always decline in recent years. They also change as the series are revised year after year. (That's why I suggest using prevalence at age by report year instead, but I digress.)

If you must use a prevalence by birth year series, there's a "trick" you can use to "hide the decline," and this "trick" has a name: You left censor the series. In an administrative autism series, you probably should not consider children below the age of 8.

I have used this "trick" myself in a blog post titled Is Precipitation Associated with Autism? Now I'm Quite Sure It's Not. I said: "I'm left-censoring autism caseload starting at 2000." So there you go. Joseph, from the Residual Analysis blog, has admitted to using a "trick" to "hide the decline." Feel free to quote me on that.

Once again, I think AGW deniers need to grow up a little.

Statistical Proof of Anthropogenic Global Warming v2.0

2009-12-03T08:44:00.000-08:00

Given that I'm again looking at climate data, I decided to write a follow-up of the post that launched this blog (originally here.) That was an analysis intended for skeptical laypersons, illustrated graphically at each step. Essentially, I demonstrated that you can associate CO2 emissions with temperatures in a way that eliminates the possibility of coincidence.

You see, it's very easy to associate CO2 with temperatures in a "naive" way. It's also very easy to associate the number of pirates with temperatures this way. In fact, you can associate any two trends in this manner.

I suggested removing the trends from the data, and then comparing the resulting detrended series. I explained it a little differently back then, but that's what it amounts to. Since that time I have learned this is called detrended cross-correlation analysis.

There were a number of criticisms of my analysis posted in comments. I didn't find any of them very convincing, to be honest, but they can be addressed as follows.

Instead of using northern-hemisphere land temperatures, I will use global sea-surface temperatures (HadSST2 data set.) The criticism was that fluctuations of CO2 emissions could be associated with (i.e. confounded by) fluctuations of the urban heat island effect. I think that's a bit of a stretch, and this sort of objection has been addressed elsewhere, but I thought I would simply use sea-surface temperatures to eliminate the potential confound altogether.
Instead of using cumulative CO2 emissions, I will use an ice core-based CO2 reconstruction from Etheridge et al. (1998), combined with Mauna Loa CO2 data. I will use Mauna Loa data only for 1979 onwards, and I will adjust it by subtracting 0.996 ppmv from each data point. There's an excellent match between the Etheridge et al. data and Mauna Loa data for the years 1958 to 1978, but there's a tiny offset of 0.996 ppmv between the two.
I will use the logarithm of the CO2 concentration. This is theoretically more accurate. The equilibrium temperature of the planet depends on the concentration of green house gases, logarithmically. That's why climate scientists talk about sensitivity to CO2 doubling.

I will start by posting a graph of the original sea surface temperature (SST) and (logarithm of) CO2 series. This is Figure 1.

I said we would remove the trend from the series. What exactly is the trend, you might ask. There are many ways to model a trend. We could use a straight line, but then you could say that the series are not linear. The wobbles around each of the linear trends might also coincide. A polynomial trendline can model trends that are not linear. A 2nd-order polynomial trendline might be enough. I went straight for the cubic or 3rd-order polynomial trendline, which gives a slightly better fit. Those are the yellow lines that you see in Figure 1.

To detrend a series, you simply subtract the trendline from the series. The result of the operation can be seen in Figure 2 below.

The scales are a little different, but you should be able to tell, visually, how Figure 2 is obtained from Figure 1.

I added a vertical dashed brown line around the year 1890. I suspect data is simply wrong prior to 1890, for no other reason than the fact that it doesn't look good visually. I will analyze both the entire range (1850-2008) and the shorter one (1890-2008). Curiously, I had previously found something similar in a comparison of SSTs and named storms, but I assumed the storm data was the one in error.

Obviously, changes in CO2 won't be reflected in temperature fluctuations instantaneously. It takes time for heat to be trapped. Or to put in more technical terms, CO2 is logarithmically proportional to the equilibrium temperature. So there should be a lag.

Looking at the whole series, a statistically significant association between the detrended series starts to become significant with a lag of 6 years. The best lag (based on correlation coefficient) is 15 years. At a lag of 15 years, the association is significant with 99.997% confidence.

In my previous analysis I had found a lag of 10 years was the best lag, not 15. Interestingly, a lag of 10 years is exactly the best lag in the current analysis if I only consider data starting at 1890. In my estimation, 10 years is the true best lag, and this is completely justifiable theoretically by other means.

When you look at data from 1890 onwards, even a lag of zero will result in a statistically significant association. With a lag of 10 years, the association is significant with 99.99996% confidence. I think it's worth looking at a graphical representation of this association: the following scatter chart (Figure 3.)

In Figure 3, each dot represents a year. The graph tells us that the higher the detrended CO2 concentration in a given year, the higher the detrended sea-surface temperature, 10 years later. We can also calculate the 95% confidence interval of the slope of the trend, which is 13.7 to 29.7 in this case.

Given the methodology used, and the direction of the lag, this result can't be anything but indicative of a causal association between CO2 and sea-surface temperatures.

The association can't be explained by:

Coincidence.- Because we removed series trends, and because the associations are highly significant, mere-chance coincidences are exceedingly improbable.
Correlation is not causation.- For the same reasons, because of the lag, and because there appear to be no confounds, the only plausible explanation for the correlation is causation in this case.
Urban heat island.- The urban heat island effect should not be relevant to sea-surface temperatures.
Error and bias.- Errors would tend to hinder the analysis rather than help, just like we see with the data prior to 1890. Any systematic bias should be taken care of by the detrending step.
Conspiracy.- It would be preposterous to suppose that someone doctored the data sets (in just the right manner) anticipating this type of analysis several years in advance.
Assorted Non-sequiturs.- Arguments such as "Al Gore sucks" can be dismissed off-hand.

Skeptic's Circle #125

2009-12-02T12:53:00.000-08:00

The latest Skeptic's Circle (#125) is up at Effort Sisyfus, which is run by TechSkeptic. I have a couple of posts in the circle this time around. Hint: Take the blue pill :)

DIY - Very Simple "Hockey Stick"

2009-11-30T09:08:00.000-08:00

A while back, when I analyzed raw data from Mann & Jones (2003), I came up with my own "hockey stick" graph. I didn't post it then, but I thought it might be interesting now, in light of the CRU incident.

AGW "skeptics" are pushing the idea that Phil Jones' "trick to hide the decline", and the VERY ARTIFICIAL correction I discussed in a prior post are essentially evidence of tampering with the "hockey stick" reconstruction.

In reality, the artificial correction refers to rudimentary, probably temporary code (apparently marked with all-caps comments to caution CRU researchers not to use it as final code) that corrects temperatures derived from tree-ring widths, due to a problem known as "tree-ring divergence." It's highly improbable the artificial correction was ever used in any published paper.

This "hockey stick" does not require you to write any algorithms. The only "tricks" involved in producing it are the following:

Temperature data up to 1980 comes from Mann & Jones (2003) (data made available by NOAA.)
Temperature from 1981 onwards comes from the CRUTEM3v global data set.
The red line is a 25-year central moving average of the temperature series.

You can try this yourself with different data sets. It's not very difficult. If you don't trust CRU temperature data, use GISSTemp. If you don't think the Mann & Jones (2003) reconstruction should be used, there are plenty of other historical reconstructions that use methods other than tree-rings. Do report back if it doesn't work. Comment moderation is never enabled here.

[Errata 11/30/2009: The post initially said the red line was a 75-year central moving average. It's actually a 25-year CMA.]

VERY ARTIFICIAL quote-mining

2009-11-26T08:58:00.000-08:00

The CRU stolen emails incident is a big mess, isn't it? What I mean is that it could easily hinder real work that needs to get done, not just in climate science.

I thought I'd lend a hand as a computer scientist who is entirely independent of the politics of AGW. As you can see, I have taken an interest in the topic in the past, but most of the time I'm involved in science debates of a completely different nature.

I've read through some of the quote-mined emails, and I don't really see anything that looks like conspiracy talk of any sort, attempts to cover up data and such. In my view, it mostly amounts to innocuous chatter among scientists, discussion of statistical techniques, speculation, and some honest skepticism. For example, that 2008 was a "cold-ish" year is no secret, and I myself had said that if 2009 is also a cold year, it's possible IPCC predictions might be falsified. (Incidentally, it's not.)

When it comes to accusations of "data cooking," there is one post that caught my attention: Hiding the Decline: Part 1 – The Adventure Begins by Eric S. Raymond. Admittedly, it initially caught my attention because I've thought highly of Mr. Raymond ever since I read The Cathedral and the Bazaar, about 10 years ago. Mr. Raymond opens the post with a snippet of IDL code:


;
; Apply a VERY ARTIFICAL correction for decline!!
;
yrloc=[1400,findgen(19)*5.+1904]
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,- 0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
2.6,2.6,2.6]*0.75 ; fudge factor
if n_elements(yrloc) ne n_elements(valadj) then message,’Oooops!’
;
yearlyadj=interpol(valadj,yrloc,timey)

This certainly looked dodgy to me at first glance. Basically, it looks like a correction that arbitrarily reduces temperatures in the 1930s, and increases them starting in the 1970s. Mr. Raymond posts a graph of the valadj array (which he calls "coefficients") and proclaims that "this isn’t just a smoking gun, it’s a siege cannon with the barrel still hot."

I decided to take a closer look. I found a copy of the FOI2009.zip file, still available at RapidShare. I'm not providing a link, because I'm not completely sure that's legal.

I will post the entire code from the file in question below. I'll highlight the snippet quoted by Mr. Raymond in yellow, and I'll highlight in green other parts of the file that I'd like to discuss.


;
; Now prepare for plotting
;
loadct,39
multi_plot,nrow=3,layout='caption'
if !d.name eq 'X' then begin
  window,ysize=800
  !p.font=-1
endif else begin
  !p.font=0
  device,/helvetica,/bold,font_size=18
endelse
def_1color,20,color='red'
def_1color,21,color='blue'
def_1color,22,color='black'
;
restore,'compbest_fixed1950.idlsave'
;
plot,timey,comptemp(*,3),/nodata,$
  /xstyle,xrange=[1881,1994],xtitle='Year',$
  /ystyle,yrange=[-3,3],ytitle='Normalised anomalies',$
;  title='Northern Hemisphere temperatures, MXD and corrected MXD'
  title='Northern Hemisphere temperatures and MXD reconstruction'
;
yyy=reform(comptemp(*,2))
;mknormal,yyy,timey,refperiod=[1881,1940]
filter_cru,5.,/nan,tsin=yyy,tslow=tslow
oplot,timey,tslow,thick=5,color=22
yyy=reform(compmxd(*,2,1))
;mknormal,yyy,timey,refperiod=[1881,1940]
;
; Apply a VERY ARTIFICAL correction for decline!!
;
yrloc=[1400,findgen(19)*5.+1904]
valadj=[0.,0.,0.,0.,0.,-0.1,-0.25,-0.3,0.,-0.1,0.3,0.8,1.2,1.7,2.5,2.6,2.6,$
  2.6,2.6,2.6]*0.75         ; fudge factor
if n_elements(yrloc) ne n_elements(valadj) then message,'Oooops!'
;
yearlyadj=interpol(valadj,yrloc,timey)
;
;filter_cru,5.,/nan,tsin=yyy+yearlyadj,tslow=tslow
;oplot,timey,tslow,thick=5,color=20
;
filter_cru,5.,/nan,tsin=yyy,tslow=tslow
oplot,timey,tslow,thick=5,color=21
;
oplot,!x.crange,[0.,0.],linestyle=1
;
plot,[0,1],/nodata,xstyle=4,ystyle=4
;legend,['Northern Hemisphere April-September instrumental temperature',$
;  'Northern Hemisphere MXD',$
;  'Northern Hemisphere MXD corrected for decline'],$
;  colors=[22,21,20],thick=[3,3,3],margin=0.6,spacing=1.5
legend,['Northern Hemisphere April-September instrumental temperature',$
  'Northern Hemisphere MXD'],$
  colors=[22,21],thick=[3,3],margin=0.6,spacing=1.5
;
end

Let's talk about the most important finding first. Notice the second section highlighted in green, right below the snippet quoted by Mr. Raymond. There are 4 statements there. The first two start with ";" which means they are commented out. Why is that important? The adjusted yearly data is assigned to variable yearlyadj. The only reference in the file to variable yearlyadj is in the first line that is commented out, where it says yyy+yearlyadj. Notice that the corresponding line that is not commented out only uses yyy in place of that. In other words, as this code stands, the adjusted yearly data is not used at all.

You might ask, why is this "VERY ARTIFICIAL" correction there at all then? I can only guess and speculate. When you're writing software, and you find bugs, a non-brute-force way to debug code is to propose hypotheses as to what is causing the bugs. Then you test these hypotheses. One way to test hypotheses might involve fudging code; trying out different ideas. When I do this I might add strings like "%%%%" to the code, so I know I need to remove that code later. I suppose adding something like "VERY ARTIFICIAL" would work as well.

My guess is that at some point the scientists wanted to see what the plot would look like with this correction, but this correction was obviously not part of the final version of the code.

Note also that this is code for plotting data. It's not code for producing a data set. Claims to the effect that the "data was cooked," seem spurious and overly dramatic. At worst, a graph might not have exactly reflected the raw data, but I wouldn't worry about raw data sets being compromised by the code above. (When it comes to the "hockey stick," what matters is what the raw data tells us.)

To borrow the words of blogger Skeptico regarding a similar incident 4 years ago, I think AGW deniers need to grow up a little.

Early Swine Flu Trend in the US

2009-04-30T07:06:00.000-07:00

I searched Google News for the number of confirmed US cases of Swine flu reported by the Associated Press each day from April 23 to April 28. For the April 29 data point, I used the count currently posted at the CDC website. The chart I came up with follows.

The trend could be exponential, which would be the mathematical expectation. An exponential fit gives a R² (goodness of fit) of 0.94 (very good fit). If the exponential trend were to be maintained, by May 7 there should be over 4,000 confirmed cases of Swine flu in the US. If this prediction fails, it could be an indication that containment measures are having an effect. We'll see.

Update 5/7/2009

I've continued to follow the count of confirmed cases in the US. While it didn't go into the thousands by May 7, an exponential model continues to fit the series quite well.

While cases have reportedly plateaued in Mexico, the same is not true of the US just yet.

Graph of NH SSTs and Named Storms Questioned

2008-08-15T08:51:00.001-07:00

I have written about the association between the number of named storms in the Atlantic basin and Northern Hemisphere sea surface temperature anomalies several times now (last time here). I am quite confident there's a causal association there (even considering the possibility of coincidental trends).

The problem is that my posts on the subject have begged disbelief. You see, the scientific literature is not clear on the matter, and not even top climate scientists seem to agree on whether the association exists. That's why I'm making this spreadsheet available.

In particular, there is a graph that is very difficult to deny. Sometimes you can express doubt about mathematical analyses on technical grounds, but clear and easily reproducible graphs are difficult to argue with. The graph in question is that of 17-year central moving averages of northern hemisphere sea surface temperature anomalies, and the number of named storms in the Atlantic basin, from the 1850s to the present time.

In the new spreadsheet I'm making available, I calculated both 15-year and 21-year moving averages of both data sets. You will find comments in column headers with the URLs of where the raw data comes from. Having to do this seems over the top, but there really are people who apparently don't believe the original graph is real; plus they seem to be misunderstanding the graph completely, as you can see in the comments section of this post at AccuWeather.com.

The 15-year and 21-year CMA graphs are posted below, in that order.

Comment Policy

I will state my comment policy here, for future reference. I do not enable comment moderation. The only comments I delete are those that are clearly in violation of Blogger's Content Policy. Scrutiny is more than welcome. If you believe I made a mistake, tell me. If you believe I'm making things up, you absolutely should tell me, but you better be right.

NOAA Study Seems To Confirm Observation From 07/14 Post

2008-08-12T13:06:00.000-07:00

No so long ago I wrote a follow-up to an earlier analysis on the association between the number of named storms in the Atlantic basin and northern hemisphere sea surface temperatures. At the end of the post I listed a number of conclusions, one of which was the following.

The graph provides support for the contention that old storm records are unreliable. I would not recommend using storm counts prior to 1890.

I had posted a graph of 17-year central moving averages of NH sea surface temperature and named storm series, reproduced below. You will note I had placed a vertical line around the year 1890 in order to indicate there was some sort of point of change there.

I didn't use any mathematical analysis to determine that 1890 was in any way special. It was simply obvious, visually, that something was not right in the named storms series prior to 1890. Of course, the central moving average smoothing helped in terms of being able to see that.

Enter Vecchi & Knutson (2008), a NOAA study of North Atlantic historical cyclone activity. The authors determined, based on known ship tracks, that early ships missed many storms, especially in the 19th century.

Now, this study is being touted as evidence that global warming and the number of storms in the Atlantic are not associated. Clearly, that is nonsense, if you just look at the figure above. If you'd like to see some Math, I have done a detrended cross-correlation analysis as well. All that is necessary to demonstrate an association is to do a linear detrending on series that go from 1900 to the present time. The detrending should take care of any problems related to unreliability of old storm counts. I can further report that even after detrending the series based on 6th-order polynomial fits, a statistically significant association is still there, provided storms are presumed to lag temperatures by at least one year.

About The Disingenuous "Global Warming Challenge" by JunkScience.com

2008-08-12T06:23:00.000-07:00

I read somewhere that JunkScience.com had issued a "global warming challenge" some time back that is promoted as follows.

$500,000 will be awarded to the first person to prove, in a scientific manner, that humans are causing harmful global warming.

That's also what people will say whenever they tout the "challenge." If you are certain anthropogenic global warming is real, you should be able to prove it. Who wouldn't want to make $500,000?

But as you can imagine, there's a catch. You need to falsify two hypotheses.

UGWC Hypothesis 1 Manmade emissions of greenhouse gases do not discernibly, significantly and predictably cause increases in global surface and tropospheric temperatures along with associated stratospheric cooling. UGWC Hypothesis 2 The benefits equal or exceed the costs of any increases in global temperature caused by manmade greenhouse gas emissions between the present time and the year 2100, when all global social, economic and environmental effects are considered.

Now, hypothesis #1 should be falsifiable now. The only issue I have with it is that they have made it unnecessarily difficult (to cover their asses no doubt) by including stratospheric cooling as a requirement. Don't get me wrong. I'm sure stratospheric cooling is an important matter to climate scientists, but why does it matter to the challenge? Isn't surface temperature warming due to anthropogenic causes interesting enough?

Technically, the issue is that there's not a lot of data on stratospheric temperatures, as far as I know. Considering lags and so forth, it's probably difficult to demonstrate an association in a decisive way. I haven't run the numbers, but this is my preliminary guess.

Hypothesis #2 is not falsifiable right now. We'd have to wait until about 2100 to either validate it or falsify it. Peak oil is probably looming or behind us, so we can't say what might happen by 2100. There are policy decisions to consider. There might be technological advances that change the general outlook. If we make certain assumptions, then sure, it's theoretically possible to give confidence ranges on certain predictions, such as sea level rises or changes in storm intensity.

Clearly, the "challenge" is designed such that it's impossible or nearly impossible to win. Despite its name, JunkScience.com is not a site about junk science. If you visit it you will see it's nothing but a propaganda outlet for global warming denialism books and videos. A site that is truly about junk science would probably discuss things like the paranormal, Homeopathy, the vaccine-autism hypothesis, etc. JunkScience.com does not.

In fact, what is the evidence that JunkScience.com has $500,000 to give out? Have they been collecting pledges? If they have collected funds, and there's no winner to their challenge, which I can almost certainly assure you there won't be, will they keep the money?

Call me cynical, but I doubt JunkScience.com is either capable or willing to give out $500,000 to anybody, regardless of the entries they receive.

Counter-Challenge

Here's a counter-challenge for JunkScience.com. Reduce the stakes if you need to. Then change the requirements of the challenge to include a single hypothesis to falsify, as follows.

Manmade emissions of greenhouse gases do not discernibly, significantly and predictably cause increases in global temperatures.

What's there to fear, JunkScience.com?

Just in case there are any doubts about anthropogenic influence in atmospheric CO2

2008-08-08T09:02:00.000-07:00

You would think this is the least controversial aspect of the global warming debate, but you'd be surprised. I realized this after reading some of the comments in a post by Anthony Watts about a recent correction in the way Mauna Loa data is calculated (see also reactions by Tamino and Lucia).

Tamino subsequently wrote an interesting post on differences in CO2 trends as observed in three different sites: Mauna Loa (Hawaii), Barrow (Alaska) and South Pole station. Most notably, there's a pronounced difference in the annual cycle between these stations, which according to Tamino, is explained by there being more land mass in the Northern Hemisphere. I would imagine higher CO2 emissions in the Northern Hemisphere might also play a role, but I'm speculating.

In this post I want to show that available data is quite clear about anthropogenic influence in atmospheric CO2. Additionally, I want to discuss how we can tell that excess CO2 stays in the atmosphere for a long time.

I will use about 170 years of data for this. There's a reconstruction of CO2 concentrations from 1832 to 1978 made available by CDIAC, and derived by Etheridge et al. (1998) from the Law Dome DE08, DE08-2, and DSS ice cores. You will note that there's an excellent match between these data and Mauna Loa data for the period 1958 to 1978. Mauna Loa data has an offset of 0.996 ppmv relative to Etheridge et al. (1998), so I applied this simple adjustment to it in order to end up with a dataset that goes from 1832 to 2004.

CDIAC also provides data on global CO2 emissions. What we need, however, is an estimate of excess anthropogenic CO2 that would be expected to remain in the atmosphere at any given point in time. We could simply calculate cumulative emissions since 1751 for any given year, but this is not necessarily accurate. Some excess CO2 is probably reclaimed by the planet every year. What I will do is make an assumption about the atmospheric half-life of CO2 in order to obtain a dataset of presumed excess CO2. I will use a half-life of 24.4 years (i.e. 0.972 of excess CO2 remains after 1 year). I should note that I have tried this same analysis with half-lifes of 50, 70 and 'infinite' years, and the general results are the same.

Figure 1 shows the time series of the two data sets.

The trends are clear enough. CO2 emissions appear to accumulate in the atmosphere and are then observed in ice cores (and at various other sites like Mauna Loa). Every time we compare time series, though, there's a possibility that we're looking at coincidental trends. A technique that can be used to control for potentially coincidental trends is called detrended cross-correlation analysis (Podobnik & Stanley, 2007). In our case, the detrended cross-correlation is obvious enough graphically, and we'll leave it at that. See Figure 2. Basically, we take the time series and remove their trends, which are given by third-order polynomial fits. You can do the same thing with linear fits or second-order first. The third-order fit is a better fit and produces more fluctuations around the trend, which makes the correlation more obvious and less likely to be explained by coincidence.

With that out of the way, how do we know that excess CO2 stays in the atmosphere for a long time? First, let's check what the scientific literature says on the subject, specifically, Moore & Braswell (1994):

If one assumes a terrestrial biosphere with a fertilization flux, then our best estimate is that the single half-life for excess CO2 lies within the range of 19 to 49 years, with a reasonable average being 31 years. If we assume only regrowth, then the average value for the single half-life for excess CO2 increases to 72 years, and if we remove the terrestrial component completely, then it increases further to 92 years.

In general, it is widely accepted that the atmospheric half-life of CO2 is measured in decades, not years.

One type of analysis that I have attempted is to select the half-life hypothesis that maximizes the Pearson's correlation coefficient of the series from Figure 1. If I do this, I find that the best half-life is about 24.4 years. Nevertheless, I had attempted the same exercise with the Mauna Loa series (1958-2004) previously, and the best half-life then seems to be about 70 years. It varies depending on the time frame, and there's not necessarily a trend in the half life. This just comes to show that there's uncertainty in the calculation, and that the half-life model is a simplification of the real world.

Another approach we can take is to try to estimate the weight of excess CO2 currently in the atmosphere, and see how this compares to data on emissions. The current excess of atmospheric CO2 is agreed to be roughly 100 ppmv. If by 'atmosphere' we mean 20 Km above ground (this is fairly arbitrary) then the volume of the atmosphere is about 1.03x10¹⁰ Km³. This would mean that the total volume of excess CO2 is 1.03x10⁶ Km³, or 1.03x10¹⁵ m³. The density of CO2 is 1.98 kg/m³, so the total weight of excess CO2 should be about 2.03x10¹⁵ Kg, or 2,030,000 millions of metric tons.

Something is not right, though. If we add all annual CO2 emissions from 1751 to 2004, we come up with 334,000 millions of metric tons total. This can't be. I'd suggest that CDIAC data does not count all sources of anthropogenic emissions of CO2. It obviously can't be considering feedbacks either. Furthermore, our assumptions in the calculations above might not be accurate (specifically that a 100 ppmv excess is maintained up to an altitude of 20Km). In any case, it's hard to see how these numbers would support the notion that the half-life of CO2 is low.

Why the 1998-2008 Temperature Trend Doesn't Mean a Whole Lot

2008-08-03T08:14:00.001-07:00

Suppose I wanted to determine whether the current temperature trend is consistent with some projected trend. In order to do this, let's say I calculate the temperature slope of the last 200 days, and its confidence interval in the standard manner. Then I check to see if the projected trend is in the confidence interval. But maybe I want a tighter confidence interval. I could use more data points in this case, say, temperatures in the last 1,000 minutes. If we assume temperature series approximate AR(1) with white noise, this should be fine.

That makes no sense at all, does it?

Intuitively, it seems that confidence intervals on temperature slopes (when we want to compare them with a long term trend) should depend more on the working time range than on the number of data points, or on how well those data points fit a linear regression. We should have more confidence on a 20-year trend than a 10-year trend, almost regardless of whether we use monthly data as opposed to annual data. Certainly, the standard slope confidence interval calculation is not going to do it. We need to come up with a different method to compare short-term trends with long-term ones.

I will suggest one such method in this post. First, we need to come up with a long projected trend we can test the method on. We could use a 100-year IPCC trend line, if there is such a thing. For simplicity, I will use a third-order polynomial trend line as my "projected trend." Readers can repeat the exercise with any arbitrary trend line if they so wish. I should note that the third-order polynomial trend line projects a temperature change rate of 2.2C / century from 1998 to 2008.

The following is a graph of GISS global annual mean temperatures, along with the "projected trend." For the year 2008 I'm using 0.44C as the mean temperature. You can use other temperature data sets and monthly data too. I don't think that will make a big difference.

We have 118 years of 11-year slopes we can analyze. There are different ways to do this. To make it easy to follow, I will detrend the temperature series according to our projected trend. This way we can compare apples with apples as far as slopes go. The detrended series is shown in the following graph.

The long term slope of detrended temperatures is, of course, zero. All 11-year slopes in the detrended series will distribute around zero. We know that the 1998-2008 slope is -1.53C / century. The question we want an answer for is whether the 1998-2008 slope is unusual compared to 11-year slopes observed historically, which would indicate there's likely a point of change away from the projected trend.

We can start by visualizing the distribution of 11-year slopes throughout the detrended series. The following is a graph of the number of years in slope ranges of width 0.2C / century. For example, the number of years that have slopes between 0.1 and 0.3 is 10.

This is roughly a normal distribution of years according to their slopes. In it, approximately 95% of years have slopes in the -2.7 to 2.7 range. That is, 4 years have slopes of -2.7 or lower, and 3 years have slopes of 2.7 or higher. I put forth that the real confidence interval for 11-year temperature slopes relative to long-term 3rd-order polynomial trend lines is approximately ± 2.7 C / century.

The 11-year slope for 1998 is only -1.53C / century, well within the estimated confidence interval. Therefore, it's a little premature to say that the 1998-2008 trend falsifies 2C / century. Of course, if 2009 is a cold year, that might change this evaluation.

Wherein I Revise Previous Sensitivity Estimate Down to 3.13C

2008-08-02T08:03:00.000-07:00

I found an annual reconstruction of CO2 atmospheric concentrations that goes from 1832 to 1978. It is made available by CDIAC and it comes from Etheridge et al. (1998). There's a more than adequate match between this data and the data collected at Mauna Loa, Hawaii for the range 1958 to 1978.

Naturally, I thought this CO2 data would be more accurate than that estimated from emissions, which I had used in my calculation of climate sensitivity to CO2 doubling. (BTW, that calculation was based on 150 years of data). So I reran the analysis, and the following is the new formula for the rate of temperature change (R) given a CO2 concentration in ppmv (C) and a temperature anomaly in degrees Celsius (T).

R = 0.0857 ( 10.398 log C - 26 - T )

The equilibrium temperature (T') is calculated as follows.

T' = 10.398 log C - 26

This means that climate sensitivity to CO2 doubling (based on this model which only considers this one forcing) is most likely 3.13 degrees Celsius.

I also rebuilt the hindcast graph, which follows.

I think this is a subjectively better hindcast than the original. Note that it even predicts a nearly flat temperature trend in the 1950s. This is simply what the more accurate CO2 data does. While sensitivity is lower (I had originally estimated it at 3.46C), the range of CO2 concentrations is wider. Estimations based on emissions produce a concentration of about 295 ppmv in 1850. Etheridge et al. (1998) determines the concentration is 283.5 ppvm at that point.

The model predicts that the rate of temperature change should be about 2.1C / century in 2007.

I also wanted to attempt a 1000-year hindcast. I had previously discussed the 1781-year temperature reconstruction that is the product of Mann & Jones (2003). It just so happens that there's also a 1000-year CO2 reconstruction from Etheridge et al. (1998). Well, this more ambitious hindcast didn't turn out to be as accurate. At first I thought this is just what happens when you fail to consider other important climate forcings. But then I went back and examined other 1000-year temperature reconstructions. I'm sure readers have seen that graph many times. It turns out that there's considerable uncertainty in these types of reconstructions.

Either way, I will post my first attempt at a 1000-year hindcast below. The red line is the reconstruction from Mann & Jones (2003). I also added a green line, which is a reconstruction based on glacier records that comes from Oerlemans (2005).

It could be better. I'm now curious as to what would happen if other major climate forcings were considered.

The "Hockey Stick" is Fine

2008-07-24T08:52:00.000-07:00

There seems to be considerable controversy over the well-known "hockey stick" temperature reconstruction of the last two millennia – Mann & Jones (2003). I have even found what look like accusations of fraud, all embedded in discussions of very complicated statistics and algorithmic procedures that the average person couldn't possibly hope to evaluate.

I'm not interested in getting involved in the politics of the whole thing. I just want to point out that the raw data of the temperature reconstruction is made available by the NOAA Paleoclimatology Program. I contend that most people reading this can double-check if the raw data tells us we are living in unusually warm times – which is basically what the "hockey stick" construct conveys.

Of course, there are those who will say that we are living in unusually warm times relative to most of the last thousand years simply because the little ice age has ended. But we can control for this fairly easily.

There is a general temperature trend historically. We can remove this trend from the data, and then check if we're still living in unusually warm times after the removal. Specifically, we want to remove the warming trend that is a natural part of the end of the little ice age.

I would suggest that a 4th-order polynomial trend line will capture the general temperature trend of the last 1781 years more than sufficiently. (Excel will produce polynomial trend lines for you, up to 6th-order ones). The trend is characterized by a medieval warm period, followed by a period of cooling, and a subsequent period of warming. We can detrend the temperature time series based on the polynomial fit and see if the modern era remains special.

This sort of detrending methodology has apparently been used in climatology before. Holme et al. (2008) point out that "sophisticated statistical methods have been applied to [climate] series, but perhaps sometimes these methods might even be too sophisticated." They further claim that "the [detrending] method provides a rigorous way of defining climate 'events', and allows comparison of long-term trends and events in time series of climatic records from different archives."

The detrending method in Holme et al. is actually more sophisticated than what we can do in a straightforward manner, but the authors are interested in long-term quasiperiodic trends.

Let's first see what the temperature time series looks like, along with the proposed 4th-order polynomial fit. We will only be looking at the global temperature reconstruction in this post.

In order to detrend the time series, we simply subtract temperatures modeled by the polynomial equation from observed (reconstructed) temperatures. The Y axis offset is not important to this analysis. (Note that in the equation shown in the figure above, x = year - 199). The result of the detrending procedure is illustrated in the following figure.

So now we have a nice detrended temperature time series, which – if I may be redundant – has an entirely flat trend. What do we do with it?

Let's sort data rows by detrended temperature in descending order. If we look at the top 5% (89) years ranked in this manner, we see that they have a detrended temperature greater than 0.123. In other words, if we were to pick a year at random from the data set, there is only a 5% chance that its detrended temperature is greater than 0.123. (If you must know, the residuals of the polynomial regression are normally distributed).

In statistics, a 5% probability is the standard for rejection of hypotheses. If we hypothesize that a given year is not an unusually warm year, its detrended temperature should be 0.123 or lower. Yet, this is not the case for many of the years in the modern era, as shown in the following figure.

All but 3 of the years from 1968 to 1980 are statistically warm years, even after detrending the whole 1781-year time series. This cannot be explained as a consequence of the culmination of the little ice age. Clearly, we are in the midst of a "climate event."

Is it an unprecedented event? If you only consider the 1968-1980 range as special, then no. There was an 11-year "climate event" between the years 668 and 678 when detrended temperatures were higher than 0.123. That is the closest precedent that can be found in the 1781-year temperature series. If we consider that temperatures have increased after 1980, then I'd have to agree with Mann & Jones that modern era global warming "dwarfs" anything from the last 2 millennia.

Global Warming Forecast - Based on 3.46C Model

2008-07-20T08:51:00.000-07:00

So far we have estimated climate sensitivity to CO₂ doubling, and tested the results of the analysis with a hindcast. I will close the series with a forecast.

It will be a simple forecast in the sense that we will only consider CO₂ trends. While I would caution this is an important limitation of the forecast, I would also note the hindcast had the same exact limitation. Of course, it's quite possible that in analyses of historic data, CO₂ acts as a proxy of other anthropogenic forcings. The behavior of this confounding in the past may differ from its future behavior.

That said, the part of the forecast that I really can't be very confident about has to do with projecting future CO₂ atmospheric concentrations. This basically amounts to attempting to predict human behavior and world-wide policy decisions. What I will do is to simply define 2 scenarios based on the Mauna Loa data, as follows.

Scenario A: A second-order polynomial forecast of CO₂ concentrations.

Scenario B: A third-order polynomial forecast of CO₂ concentrations.

Each scenario is illustrated in the following graph.

If it is true that peak oil is either looming or behind us, I would say Scenario B is considerably more likely.

To get "high" and "low" estimates I was initially planning to use the 95% confidence interval of the rate of temperature change formula. This range produces forecasts that are very similar. So instead what I did is produce new formulas for sensitivities of 3.0C (low) and 4.0C (high). For additional details on how the forecast is done, see the hindcast post.

The resulting forecasts of each scenario are illustrated in the following graphs.

Again, I consider scenario B to be more probable. We'll see how they do. Under either scenario it would seem that a global temperature anomaly of 1 degree Celsius by the early 2020s is a done deal. The model also tells us that it takes about 10 years for temperatures to level off after CO₂ concentrations do. Under scenario B, we are apparently at a peak in the rate of temperature change – roughly 2C/century. This rate will begin to drop. It will be 1.5C/century by 2035.

How Well Does a Sensitivity of 3.46C Hindcast?

2008-07-18T16:38:00.000-07:00

[Note: Revised 08/02/2008]

In the last post we estimated the most likely climate sensitivity to CO₂ doubling by means of an analysis of temperature change rates. The result (3.46C) is in the high end of the range of sensitivities considered plausible by the scientific community. A hindcast should not only tell us if the estimate is in fact too high, but it should also test some of the other results from the analysis. And to make it interesting, we will do a hindcast of the last 150 years. Sound crazy? See Figure 1.

This turned out much better than I expected. In fact, I suspect the chart might beg disbelief among some readers, so I'm making the spreadsheet available here (XLS). Formulas can be verified to match those of the analysis.

The only inputs to the hindcast are (1) CO₂ atmospheric concentrations from 1853 to 2004 (estimated in ppmv as described at the end of this post), and (2) observed temperatures from 1853 to 1856. The observed temperatures used (Column D) are actually central moving averages of period 7.

My expectation for the hindcast was that error would accumulate, and in the end we would have a deviation from the observed temperature trend, but hopefully not a big one. That's because the way temperature for year Y is predicted in the hindcast is by adding the temperature in Y-2 plus the predicted temperature change rate in Y-1 times 2. Intuitively, it doesn't seem like this technique would tend to maintain accuracy over a time series this long.

There is a good reason why the model hindcasts this well, nevertheless. First, it helps that formulas were derived in part from the data we're hindcasting. But more importantly, what we're looking at is a self-correcting system. Local variability cannot make the system resolve its imbalance any faster or slower. If temperature becomes higher than it should be, for whatever reason, the temperature change rate will drop. Similarly, temperatures lower than they should be will be corrected by a positive change in the rate. Sooner or later, the observed trend will rejoin the predicted trend.

This speculative observation is testable in the hindcast. We can break the chain of predicted temperatures, insert artificial values, and see if the model resolves. This can be done in the spreadsheet by modifying one of the predicted temperature columns (e.g. column K, any row greater than 9). What I did is introduce an artificial warming between 1910 and 1913 so it ended up at 0.1C. The results can be seen in Figure 2.

I think that's interesting, and I'm sure there's some insight about what's been occurring since 1998 somewhere in there.

For those who are interested in the details, the following is a recap of the results from the analysis that are used to produce the hindcast.

T' = 11.494 log C - 28.768

R = (T' - T) * 0.0915

An unexplained lag of 3 years for imbalance to take effect on the rate of temperature change.

Where

C = The atmospheric concentration of CO₂ given in ppmv.

T' = The equilibrium temperature, given in degrees Celsius anomalies as defined in CRUTEM3v data set.

T = The observed temperature. In the hindcast, this is actually the predicted temperature, except for 4 years we use as inputs.

R = The rate of temperature change, given in degrees Celsius per year.

The high and low hindcast predictions are based on the confidence interval given in the formula for R.

As an example, the following is how the predicted temperature for 1857 is calculated.

T(1857) = T(1855) + 2 * R(1856)

R(1856) = 0.0915 * (T'(1853)-T(1853))

That's all the hindcast is.

Next up: We'll attempt a forecast.

Here's How You Can Estimate CO2 Climate Sensitivity From Historic Data

2008-07-15T09:42:00.000-07:00

Most likely value = 3.46C

[Note: Revised 08/02/2008]

When I first became interested in the science of Global Warming (which was not too long ago) I had some substantial misconceptions. For example, I thought the current temperature anomaly (about 0.6C globally) was due to the current levels of greenhouse gases in the atmosphere, primarily CO₂ (about 380 ppmv). Reality is more complicated. The issue is not that there's some lag between greenhouse gas concentrations and temperature either – it's a bit more complicated that this.

I've been learning about a concept called CO₂ climate sensitivity, which is defined as the equilibrium temperature increase expected if the atmospheric concentration of CO₂ were to double. The word equilibrium needs to be emphasized. At current CO₂ concentrations, I would estimate the equilibrium temperature anomaly should be 0.89C, but the actual temperature anomaly is only about 0.6C. There's a significant imbalance, and the imbalance is corrected by temperature change. Simplifying, the mechanism that causes temperature change is called CO₂ forcing.

There is much debate and uncertainty about the most likely climate sensitivity value. For a good overview, see James' Empty Blog.

What I want to do in this post is go over a relatively simple analysis where we estimate climate sensitivity by using publicly available historic data. We will also come up with formulas that tell us the most likely equilibrium temperature for a given CO₂ concentration, and the most likely temperature change rate for a given actual temperature and CO₂ concentration. The plausibility of these results will be illustrated with a graph.

First, let's go over some of the underlying theory. Given the way climate sensitivity is defined, it's clear that the expected equilibrium temperature change is the same for any doubling of CO₂ concentrations, be it from 100 to 200 ppmv, or 1000 to 2000 ppmv. This tells me there's a logarithmic relationship between temperature and CO₂ concentrations (assuming all else is equal) as follows:

T' = a log C + b

T' is the equilibrium temperature and C is the atmospheric concentration of CO₂; a and b are constants. Climate sensitivity is thus

S = (a log 2C + b) - (a log C + b) = a log 2

When the observed temperature (T) differs from the equilibrium temperature (T'), there's imbalance. We will define imbalance (I) as follows.

I = T' - T

Further, I put forth that temperature change rate is given by

R = d I

where d is a constant. We're guessing a bit here, but the above is consistent with Newton's Law of Cooling.

Finally, let me define a construct (J) that I will use in the analysis. It is simply the imbalance minus the constant b, as follows.

J = I - b = a log C - T

If we know S, then we know a. When we have S, a and C for any given year, we can calculate J for any given year. Since we should be able to determine the temperature change rate (R) for any given year, we can model J vs. R (a linear relationship). The relationship between J and R should be equivalent to the relationship between I and R, except for a shift given by the constant b.

Here's the plan. We need to test different hypotheses on the value of S. The way we determine a hypothesis is good is by checking if the resulting relationship between I and R is suitable. And we measure this by means of the "goodness of fit" of the linear association between J and R. (This methodology is called "selection of hypotheses by goodness of fit" and it seems adequate in this case, judging by Figure 3, which I will mention shortly).

Before I get into the nuances of the analysis (which are important) I wanted to show the reader how I chose the best value of S. Figure 1 models S vs. the goodness of fit of the linear association between J and R.

This tells us that the value of S that makes most sense is 3.46.

After we have determined the most likely value of S, we can calculate the constant b. The linear association between J and R is as follows.

R = 0.09152J - 2.63281

The slope should be the same in the association between I and R, except here the intercept must be zero.

R = 0.09152I

Therefore, b may be calculated as follows.

0.091521I = 0.091521(I - b) - 2.63281
b = 2.63281 / 0.091521 = -28.768

Figure 2 is the scatter graph that illustrates the association between imbalance (I) and temperature change rate (R) when we assume S=3.46. This confirms the slope of the linear fit and the "goodness of fit" we had previously found.

A very important graph is one that shows the R and I time series side by side, under the same assumption (S=3.46). See Figure 3.

Figure 3 validates much of the underlying theory. It's one of those graphs that, once again, show anthropogenic global warming to be an unequivocal reality.

Figure 3 can also be used to visually check different values of S. When S is less than 3.46, you will see the imbalance (I) time series rotate in a clockwise direction. When it is greater than 3.46, it will rotate in a counter-clockwise direction. This provides subjective confidence about the adequacy of the hypothesis selection methodology.

Note that the imbalance (I) time series in Figure 3 is shifted three years to the right. An initial inspection of the graph clearly showed there was a lag of 3 years between imbalance and temperature change rate. I would've expected the effect to be immediate, but that's why it's important to put your data in graphs. I couldn't begin to theorize why it takes time for imbalance to take effect, but this finding needs to be taken into account in the analysis; otherwise the results won't make sense.

Another important aspect of the analysis is that time series noise needs to be reduced, otherwise you probably won't notice details like the 3 year lag. I calculated central moving averages of period 7 from the CRUTEM3v global data set. For example, the "smooth" temperature for 1953 is calculated as the average between 1950 and 1956. Additionally, the temperature change rate (R) is calculated based on the "smooth" temperatures, looking 4 years ahead and 4 years in the past. If you also consider the 3 year imbalance lag, this leaves us with a workable time range spanning 1859 to 2000.

How do I get CO₂ concentration data spanning that time frame? I discussed how I estimate that here. Basically, I try to find the best possible constant half-life of extra CO₂ by matching emission data with the Hawaii data. The best half-life is 70 years or so.

I should note that this technique produces pre-industrial CO₂ concentrations that are higher than I believe is generally accepted. My estimate gives about 294 ppmv for the 1700s. From ice cores, I understand the concentration has been determined to be 284 ppmv circa 1830. However, I can report that I tried a different estimation method that produces a value closer to 284 ppmv in the early 1800s, and this data produces much poorer fits in the analysis. For this reason, I went with my original estimation based on a constant half-life.

Let's look at the results of the analysis.

S = 3.46

T' = 11.494 log C - 28.768

R = 0.0915I [ 95% CI 0.074I to 0.109I ]

Temperatures are given as anomalies in degrees Celsius, as defined in the CRUTEM3v data set. The rate of change (R) is given in degrees per year.

What's the confidence interval on S? We'll leave that as an unsolved exercise. It's not only that there's uncertainty on the various data sets used, but it's unclear how we would calculate the uncertainty on the best "goodness of fit." It's not a matter of calculating confidence intervals on R² values, which is easy. We basically have to determine the likelihood that the best "goodness of fit" is other than the one we found. This seems non-trivial, but maybe a reader can suggest a method. From what I've seen in a visual inspection of Figure 3, I would say S is unlikely to fall outside the range 2.8 to 4.0. Of course, things might happen in the future which invalidate these results, as they are applicable to historic data.

Next up: We'll see how well these results hind-cast.

Hurricanes and Global Warming - Revisited

2008-07-14T07:01:00.000-07:00

I previously wrote an analysis on the association between sea surface temperature and named storms. The post met some scrutiny which was actually pretty decent, primarily from a commenter named Kenneth Fritsch over at Climate Audit. I understand Climate Audit is one of the major AGW denial blogs.

I had conjectured that when detrending time series, closer fits will tend to better control for coincidence. This intuition makes perfect sense, in my view. Consider that detrending with a linear fit is better than not detrending at all. After that, it's not hard to imagine there are coincidental time series where linear detrending does not make sense at all. I've also found time series where a second-order detrending is quite poor, and I've had to use a third-order detrending. The cumulative CO₂ emissions time series is case in point.

The problem with detrending too closely is that there is some loss of information. To give you an example, if we only had 7 data points and detrended them using a 6th-order fit, the fit would be perfect, and we'd be left with zero information. This is presumably not so much of an issue when you have many data points, but there has to be some loss of information either way.

Kenneth had tried my analysis with a 6th-order detrending and found that statistical significance was lost. This was interesting, but I subsequently pointed out that if you attempted the association by assuming there's a lag of 1 year between temperature and storms, statistical significance remained. I had previously found a lag of 1 year produced a better association than a lag of 0 years, and the 6th-order detrending confirms it. The 6th-order detrending is pretty remarkable too. There are no hints of cycles in a visual inspection of the detrended time series.

The exercise left me quite sure that there was still an association, but I got the sense that there's something missing as far as convincing some readers. I think many people are unconvinced by slopes, confidence intervals and theoretical Math. You need a good graph to be convincing. Unfortunately, both the temperature data and the storms data contain a lot of noise. You can sort of see a pattern if you look closely, but it's not something that is slam dunk convincing.

So I had an idea. We just need to smooth out the noise. And what's a simple way to smooth out noise? We just get central moving averages. In fact, this idea is so simple that I'd be very surprised no one has thought of it before. Here's what I did. For the year 1859 I calculated the "smooth" temperature as the average of raw temperatures from 1851 to 1867. For the year 1860, it was the 1852-1868 average, and so forth. Same for named storms. The resulting graph follows.

At times I think a better name for this blog might have been "Deny This." :)

Some remarks:

The effect given by a straight comparison of the time series appears to be 8 storms for every 1 degree (C). This is somewhat higher than the effect I had previously reported from an analysis of the residuals, which was 6 storms for every 1 degree.
The graph provides support for the contention that old storm records are unreliable. I would not recommend using storm counts prior to 1890.
My prediction that at an anomaly of 2 degrees (C) the average season will be similar to the 2005 season is unchanged.
The lag from the graph appears to be 2 years, and not 1 year, as suggested by various analyses of residuals.

Post on Global Warming Appears to Upset Denialists

2008-07-12T07:04:00.000-07:00

A couple weeks ago I wrote a post in my primary blog that, if I may say so myself, convincingly and conclusively shows anthropogenic global warming is a reality. I believe the analysis is such that you don't need to have a degree in Math to follow it.

Not surprisingly, some global warming "skeptics" showed up in the comments and argued some points that are, frankly, not relevant to the analysis. But they were mostly civil. More recently, however, a commenter shows up, saying things like...

Good grief! There is too much wrong with this analysis to do a thorough critique... There is nothing at all impressive about your statistics...

Personally, I find these types of comments fairly rude, but that wouldn't matter so much if the commenter had actually advanced some challenges of note. Have you ever encountered guys like this? While this is the first time I've come across global warming denialists, I do have considerable experience with their anti-science counterparts in the autism community. We call them "anti-vaxers" and "the mercury militia." I doubt global warming denialists are nearly as nasty, though. But I digress.

Additionally, it's a little funny that the guy hadn't apparently read the post at all, judging by the following comment.

Also, there's not a thinking person on the planet who disagrees that from 1850 to present both carbon dioxide and temperature have increased. That alone will cause a positively-sloped line.

In the first paragraph of my post I had made it perfectly clear that my intention was to test a methodology that controls for potentially coincidental trends. In the first paragraph! I don't think I would've bothered to do a global warming analysis otherwise. You have to keep in mind that I have no dog in this fight (except perhaps for the fact that I live in this warmed up planet). My interest in the topic is scientific and not political.

This is a good opportunity to repost more clear versions of the figures from the analysis, nevertheless. Figure 1 shows the two time series without any adjustments. Figure 2 shows the residuals of the time series relative to the modeled trend lines. I've come to realize that a more intuitive way to think of Figure 2 is as a detrending of the time series from Figure 1. Note that in Figure 2 the residuals of temperature are calculated from a temperature time series that is 10 years ahead of observed values. I've also widened the CO₂ Y scale a bit for clarity.

I encourage the reader to click on the figures to get familiar with their nuances. Print them if you prefer. I hereby also grant permission to use these images in any way the reader sees fit.

Note that Figure 2 includes linear fits of both detrended time series. The fits are completely flat. This means that the temperature residuals are not associated with the year, and neither are the cumulative CO₂ residuals. Any independent property of the year should not associate with either. If the residuals cross-associate, at 99.99999999% confidence, then it's very difficult to argue that we're not looking at an actual effect.

Let me get back to some of the points the commenter raised.

If you wish to prove Anthropogenic Global Warming, you'll need to use temperatures from the whole globe. You cannot simply ignore the entire Southern Hemisphere. And you really should test other temperature data sets using your methodology...

Here the commenter seems to be suggesting that finding an effect of CO₂ on Northern Hemisphere (NH) temperatures is not convincing enough. Unless we can show the whole planet is affected, it doesn't really matter if CO₂ is warming the NH. Plus we have to show this using all data sets. Amazing.

When I first did the analysis, I didn't know much about all the data sets available. I just wanted to find one that contains as many data points as possible. When it came time to pick a data set, I chose a NH one simply because most CO₂ is generated in the NH, and so by choosing this data set theoretically less noise would be introduced in the analysis.

The general temperature trend behavior is similar when you compare the globe with the NH and SH, even though the size of the effect of greenhouse gases varies. This is true of all data sets. If the commenter hopes the analysis won't hold if we look at different temperature data sets, frankly, he's engaging in self-deception.

When you're trying to validate a theory, you have to use measurements of what's ACTUALLY IN THE THEORY. For AGW, this means you have to model the CO2 concentrations in the atmosphere.

Here the commenter is suggesting that cumulative human CO₂ emissions are not a good proxy of the CO₂ concentrations in the atmosphere. This is not true, as I will elaborate on, but in any case, how does this explain the association found?

As far as I know, data on CO₂ atmospheric concentration is only available for the range 1958 to 2004. I don't believe this is enough for this type of analysis considering how noisy the data in question is. Would you find Figure 2 convincing if you could only see a third of the graph? But more importantly, early on I realized that if I wanted to make an argument about anthropogenic global warming, it was key to look at the human contribution of CO₂.

I have modeled cumulative CO₂ emissions vs. atmospheric concentrations at Mauna Loa, Hawaii. The fit is excellent. For those who are versed in statistics, if I put both data sets in a scatter and do a linear fit, the R² of the fit is 0.9981.

I can get slightly better fits by assuming there's a constant half-life of CO₂. To do this I use a simple model where our total atmospheric contribution at any point in time is calculated as follows.

total(year) = (total(year - 1) + emissions(year)) * constant

The constant is what tells us how much of the extra CO₂ we've put into the atmosphere is lost after 1 year. Of course, we're assuming that naturally produced CO₂ is in equilibrium with the environment; which was roughly the case before the industrial revolution.

I've tested different values of constant and compared the resulting R² fit measures of the linear association between total emissions and atmospheric concentrations. The results can be seen in the following graph.

What this tells us is that the best values of constant are somewhere between 0.99 and 0.9908. These translate to an atmospheric half-life between 69 and 75 years.

None of this detracts from the fact that cumulative emissions are an excellent proxy of our contribution to atmospheric concentrations. But in case readers have any doubts, the following is a graph of anthropogenic CO₂ contribution where we assume a half-life of 69 years. Please compare and contrast with Figure 1.

Evidently, this is all just a distraction from the facts in evidence: An association was found, and data imprecisions cannot explain it away.

Shouldn't It Be Considerably Warmer?

2008-07-07T09:20:00.000-07:00

In a prior residual correlation analysis of cumulative CO₂ emissions and northern hemisphere temperatures the effect found appeared to be much larger than expected for short-term fluctuations. It was a clear effect, too, in the sense that it was evident graphically. I speculated that cumulative CO₂ emissions are probably not a good reflection of actual atmospheric concentrations because some CO₂ probably does get removed from the atmosphere after some time.

That finding peeked my interest though. In the original analysis, I basically assumed the half-life of CO₂ was 'infinite'. We were only interested in fluctuations from the general trend, so the assumption was sufficient to prove a point then.

I subsequently went ahead and calculated human CO₂ contribution assuming a constant atmospheric half-life of 50 years. (A constant half-life doesn't match up with the numbers very well, but we'll set this aside for the time being). Going from a half-life of 'infinite' to a half-life of 50 years, I expected to see a decreased effect.

Instead, the effect was about the same, using the best fluctuation lag I had previously found: 8 years. The slope was 3.181x10^-5 ± 9.927x10^-6. By matching up with atmospheric concentration data sampled at Mauna Loa, Hawaii, this translates to 0.081 (± 0.025) degrees (C) for every 1 ppmv increase in CO₂ concentration. (I've done the analysis in other ways which I'm not going to go into, and I'm confident this is about right).

Keeping in mind that this was a northern hemisphere temperature analysis, the effect is still huge. Assuming the relationship is linear, it would mean that a fluctuation of 100 ppmv should result in a temperature fluctuation of about 8 degrees (C). At this point is when I started to think of where the error might be. Of course, there are subtleties involved in how such a result should be interpreted, and I'll get to that, but I kept coming back to a graph I had previously seen.

In this graph we see that, historically, a fluctuation of 100 ppmv CO₂ corresponds to a fluctuation of 8 to 10 degrees (C). I realize there are feedbacks involved, but this is interesting nevertheless.

Could it be that at current CO₂ levels the expected temperature anomaly should be 5 or 10 degrees, as opposed to 1 degree? Let's consider the finding that a fluctuation of 1 ppmv should result in a temperature increase of about 0.05 degrees globally. In the analysis, 8 years were enough for this temperature increase to be realized for such a small fluctuation. Let's round that to 10. Temperature cannot increase with arbitrary speed I suppose. If it takes 10 years for a 0.05 degree increase, could it be that it takes 1,000 years for an expected 5 degree increase to materialize?

No, I don't think so. The rate of temperature increase cannot be constant or bounded by such a low value. If it were, we would not be able to detect short-term CO₂ increase effects. Temperature would already be slowly working its way up towards a target and small green house gas fluctuations would not have an effect in the rate of increase. So instead of 1,000 years, we could be talking about hundreds or less.

What's going on with the data is not very intuitive, so I came up with an analogy that I believe is helpful. Imagine the planet is a car and its temperature is the speed of the car. Pumping CO₂ into the atmosphere would be analogous to pressing the gas pedal. When you press the gas pedal, there will be an immediate effect: the speed of the car (temperature) will begin to increase, but it will take some time until it reaches a stable speed. The more you press the gas pedal, the faster the speed increase, but the target stable speed is farther ahead.

This suggests we've been looking at the results of the fluctuation analysis all wrong. It tells us not about the effects of CO₂ concentrations on temperature, but about its effects on temperature increase. This is an important distinction. In the end, what we're seeing in the analysis is that for every 1 ppmv fluctuation, there's a fluctuation of about 0.008 degrees per year in the rate of increase of temperature (maybe 0.005 globally). But once again, this relationship cannot possibly be linear. It all gets fairly complicated from this point forward.

I presume climate models take this into account, either implicitly or explicitly. But I've never heard it explained this way. It is mistaken to suppose that current CO₂ levels are what drive current temperature levels; they actually drive the rate of increase of temperature up to a target temperature that is probably very far off yet. I'm no climate scientist, but this seems quite obvious in retrospect.

If my intuition is correct, some additional questions come to mind.

If CO₂ were to level off at current levels, would temperature continue to increase? For how long? Up to what point?
Does this all mean CO₂ levels should be brought down to at most 300 ppmv for species in this planet to be able to survive long term?
Should we expect an acceleration of the rate of increase of temperature? Is there a limit to how fast it can increase?

"There is a much better correlation between sun activity and temperature"

2008-07-05T15:37:00.000-07:00

Shortly after I wrote my first post on global warming, a commenter noted that "there is a much better correlation between sun activity and temperature." I've read other blog discussions on the topic, and this seems to come up from time to time.

So I decided to put the data in scatters to see if there's any merit to this claim. I'm not going to standardize the data in any way. These will be straight plots of existing data.

First, let's look at a scatter (Figure 1) of atmospheric CO₂ concentration vs. global temperature anomalies 8 years later from 1959 to 1999 (corresponding to 1967 to 2007 for temperature).

Why 8 years later? This is the best lag I found in my initial analysis of CO₂ emissions vs. temperature anomalies. Even without this lag, you will find a similar association. The 8 year lag is probably an underestimate when we're talking about long-term increases in CO₂. That was a lag applicable to a fluctuating trend. (And yes, this is bad news).

Finally, let's look at a scatter (Figure 2) of SunSpot number vs. global temperature anomaly, between 1881 and 2007.

Is that what they call a "much better correlation"?

Hurricanes and Temperature are Indeed Associated

2008-06-28T11:41:00.000-07:00

There is apparently considerable climate science that can be cited to show there's a clear association between global warming and either the number of hurricanes in any given season or their intensity. See, for example, Hurricanes and Global Warming - Is There a Connection?, written by a number of climate scientists who run RealClimate.org. There is both basic science and computer modeling that can be used to predict what should occur under certain warming scenarios.

I'm generally inclined to trust scientific consensus and published science, particularly if it's peer-reviewed, unless I can advance a seriously strong argument explaining why I do not. Nevertheless, there's nothing like analyzing data first hand. Because I understand this, and because I understand some people out there don't trust some published science at all under the pretext of "conflicts of interest," I've acquired the habit of writing posts where I walk the reader through very accessible analyses of publicly available data. I combine this with a very lenient comment policy. My pledge is to only remove comments that clearly violate Blogger's content policy.

I already did this type of analysis in my post titled Anthropogenic Global Warming is Absolutely Occurring. This time I will look into the claim that global warming might have had an effect in the number of named storms in the Atlantic Basin, given that some people appear to doubt this claim. In doing so, I will try to go over additional details of the methodology which I might have left out in my previous post.

I will use data on the number of named storms from 1851 to 2006 provided by NOAA. I will use ocean surface temperature data for the northern hemisphere provided by the Climatic Research Unit of the University of East Anglia. For accuracy, since we're interested in the hurricane season, I will use June-November averages for each year.

Let's start by putting these two data sets in a chart, side by side. This will be Figure 1, which also shows trend lines for both temperature and storm trends. The trend lines are third-order polynomial fits (easily produced with Excel).

The reader will note that both trends are pointing upward, at least for the last 60 years. This is not what we are interested in, however. We want to control for the fact that there could be a coincidence of upward trends. That's where the third-order polynomial fits come in.

The polynomial fits provide us a time-based model of each trend. For any given year they tell us what the "expected" temperature and number of storms should be. Of course, a given year might have more or less storms than expected. It will also have a higher or lower temperature than expected. In the end, what we want to find out is whether years with higher temperature than expected tend to have more storms than expected, and vice versa.

By subtracting trend line equation values from observed values, residuals of temperature and storms can be produced for each year. These residuals represent how different from "expected" an observed value is in a given year. Residuals are generally time-independent. In our case, if you produce a scatter chart of year vs. temperature residual or storm residual, you will see the scatter trend is entirely flat. This is a basic confirmation that should be done after getting the set of residuals.

Figure 2 is a scatter chart of temperature residuals vs. storm residuals. The trend of this scatter should be flat, unless there's association between temperature and number of storms.

What we see in Figure 2 is that if we try to fit a linear trend to the scatter, we do get a positive slope of 3.43. Now, we need to verify that we can state, with statistical confidence, that the slope is actually positive. In this case it is. The 95% confidence interval of the slope is 0.25 to 6.61. This is not a slam dunk finding like the one for the correlation between cumulative CO₂ emissions and temperature, but it is statistically significant, which means an association between temperature and number of storms is demonstrated.

Given the methodology used, this result cannot be explained as a coincidental trend.

There are some peculiarities about the data which are interesting. For example, it is clear that the 2005 Atlantic season was an unusual one, even after controlling for the time trend of named storms. It could be placed in a group of seasons that only occur every 50 years or so. Evidently, the fact that the seasons that came after 2005 did not measure up is inconsequential to the finding that temperature associates with the number of named storms.

We can, however, pose the following question: What sort of temperature increase would be required for the average season to be like the 2005 season? Given the slope of the scatter in Figure 2, it would seem that a temperature anomaly of 4.05 degrees (C) would be required for this. The current temperature anomaly is about 0.6 degrees (C), so such an eventuality appears to be far off. Or is it?

I ran a second residual correlation analysis of temperature vs. number of named storms one year later. This actually produces a considerably steeper slope (6.36) and the confidence interval is entirely positive even at 99.993% confidence. I can't really explain why this would be the case. But here's the thing. If we were to take this new slope at face value, a temperature anomaly of 2.18 degrees (C) would be enough to make the average season similar to the 2005 season.

Anthropogenic Global Warming is Absolutely Occurring

2008-06-28T11:36:00.000-07:00

[Originally posted at Natural Variation.]

I need to ask for the reader's indulgence, as this post is not about autism, except insofar as determining the merit of correlations has become a perseveration of mine. You see, it is trivial to come up with naive correlations of autism trends vs. practically anything about the modern world. The administrative prevalence of autism has been increasing almost always since records have been kept. Concurrent upward trends of nearly anything, from vaccines to environmental pollution, from trans fats to electromagnetic radiation, and so on, are easy to come by.

In my latest post at LB/RB I suggested that instead of correlating trends in a naive manner, we could attempt to correlate the residuals of time regression models of each trend. A residual is a delta or difference between an observed value and a modeled value. (Here's a concise explanation).

When modeling real world phenomena, regression models will never (or almost never) be perfect fits. For all sorts of reasons, even if simply random fluctuation, there will be deviations from a modeled trend. If there's a causative relationship between two trends, the residuals of (or deviations from) corresponding close-fitting regression models should correlate with one another as well. By this I don't mean that the residuals should always be in the same direction; but they should be in the same direction more often than not, in average.

The nice thing about this technique is that it is completely accessible to anyone with Excel installed. It can also be illustrated graphically, as the reader will see.

So it occurred to me to test this idea in a different field of science where there's controversy over correlation vs. causation. I thought global warming would be a great candidate. After all, the spoof about a decrease in the number of pirates correlating with many other arbitrary trends appears to originate in the global warming debate (see this).

To summarize what I found, there is a strong and statistically significant correlation between cumulative human CO₂ emissions and northern hemisphere temperature anomalies. Because of the methodology used, I'm quite confident this cannot be explained by coincidence, data collection errors, solar output as a confound, or causation in the opposite direction.

Now, I fully recognize that I'm only superficially familiar with the debate over anthropogenic global warming. I am also not versed in climatology. Therefore, I cannot be entirely sure that this type of analysis hasn't been done before. Google and Google Scholar searches didn't seem to turn up anything, and given the importance of the topic, I thought it was not only prudent but necessary to put this evidence out there. As always, scrutiny and discussion are welcome.

Northern hemisphere temperature data from 1850 to 2004 was obtained from the Climatic Research Unit of the University of East Anglia, UK.

Global CO₂ emission data was obtained from CDIAC. I did not use CO₂ atmospheric concentration data because temperature increases can theoretically cause this concentration to increase. Human emissions are what we're interested in. More specifically, I calculated cumulative CO₂ emissions for every year since 1850. Greenhouse temperature anomalies are presumably caused by the total amount of CO₂ in the atmosphere, not by the emissions in any given year. Since CO₂ stays in the atmosphere for 50 to 200 years (source) modeling the cumulative human contribution of CO₂ should be adequate enough.

Figure 1 (click to enlarge) is a graph of the general time trends of these two sets of data. It also shows the modeled trend lines we will use to calculate residuals. In this analysis we're using third-order polynomial models. They seem to give a considerably closer fit than second-order polynomial models.

I calculated the residuals and built a scatter graph matching cumulative CO₂ (X axis) and temperature (Y axis) residuals for each year from 1850 to 2004. As expected, the slope of a linear regression of the scatter was positive (1.9x10^-5) and statistically significant (95% confidence interval 1.13x10^-5 to 2.66x10^-5).

[Note: Instructions on how to calculate the slope confidence interval of a linear regression with Excel can be found here.]

I suspected, however, that there should be lag between cumulative CO₂ fluctuations and temperature fluctuations. It presumably takes some time for heat to be trapped. I proceeded to create a moving average trend line of the temperature residuals. It did in fact have a similar shape to the cumulative CO₂ residuals graph, but it appeared to lag it by about 10 years. The reader should be able to roughly see this lag in Figure 1.

So I re-ran the whole analysis by only considering the years 1850 to 1997 and correlating CO₂ residuals with residuals of temperature 10 years later. The correlation between these two sets of data is remarkable. Let's start with a bar graph of both sets of residuals, Figure 2.

Figure 2 is a good graph to get a subjective sense of the correlation. Let's see if the math confirms this. Figure 3 is the scatter graph of the residuals.

The slope of a linear regression of the scatter is 2.6x10^-5, and it is statistically significant (95% confidence interval 1.88x10^-5 to 3.33x10^-5). Even the 99.99999999% confidence interval is entirely positive. Unless anthropogenic global warming is a reality, there is no apparent reason why the residuals of cumulative human CO₂ emissions should correlate so well with the residuals of temperature 10 years later throughout the last 150 years.

The slope of the scatter is actually more steep than expected, if you consider the naive correlation between cumulative CO₂ emissions and temperature. There are probably several reasons for this. The one I believe to be the most likely is that over time CO₂ does get removed from the atmosphere. Adding this consideration to the analysis should produce a more accurate slope. The other potential reasons don't bode so well for our species.

[Update 2/22/2010: I have written a follow-up titled Statistical Proof of Anthropogenic Global Warming v2.0.]

Hello World

2008-06-28T11:33:00.001-07:00

I will use this blog to write about topics unrelated to autism. This was prompted by a post on Global Warming that I wrote on my primary blog, Natural Variation. I have at least a few more such posts planned.