Terry Fields wrote:
stuart noble wrote:

Thanks. Yes, I really must get into Excel charts, but I have an aversion
to pie charts and graphs.
What puts me off all this monitoring is the extent to which I know I can
influence the readings by slightly varying my everyday physical
activities in the previous couple of hours. Simple things like taking a
shower etc.

You might like to see my post of a few minutes ago to Roger, where I
illustrate several ways of manipulation of my BP data. For your
interest I've included a BP vs Time of Day graph.

Yes, thanks.
I make sure I see the doc first thing in the morning when my readings
are usually good. That way "we" don't have a problem

The message
from Terry Fields contains these words:

You might be interested in the following exercise, where I took my BP
data and treated it to a) a 10-point smoothing exercise, and b) a
linear regression, which had a lower variance than a second-order
polynomial (not shown)(all data manipulation courtesy of Excel
spreadsheet).

Snip all detail

Terry, I don't have Excel so don't have any familiarity with the
underlying mechanics of Excels smoothing filters (which in essence what
that regression is). The Met Office filter, at least when it has the
full range of numbers, has the advantage that it weights the closest
data higher than the more distant data while introducing no variation in
the overall total. Where it seems to go wrong is the fiddle used to
extend the smoothed curve up to the end of the data. I could be wrong
but it doesn't appear that Excel does anything to correct any possible
bias that is introduced by a short term effect at the end of the
sequence but almost certsainly there is some form of weighting in its
polynomial regression. It wouldn't work with a non linear data
otherwise.

You can place too much reliance on manipulated data. For instance your
straight line, the 'linear regression' is almost certainly an artifact.

I can't be sure but I suspect that your moving average is the simple one
with no weighting.

As you have Excel it would be relatively simple for you to take the Met
Office figures and see how polynomial regression differs from the MO
filter both during the period where the filter can work on all 21 years
and in the final 10 where the end bias gets stronger and stronger.

--
Roger Chapman

Roger wrote:

As you have Excel it would be relatively simple for you to take the Met
Office figures and see how polynomial regression differs from the MO
filter both during the period where the filter can work on all 21 years
and in the final 10 where the end bias gets stronger and stronger.

I don't like Excel much, but have to use it for the sake of
compatibility - I work from home on contract, and the stuff I produce
has to be fully compliant with with other systems. A spin-off of this
is that I don't know how the smoothing works, as you say it's probably
a simple moving-avarage. I only included that ghraph for interest's
sake.

I'd like to try playing with the MetO data, but two things aren't
working in my favour: a brief search of the MetO site didn't turn up
the figures - have you come across them. by any chance?; and I'm
supposed to be putting together a presentation for a formal meeting
next week, but more interesting things keep turning up :-/

Terry Fields wrote:
The Natural Philosopher wrote:

Terry Fields wrote:
The Natural Philosopher wrote:

Terry Fields wrote:
Roger wrote:

I am tempted to write to the Met Office about the bizarre way they
terminate their smoothing filter. Doubt if that will even provoke a
response let alone nudge them onto something a bit less extreme.
I'm sure you'll get a reply, but the chances are it will be some sort
of fob-off.

But why have a smoothing filter at all? Why not perform some sort of
regression or other statistical analysis on the data set?
What is a smoothing tool if not an example of that?
It's the difference bewteen 'smoothing' the data, which doesn't seem
to do anything but tidy up the appearance, and statistics that tell
you everything about the data - IOW, chalk and cheese.

Oh dear oh dear.

I see..

You're always at liberty to publish data that has been smoothed, and
compare that with the same data that has been statistically analysed,
to demonstrate how much better smoothing is for determining the
underlying trends.

You miss the point. Smoothing is what statistics do. Its just another
compression algorithm.

You are comparing apples and apples and calling them oranges.

The innate characteristic of any compression algorithm is that it
innately implies an underlaying pattern in the data, and seeks to find
that pattern and suppress the noise.

What you are essentially doing when analysing, smoothing, or whatever a
time series, is applying a low pass filter to it: What the filter has as
output is critically dependent on the form of the filter. You are merely
comparing two different forms of filter..and if they give different
results, that merely calls into question the validity of using ANY of them.

The message
from Terry Fields contains these words:

I'd like to try playing with the MetO data, but two things aren't
working in my favour: a brief search of the MetO site didn't turn up
the figures - have you come across them. by any chance?;

Not as yet. As it is fine atm I am spending very little time in the
house. If the worst comes to the worst I could always extract the
figures from the graph but they would be rather imprecise.

--
Roger Chapman

Roger wrote:

The message
from Terry Fields contains these words:

I'd like to try playing with the MetO data, but two things aren't
working in my favour: a brief search of the MetO site didn't turn up
the figures - have you come across them. by any chance?;

Not as yet. As it is fine atm I am spending very little time in the
house. If the worst comes to the worst I could always extract the
figures from the graph but they would be rather imprecise.

I can think of a way that might lend some precision, but it'll be
tedious: bring up the graph in say Paint Shop Pro, hover the cursor
over each point in turn, and use the pixel-corordinate readout to give
each point a value in X,Y format. Calibrate by doing the same for each
axis. Might get the points to a precision of 1 percent - which could
be enough for our purposes.

The Natural Philosopher wrote:

What you are essentially doing when analysing, smoothing, or whatever a
time series, is applying a low pass filter to it: What the filter has as
output is critically dependent on the form of the filter. You are merely
comparing two different forms of filter..and if they give different
results, that merely calls into question the validity of using ANY of them.

I know it's a been as while since I did any statistics in anger, but
from my first lecture in college in 1963 up to what's laughingly
called retirement, I've never heard of statistics being called a LPF.

Perhaps the picture in my mind, of a LPF rejecting HF components of
something, isn't what I see as being done by statistics, which is to
discard nothing in orderto return the best estimates of the provenance
of the data.

How would a comparison of say, satellite data analysed by two
different organisations, with ditto from ground-based data, by putting
them through an analysis of variance, be classed as low-pass
filtering?

Apologies if I appear dim, but I'm struggling to visualise the LPF
concept.

I hope it isn't time for round two already but The Independent front
page today has the latest twist on the GW saga - methane emissions in
the Arctic.

--
Roger Chapman

Terry Fields wrote:
The Natural Philosopher wrote:

What you are essentially doing when analysing, smoothing, or whatever a
time series, is applying a low pass filter to it: What the filter has as
output is critically dependent on the form of the filter. You are merely
comparing two different forms of filter..and if they give different
results, that merely calls into question the validity of using ANY of them.

I know it's a been as while since I did any statistics in anger, but
from my first lecture in college in 1963 up to what's laughingly
called retirement, I've never heard of statistics being called a LPF.


Note that I did say with respect to a *time series*.

Indeed you probabably would not, but let me assure you that the digital
implementation of a low pass filter is a statistically weighted moving
average of a sort.

Juts like e.g. a 'moving average'.

A resistor and a capacitor does a beautiful moving average as it happens ;-)
More resistors and capacitors change its characteristics.

Anyway, as ypu know my background is in analog and digital electronics,
and thats a unique viewpoint that allows me to see the things that you
regard - possibly because the methodology you use appears oo different,
to be different things: They are not. They are data compressors. Of
which a low pass filter - or indeed any filter - is a classic example,
but done 'analogue'




Perhaps the picture in my mind, of a LPF rejecting HF components of
something, isn't what I see as being done by statistics, which is to
discard nothing in orderto return the best estimates of the provenance
of the data.

An average is a complete and most basic reduction of a data set to ONE
data point.

Low pass filters discard nothing: its simply that the shorter the
duration of a section of the raw graph is, the less it affects the final
output.



How would a comparison of say, satellite data analysed by two
different organisations, with ditto from ground-based data, by putting
them through an analysis of variance, be classed as low-pass
filtering?

Sounds a perfect description to me.A differential amplifier and low pass
filter..


Apologies if I appear dim, but I'm struggling to visualise the LPF
concept.

That, I am afraid, is your problem.

Broadly speaking, any device which allows the time variance of a signal
to affect the amplitude of the output as well as the amplitude of that
signal does, is some kind of filter.

The moment you derive an output at a given time, not just from the input
data at that time, but from other times as well, you are applying some
sort of frequency style filter in practice, even if you think you are
doing statistics.

Roger wrote:
I hope it isn't time for round two already but The Independent front
page today has the latest twist on the GW saga - methane emissions in
the Arctic.

Yup. Not sure what the breakdown mechanism of methane is..guess it sort
of oxidises in due course.

Its not particularly soluble is it?

Perhaps its the answer to 'renewable' aircraft fuel. Scoop it up and
burn it ;-)

Or catalyse it with CO2 to make ethanol..

Roger wrote:

I hope it isn't time for round two already but The Independent front
page today has the latest twist on the GW saga - methane emissions in
the Arctic.

ROFL.

If they're right, the release is just in time to counter the Jan 07 -
Jan 08 cooling ;-)

"Terry Fields" wrote in message
news

Roger wrote:

As you have Excel it would be relatively simple for you to take the Met
Office figures and see how polynomial regression differs from the MO
filter both during the period where the filter can work on all 21 years
and in the final 10 where the end bias gets stronger and stronger.

I don't like Excel much, but have to use it for the sake of
compatibility - I work from home on contract, and the stuff I produce
has to be fully compliant with with other systems. A spin-off of this
is that I don't know how the smoothing works, as you say it's probably
a simple moving-avarage. I only included that ghraph for interest's
sake.

I'd like to try playing with the MetO data, but two things aren't
working in my favour: a brief search of the MetO site didn't turn up
the figures - have you come across them. by any chance?; and I'm
supposed to be putting together a presentation for a formal meeting
next week, but more interesting things keep turning up :-/


http://www.r-project.org/ if you want to play.

"Terry Fields" wrote in message
news

Roger wrote:

I hope it isn't time for round two already but The Independent front
page today has the latest twist on the GW saga - methane emissions in
the Arctic.

ROFL.

If they're right, the release is just in time to counter the Jan 07 -
Jan 08 cooling ;-)

I would like to see the evidence that the seabed have warmed at all.
It sounds like another scare story just to ask for more funding.
They need money to do endless surveys, and change the fudge factors in their
models.

The message
from The Natural Philosopher contains these words:

I hope it isn't time for round two already but The Independent front
page today has the latest twist on the GW saga - methane emissions in
the Arctic.

Yup. Not sure what the breakdown mechanism of methane is..guess it sort
of oxidises in due course.

IIRC it is not particularly long lived in the atmosphere, a few years at
the most

Its not particularly soluble is it?

Not AFAIK but I have no detailed knowledge of the subject.

Perhaps its the answer to 'renewable' aircraft fuel. Scoop it up and
burn it ;-)

Or catalyse it with CO2 to make ethanol..

Either case would be extraordinarily difficult given the widespread
nature of the release.

--
Roger Chapman

The message
from Terry Fields contains these words:

Not as yet. As it is fine atm I am spending very little time in the
house. If the worst comes to the worst I could always extract the
figures from the graph but they would be rather imprecise.

I can think of a way that might lend some precision, but it'll be
tedious: bring up the graph in say Paint Shop Pro, hover the cursor
over each point in turn, and use the pixel-corordinate readout to give
each point a value in X,Y format. Calibrate by doing the same for each
axis. Might get the points to a precision of 1 percent - which could
be enough for our purposes.

I don't have PSP. I do however have an old CAD package (Turbocad v6.5)
but in the past I have had problems importing objects into that.
Alternatively I could pretend it was a scanned map and take each years
co-ordinates but given the small size of the original graph | wouldn't
be at all hopeful of 99% accuracy except by sheer chance.

--
Roger Chapman

Roger wrote:

The message
from Terry Fields contains these words:

I'd like to try playing with the MetO data, but two things aren't
working in my favour: a brief search of the MetO site didn't turn up
the figures - have you come across them. by any chance?;

Not as yet. As it is fine atm I am spending very little time in the
house. If the worst comes to the worst I could always extract the
figures from the graph but they would be rather imprecise.

I'm looking through various blogs to see if anyone has posted a link
to the Hadley data. Found plenty of mentions, but no link.

I did find this, however, which extends the data to July 08:

http://reallyrealclimate.blogspot.co...e-anomoly.html

It shows three data sets - UAH, HadCrut, RSS - and the author has
added linear regressions.

It's interesting to note that the Hadley data is out of line with the
other two, from 2007 onward.

....and there's more data reduction on he

http://reallyrealclimate.blogspot.co...max-results=11

than you can shake a stick at.

All for interest only!

The message
from Terry Fields contains these words:

I'm looking through various blogs to see if anyone has posted a link
to the Hadley data. Found plenty of mentions, but no link.

I did find this, however, which extends the data to July 08:

http://reallyrealclimate.blogspot.co...e-anomoly.html

It shows three data sets - UAH, HadCrut, RSS - and the author has
added linear regressions.

Those straight lines are artifacts with a slope heavily influenced by
the 1998 maximum.

It's interesting to note that the Hadley data is out of line with the

other two, from 2007 onward.

Doesn't really get out of kilter until the start of 2008 and Hadley has
previous form where the other lines drop steeply.

....and there's more data reduction on he

http://reallyrealclimate.blogspot.co...max-results=11

than you can shake a stick at.

All for interest only!

Plenty of graphs starting with the 1998 peak and another batch starting
at 2002 after the post 1998 low. I didn't see a single graph covering
the period 1999 -2007 (or even 2008) which would have given a linear
regression with a positive slope.

--
Roger Chapman

The message
from Terry Fields contains these words:

I'd like to try playing with the MetO data, but two things aren't
working in my favour: a brief search of the MetO site didn't turn up
the figures - have you come across them. by any chance?

http://hadobs.metoffice.com/hadcrut3...download.html?

If you can find out what it all means. :-)

--
Roger Chapman

Roger wrote:

The message
from Terry Fields contains these words:

I'd like to try playing with the MetO data, but two things aren't
working in my favour: a brief search of the MetO site didn't turn up
the figures - have you come across them. by any chance?

http://hadobs.metoffice.com/hadcrut3...download.html?

If you can find out what it all means. :-)

I've worked out that 1.0000000E+30 means 'data missing'....

.....but not a lot else :-(

The best hope might be to find the actual data needed on a blog, but
that seems to be just as obscure....

The message
from Terry Fields contains these words:

http://hadobs.metoffice.com/hadcrut3...download.html?

If you can find out what it all means. :-)

I've worked out that 1.0000000E+30 means 'data missing'....

.....but not a lot else :-(

The best hope might be to find the actual data needed on a blog, but
that seems to be just as obscure....

I am some way into extracting approximate figures from the graph. Do you
have a valid email address I can send it to?

--
Roger Chapman

Roger wrote:

The message
from Terry Fields contains these words:

http://hadobs.metoffice.com/hadcrut3...download.html?

If you can find out what it all means. :-)

I've worked out that 1.0000000E+30 means 'data missing'....

.....but not a lot else :-(

The best hope might be to find the actual data needed on a blog, but
that seems to be just as obscure....

I am some way into extracting approximate figures from the graph. Do you
have a valid email address I can send it to?

It might be easier to ftp the figures as a txt file to your webspace,
that way they'd be available for others too. But even a as table of
figures they wouldn't take much bandwidth, so you could post them on
here - or on a test or misc group if people were unhappy with that.

The message
from Terry Fields contains these words:

I am some way into extracting approximate figures from the graph. Do you
have a valid email address I can send it to?

It might be easier to ftp the figures as a txt file to your webspace,
that way they'd be available for others too. But even a as table of
figures they wouldn't take much bandwidth, so you could post them on
here - or on a test or misc group if people were unhappy with that.

I will cut and post the figures here then, not having any active
webspace nor any experience of ftp.

As the figures will be in a ms works spreadsheet I thought it would make
things easier for you as excel should be able to read such files.

I might manage to post as early as tomorrow. OTOH I might not. :-)

--
Roger Chapman

Roger wrote:

The message
from Terry Fields contains these words:

I am some way into extracting approximate figures from the graph. Do you
have a valid email address I can send it to?

It might be easier to ftp the figures as a txt file to your webspace,
that way they'd be available for others too. But even a as table of
figures they wouldn't take much bandwidth, so you could post them on
here - or on a test or misc group if people were unhappy with that.

I will cut and post the figures here then, not having any active
webspace nor any experience of ftp.

As the figures will be in a ms works spreadsheet I thought it would make
things easier for you as excel should be able to read such files.

I might manage to post as early as tomorrow. OTOH I might not. :-)

Hmmm...that might count as binary file, which is verboten in text
groups :-( and it would mean that others wouldn't see the data. I'm
limited by experience and only having Excel to work with - perhaps
someone else with an interest could do something more sophisticated
that compare polynomials...

Terry Fields wrote:
Roger wrote:

The message
from Terry Fields contains these words:
I am some way into extracting approximate figures from the graph. Do you
have a valid email address I can send it to?
It might be easier to ftp the figures as a txt file to your webspace,
that way they'd be available for others too. But even a as table of
figures they wouldn't take much bandwidth, so you could post them on
here - or on a test or misc group if people were unhappy with that.
I will cut and post the figures here then, not having any active
webspace nor any experience of ftp.

As the figures will be in a ms works spreadsheet I thought it would make
things easier for you as excel should be able to read such files.

I might manage to post as early as tomorrow. OTOH I might not. :-)

Hmmm...that might count as binary file, which is verboten in text
groups :-( and it would mean that others wouldn't see the data. I'm
limited by experience and only having Excel to work with - perhaps
someone else with an interest could do something more sophisticated
that compare polynomials...

Comma delimited is OK.

endymion wrote:

Despite all of this my home ( 2 bed bungalow) comes out at above the
national average even though my appliances are using less than half the
national average according to them.

Bungalows lose more heat per m² than other comparable houses; they have
a lot of exposed areas.
--
Hugo Nebula
"If no-one on the internet wants a piece of this,
just how far from the pack have you strayed"?

The message
from Roger contains these words:

I might manage to post as early as tomorrow. OTOH I might not. :-)

Or even today. Figures below are for the unsmoothed annual anomaly from
1850 to 2007. I also have the figures for the smoothed curve if anyone
is interested. I used them and the smoothing filter to check for
extraction errors. I think most of the figures are accurate to 0.01 but
I suspect there may be some minor distortion in the graph which might
make matters worse.

The smoothed curve and my smoothing exercise have a reasonable fit with
only 5 years differing by 0.01 or more. (1850 - 0.023,1851 - 0.14, 1955
& 1956 - 0.01 opposite signs), 2007 - 0.014).

Over to you Terry.

-0.12
-0.03
-0.07
-0.09
-0.04
-0.01
-0.16
-0.21
-0.25
-0.14
-0.1
-0.29
-0.34
-0.11
-0.16
-0.01
0
-0.05
0.13
0.02
0.03
0.02
0.02
-0.01
-0.13
-0.05
-0.04
0.29
0.31
0.06
0.07
0.05
0.07
0.04
-0.04
-0.06
0.02
-0.05
0.03
0.13
-0.1
-0.05
-0.1
-0.17
-0.11
-0.02
0.14
0.1
-0.1
0.05
0.14
0.03
-0.1
-0.19
-0.22
-0.03
0
-0.17
-0.25
-0.28
-0.27
-0.28
-0.2
-0.18
-0.03
0.06
-0.18
-0.25
-0.07
-0.1
-0.04
0
-0.09
-0.05
-0.08
0.01
0.1
0.03
0.04
-0.09
0.12
0.14
0.11
-0.07
0.08
0.08
0.12
0.24
0.25
0.29
0.3
0.35
0.23
0.25
0.37
0.25
0.06
0.08
0.04
0.05
-0.05
0.11
0.19
0.23
0.06
-0.01
-0.04
0.2
0.26
0.17
0.14
0.24
0.25
0.24
-0.03
0.06
0.13
0.14
0.15
0.27
0.21
0.11
0.23
0.34
0.06
0.14
0.05
0.34
0.24
0.34
0.38
0.41
0.29
0.47
0.27
0.26
0.31
0.45
0.45
0.38
0.53
0.48
0.34
0.38
0.45
0.56
0.41
0.64
0.8
0.54
0.52
0.68
0.74
0.74
0.72
0.76
0.71
0.69

--
Roger Chapman

Roger wrote:

The message
from Roger contains these words:

I might manage to post as early as tomorrow. OTOH I might not. :-)

Or even today. Figures below are for the unsmoothed annual anomaly from
1850 to 2007. I also have the figures for the smoothed curve if anyone
is interested. I used them and the smoothing filter to check for
extraction errors. I think most of the figures are accurate to 0.01 but
I suspect there may be some minor distortion in the graph which might
make matters worse.

The smoothed curve and my smoothing exercise have a reasonable fit with
only 5 years differing by 0.01 or more. (1850 - 0.023,1851 - 0.14, 1955
& 1956 - 0.01 opposite signs), 2007 - 0.014).

Over to you Terry.

A labour of love....

Thank goodness Excel allows one to fill in a data series; I'd have
hated to have typed in all the years from 1850 - 2007 :-(

I've carried out the various analyses that Excel allows: moving
average of chosen span; polynomials of various orders; linear
regression; and logarithmic.

The last two produced silly trendlines and weren't proceded with.

Polynomials of second and third order produced interesting trendlines.

Moving averages of 3 and 10 points were carried out.

Results he

10 point moving average:

http://farm4.static.flickr.com/3179/...4007573e_o.jpg

5 point moving average:

http://farm4.static.flickr.com/3005/...1acc803a_o.jpg

Second-order polynomial:

http://farm4.static.flickr.com/3157/...5f46c52f_o.jpg

Third-order polynomial:

http://farm4.static.flickr.com/3258/...0b5832f2_o.jpg

Points to note:

- only the 5-point moving average picks up a downturn circa 2004 or
so

- the 3rd order polynomial has a slightly higher correlation
coefficient than the 2nd order. The data at the 1850 end of the scale
suggests recover from the Little Ice Age....

Discuss....;-)

[if anyone can do more sophisticated analyses than allowed by Excel,
I'm sure the results would be of interest]

The message
from Terry Fields contains these words:

Points to note:

- only the 5-point moving average picks up a downturn circa 2004 or
so

- the 3rd order polynomial has a slightly higher correlation
coefficient than the 2nd order. The data at the 1850 end of the scale
suggests recover from the Little Ice Age....

Discuss....;-)

I think you have discovered why the Met Office have chosen a 21 point
binomial filter to smooth the curve. The more 'sophisticated' filters
ignore short term trends.

I would be interested to see how the polynomial filters would cope with
a subsequent downturn so how about inverting the 1950 - 2004 about 2005
(2006 = 2004 .... 2060 = 1950) and then doing several finish dates to
see when the downtown can first be clearly seen as well as what a mature
downturn would look like.

--
Roger Chapman

Roger wrote:

I think you have discovered why the Met Office have chosen a 21 point
binomial filter to smooth the curve. The more 'sophisticated' filters
ignore short term trends.

Interesting, but see below.

I would be interested to see how the polynomial filters would cope with
a subsequent downturn so how about inverting the 1950 - 2004 about 2005
(2006 = 2004 .... 2060 = 1950) and then doing several finish dates to
see when the downtown can first be clearly seen as well as what a mature
downturn would look like.

Now done that; results he

Second-order polynomial:

http://farm4.static.flickr.com/3070/...e4e85748_o.jpg

Third order:

http://farm4.static.flickr.com/3052/...dbf159c8_o.jpg

5-point moving average:

http://farm4.static.flickr.com/3041/...01d63059_o.jpg

10-point moving average:

http://farm4.static.flickr.com/3031/...072cc3b6_o.jpg

20-point moving average:

http://farm4.static.flickr.com/3109/...846c5b23_o.jpg


Several points spring to mind he

- the polynomials are, well, rubbish

- the 10- and 20-point moving averages shift everything to the
right

- the 5-point moving average picks up the downturnclearly afetr 2 to
3 years, as might be expected.

The thing that appears to come out of this exercise is that to pick up
trends at the earliest opportunity, long moving averages are not the
tool to use. Hadley seems to use a (weighted) 20-pointer; I'll bet
they've done every kind of data run, in a more sophisticated manner
then we have, and have a good idea of the current position.

One of the bloggers said that the 'downturn' data doesn't appear on
the Hadley's 'Myth' page....I can't help wondering why. Perhaps, like
Micawber, they're waiting for something to turn up (or down).

The message
from Terry Fields contains these words:

Several points spring to mind he

- the polynomials are, well, rubbish

Well they are no use for this sort of application anyway.

- the 10- and 20-point moving averages shift everything to the
right

- the 5-point moving average picks up the downturnclearly afetr 2 to
3 years, as might be expected.

The thing that appears to come out of this exercise is that to pick up
trends at the earliest opportunity, long moving averages are not the
tool to use. Hadley seems to use a (weighted) 20-pointer; I'll bet
they've done every kind of data run, in a more sophisticated manner
then we have, and have a good idea of the current position.

The Met Office filter is 21 points and seems to do the job very well
except for the limits of the sequence and even there I am coming to the
view that it is difficult to specify something else that lessens the
weighting effect of the final year without introducing another factor
potentially at least as bad.

It is generally accepted that El Nino and La Nina episodes have a
significant warming or cooling effect respectively so the Met Office may
well have a graph somewhere that attempts to compensate for those
factors.

One of the bloggers said that the 'downturn' data doesn't appear on
the Hadley's 'Myth' page....I can't help wondering why. Perhaps, like
Micawber, they're waiting for something to turn up (or down).

Those who see a definite trend already are cherry picking their data.
Given a few more years the downturn will become evident if it is
anything other than a transient event. The figures below are for the
smoothed curve 1850 - 2060 using the inverted around 2005 which shows
the downturn clearly and centred on 2005. (I hope no one picks these
figures up and takes the forward projection as gospel.)

-0.09
-0.08
-0.08
-0.08
-0.09
-0.11
-0.13
-0.15
-0.17
-0.19
-0.19
-0.19
-0.18
-0.15
-0.12
-0.07
-0.04
-0.01
0.01
0.02
0.02
0.01
-0.00
-0.01
0.00
0.03
0.06
0.09
0.11
0.11
0.09
0.07
0.05
0.02
0.01
-0.00
-0.00
-0.00
-0.01
-0.02
-0.03
-0.05
-0.06
-0.06
-0.05
-0.02
0.00
0.02
0.03
0.02
0.00
-0.03
-0.06
-0.09
-0.11
-0.12
-0.14
-0.17
-0.19
-0.21
-0.22
-0.21
-0.18
-0.15
-0.13
-0.12
-0.11
-0.11
-0.10
-0.09
-0.07
-0.06
-0.05
-0.04
-0.02
-0.00
0.01
0.02
0.03
0.04
0.05
0.06
0.07
0.07
0.09
0.12
0.15
0.19
0.23
0.26
0.27
0.28
0.27
0.26
0.23
0.19
0.15
0.11
0.09
0.07
0.08
0.09
0.10
0.10
0.10
0.10
0.11
0.13
0.15
0.17
0.18
0.18
0.17
0.15
0.14
0.13
0.13
0.15
0.16
0.18
0.19
0.19
0.19
0.19
0.18
0.19
0.21
0.24
0.27
0.31
0.33
0.35
0.35
0.35
0.35
0.35
0.37
0.39
0.41
0.42
0.43
0.44
0.44
0.45
0.47
0.50
0.54
0.57
0.60
0.62
0.64
0.66
0.69
0.71
0.72
0.72
0.72
0.71
0.69
0.66
0.64
0.62
0.60
0.57
0.54
0.50
0.47
0.45
0.44
0.44
0.43
0.42
0.41
0.39
0.37
0.35
0.35
0.35
0.35
0.35
0.33
0.31
0.27
0.24
0.21
0.19
0.18
0.19
0.19
0.19
0.19
0.18
0.16
0.15
0.13
0.13
0.14
0.15
0.17
0.18
0.18
0.17
0.15
0.13
0.11
0.10
0.10
0.10
0.08
0.06
0.03

--
Roger Chapman

Roger wrote:

Those who see a definite trend already are cherry picking their data.
Given a few more years the downturn will become evident if it is
anything other than a transient event. The figures below are for the
smoothed curve 1850 - 2060 using the inverted around 2005 which shows
the downturn clearly and centred on 2005. (I hope no one picks these
figures up and takes the forward projection as gospel.)

What smoothing technique did you use?

This is how your data looks in graphical form he

http://farm4.static.flickr.com/3256/...bbd292db_o.jpg

Terry Fields wrote:
Roger wrote:

I think you have discovered why the Met Office have chosen a 21 point
binomial filter to smooth the curve. The more 'sophisticated' filters
ignore short term trends.

Interesting, but see below.

I would be interested to see how the polynomial filters would cope with
a subsequent downturn so how about inverting the 1950 - 2004 about 2005
(2006 = 2004 .... 2060 = 1950) and then doing several finish dates to
see when the downtown can first be clearly seen as well as what a mature
downturn would look like.

Now done that; results he

Second-order polynomial:

http://farm4.static.flickr.com/3070/...e4e85748_o.jpg

Third order:

http://farm4.static.flickr.com/3052/...dbf159c8_o.jpg

5-point moving average:

http://farm4.static.flickr.com/3041/...01d63059_o.jpg

10-point moving average:

http://farm4.static.flickr.com/3031/...072cc3b6_o.jpg

20-point moving average:

http://farm4.static.flickr.com/3109/...846c5b23_o.jpg


Several points spring to mind he

- the polynomials are, well, rubbish

- the 10- and 20-point moving averages shift everything to the
right


Of course. Phase delay..

- the 5-point moving average picks up the downturnclearly afetr 2 to
3 years, as might be expected.

I am a bit curuous as to where the data from the years 2010-2050 comes
from...
The thing that appears to come out of this exercise is that to pick up
trends at the earliest opportunity, long moving averages are not the
tool to use.

The standard advice when using these to e.g. track stock prices is to
buy/sell when the data crosses the average. This is margnally better
than tossing dice or gazing into a crystakl ball.

Hadley seems to use a (weighted) 20-pointer; I'll bet
they've done every kind of data run, in a more sophisticated manner
then we have, and have a good idea of the current position.

One of the bloggers said that the 'downturn' data doesn't appear on
the Hadley's 'Myth' page....I can't help wondering why. Perhaps, like
Micawber, they're waiting for something to turn up (or down).

The Natural Philosopher wrote:

I am a bit curuous as to where the data from the years 2010-2050 comes
from...

Roger suggested inverting the data from 1950 - 2005 and adding it on
from 2006, to give a mirror image of the datatset based about 2005.
The idea was to try to identify as soon as possible where any downturn
could be confirmed. The second tranche of graphs was that exercise.

"The Natural Philosopher" wrote in message
...
Terry Fields wrote:
Roger wrote:

snip

I am a bit curuous as to where the data from the years 2010-2050 comes
from...

The data seem to be an exact mirror image of the preceding years? Simple
enough to do but meaningless?


--
Bob Mannix
(anti-spam is as easy as 1-2-3 - not)

Terry Fields wrote:
The Natural Philosopher wrote:

I am a bit curuous as to where the data from the years 2010-2050 comes
from...

Roger suggested inverting the data from 1950 - 2005 and adding it on
from 2006, to give a mirror image of the datatset based about 2005.
The idea was to try to identify as soon as possible where any downturn
could be confirmed. The second tranche of graphs was that exercise.

Oh, I see. Using pictures rather than maths to understand the filter
characteristics.

Well you have completely reinvented the wheel, and discovered that
predicting the future needs a model. That can only be proven to be
correct (or not) once the future has happened.

The triumph of science is to pick models other than simple polynomials,
sine waves and the like, that reflect what seems to be the underlying
*mechanisms*. And in this case thats a definite plural.

Which is what climate change science is all about.

Just peering at the data and applying random simplistic filters wont
actually get you very far at all.

As I said, if what you are looking for is sine waves, what you will get
is sine waves.

I don't have to do all this: I did it years ago when working on software
for a digital sampling and storage oscilloscope: when we got up towards
the sampling frequency, we had various filters we could apply: the sine
interpolation filter attempted to fit a sine wave to the data points. It
always managed to do precisely that, irrespective of how sinusoidal the
original signal might have been, whilst a moving average more or less
wiped out te data at thse frequencies completely. It was better, in that
it wasn't subject to such wild extrapolations, but it was worse, in that
the bandwidth was totally lost.

In short, you cant do prediction with these sorts of brute force curve fits.

In terms of the actual climate change models so far proposed, there are
two very very conflicting mechanisms at leats: Pollution from CO2 and
methane, acting to raise temperatures, and pollutin from particulate
emissions like carbon soot from diesel and coal, acting in reverse.

Leaving aside amplification, this would tend to make periods of rapid
carbon based economic expansion, initially cooling in effect, as the
greater amount of atmospheric soots is a net cooler: once into
recession, these will wash out, leaving the longer term gaseous
pollutants well able to increase temperatures rapidly. So watch out for
a sharp upswing as the global recession bites deeper.

With current La Nina type conditions likely to come to an end, that
should be a triple whammy.

The message
from Terry Fields contains these words:

What smoothing technique did you use?

The Met Office's 21 point binominal filter except that in ms works case
the total weighting only adds up to 0.99999893 and trying to encapsulate
the whole function in one cell gave the wrong answers.


This is how your data looks in graphical form he

Thanks

--
Roger Chapman

The message
from "Bob Mannix" contains these words:

I am a bit curuous as to where the data from the years 2010-2050 comes
from...

The data seem to be an exact mirror image of the preceding years? Simple
enough to do but meaningless?

Done to see how Excels polynomials coped with a sharp change in
direction - they don't.

--
Roger Chapman