Meta tags:

Headings (most frequently used words):

data, to, step, posts, analytics, finding, raw, temperature, monthly, gridded, in, easy, use, format, style, vs, content, by, is, the, climate, changing, nothing, see, here, move, along, categories, recent, top,

Text of the page (most frequently used words):
the (82), data (55), for (40), lat (34), txt (31), stations (27), with (25), lon (23), and (22), temperature (21), awk (19), year (17), printf (17), from (16), print (15), ftp (15), day (15), are (14), 1900 (13), ghcn (13), daily (13), this (12), raw (12), was (12), ncdc (12), 9999 (12), #monthly (11), anomalies (11), posted (11), coords (11), gov (11), substr (11), step (10), sector (10), end (10), that (10), noaa (10), all (9), latitude (9), 240 (9), giss (9), pub (9), here (8), 2009 (8), tstations (8), 100 (8), easy (7), look (7), list (7), dly (7), like (6), comments (6), log (6), have (6), climate (6), uncategorized (6), sectors (6), tavg_1x1 (6), sort (6), temp (6), let (6), bad (6), temperatures (6), coordinates (6), 239 (6), 231 (6), tmax (6), tmn (6), name (5), content (5), arctic (5), use (5), code (5), analysis (5), genezeien (5), above (5), has (5), average (5), done (5), degree (5), 1961 (5), 1990 (5), adjusted (5), not (5), 1999 (5), next (5), out (5), which (5), find (5), ghcnd (5), filename (5), tmx (5), station (5), coord (5), date (5), wordpress (4), com (4), analytics (4), see (4), along (4), format (4), december (4), long (4), one (4), will (4), their (4), first (4), int (4), within (4), graphs (4), time (4), were (4), wget (4), aj000037749 (4), 7000 (4), files (4), 256 (4), 250 (4), 212 (4), 230 (4), echo (4), exit (4), then (4), tmin (4), length (4), 1932 (4), site (3), comment (3), nothing (3), move (3), trends (3), changing (3), effects (3), style (3), gridded (3), finding (3), posts (3), results (3), methods (3), unadjusted (3), range (3), effect (3), method (3), avg (3), trend (3), two (3), been (3), graph (3), else (3), years (3), clearly (3), take (3), days (3), how (3), does (3), available (3), state (3), base (3), keep (3), celsius (3), having (3), looks (3), starting (3), those (3), nsec (3), fairly (3), bin (3), bash (3), grid (3), stationid (3), missing (3), upon (3), you (3), tempstations (3), each (3), latlonids (3), daily_1x1 (3), split (3), latlon (3), info (3), month (3), january (3), http (3), get (2), website (2), required (2), sign (2), subscribed (2), subscribe (2), now (2), free (2), about (2), urban (2), heating (2), top (2), down (2), going (2), new (2), ends (2), 40n (2), 45n (2), nyc (2), zavg (2), nyt (2), nyn (2), area (2), versus (2), uhi (2), considered (2), read (2), latitudes (2), plots (2), pass (2), using (2), computed (2), mean (2), other (2), constant (2), chosen (2), easily (2), basesum (2), basen (2), continuous (2), used (2), averaging (2), just (2), century (2), 613 (2), source (2), follow (2), previous (2), annualt (2), wave (2), combined (2), annual (2), point (2), where (2), exec (2), uncompress (2), exclude (2), directories (2), pwd (2), start (2), codes (2), metadata (2), contents (2), elev (2), gsnflag (2), hcnflag (2), wmoid (2), geokcay (2), 37749 (2), min (2), max (2), temps (2), line (2), yearmo (2), day1 (2), day2 (2), day3 (2), aj000037749193602prcp (2), aj000037749200910tmax (2), 224 (2), 252 (2), 220 (2), 221 (2), 225 (2), 253 (2), 229 (2), 243 (2), 181 (2), 187 (2), 170 (2), 173 (2), build (2), grep (2), into (2), lines (2), continue (2), would (2), parsing (2), file (2), remove (2), minimum (2), avoid (2), regions (2), than (2), based (2), most (2), installed (2), since (2), experience (2), primarily (2), though (2), documentation (2), make (2), three (2), some (2), analyses (2), leave (2), steps (2), 180 (2), calculate (2), these (2), subsep (2), 100c (2), sites (2), quality (2), control (2), columns (2), longitude (2), 2010 (2), coop (2), started, design, email, write, loading, collapse, bar, manage, subscriptions, view, reader, report, privacy, already, account, create, blog, recent, categories, present, sources, apply, straight, forward, track, gather, odd, side, gridding, york, cntrl, prk, 7800, 9700, begin, 40n_latitude3, well, too, good, hide, greater, 35n, depth, https, justdata, surprises, there, none, entire, band, last, warming, found, perhaps, easier, eliminated, legend, rearranged, equatorial, near, x10, lats, tavg_bylat_globe, something, needs, reduce, variance, bands, compare, whereas, further, after, conversion, note, subtitle, meteorological, include, ocean, chaotic, variable, seen, relatively, quiet, plot, anomaly, differ, real, minus, surprise, left, offset, complete, compute, spreadsheet, continuous_anomaly_globe, established, basic, overview, what, demonstrate, transforming, across, globe, illustrates, wild, fluctuations, land, moving, reason, due, increase, number, prior, turn, represent, post, tomorrow, intuitively, oscillates, northern, seasons, processing, script, sift, measures, continuous_tavg_globe, correlation, sine, drop, order, minimize, appearing, dropping, selected, had, strong, sinusoidal, pattern, poorly, correlated, linear, probably, related, peaks, troughs, begins, idea, copy, pairs, check, populate, 180x360, anything, goes, box, cat, while, select, poll, value, best, pair, fun, output, should, tavg, divided, builtin, added, obs, tavg_globe, included, geographical, methodology, allows, relocation, overlap, minimal, impact, more, readings, completely, arbitrary, decision, reasoning, high, dropout, rates, inhospitable, home, sun, linux, unix, virtual, 40gb, disk, maintains, nice, hard, downloaded, decompressed, readme, repository, ubuntu, virtualbox, majority, researchers, work, because, cru, accessible, evidence, surfaced, suggests, entities, independent, adjustment, may, suspect, heard, frequently, underlying, current, consensus, robustly, supportive, decided, undertake, simple, quite, surprised, posting, suggestions, links, abstract, eugene, zeien, applied, physics, 1991, support, university, iowa, author, also, tobs, aka, observation, bias, discussed, feel, suggestion, coding, problems, contrary, typical, blogging, entry, grow, additional, completed, rather, appear, information, gives, context, later, visited, before, looking, latest, progress, scroll, self, explanatory, many, familiar, associative, arrays, natural, fit, unknown, extent, takes, delete, freshen, lists, rounded, nearest, integer, tempsta, dailies, comma, between, gsub, finished, nan, monthly_1x1, maximum, below, discarded, uses, key, values, occasionally, 999, sneak, reality, allowed, could, tighter, weed, any, irrational, only, worth, intent, discarding, subject, extreme, weather, events, observer, hardships, discard, whole, when, single, fill, surrounding, 1200km, away, believe, favors, observe, 34mb, column, comes, remaining, row, thereafter, separate, space, monthy_1x1, pdfs, original, paper, records, www7, ips, html, nasa, gistemp, station_data, prepared, interest, climexp, knmi, prefer, your, try, bit, mess, but, food,

Text of the page (random words):
data analytics data analytics about anomalies arctic trends urban heating effects posted by genezeien january 16 2010 finding raw temperature data ftp ftp ncdc noaa gov pub data ghcn daily a bit of a mess but so is raw food if you prefer your data in graphs try http climexp knmi nl go prepared with latitude longitude coordinates to find the temperature station of interest raw and adjusted data are available here http data giss nasa gov gistemp station_data these are adjusted temperatures http www7 ncdc noaa gov ips coop coop html pdfs of original paper records leave a comment posted in uncategorized posted by genezeien january 15 2010 monthly gridded data in easy to use format monthy_1x1 txt 34mb file first column is date i e year month 1 12 so january of 1932 comes out as 1932 00 and december of 1932 is 1932 92 remaining columns have latitude longitude in the first row and temperatures in celsius thereafter columns are separate by a space quality control if a 1 1 degree sector has only one day s worth of data that is the temperature for that month tmax tmin 2 the intent is to avoid discarding data from sites that are subject to extreme weather events or observer hardships giss quality control method is to discard a whole month for a site when a single day is missing then fill in from surrounding stations up to 1200km away i believe the giss method favors sites which are easy to observe daily minimum of maximum temperatures above 100c or below 100c were discarded ghcn uses 9999 as a key for missing values and occasionally a 999 would sneak in in reality the allowed range could be tighter to weed out any other irrational temperatures bin bash monthly data wget nr ftp ftp ncdc noaa gov pub data ghcn v2 o monthly log find ftp ncdc noaa gov pub data ghcn name z exec uncompress daily data wget nr exclude directories grid ftp ftp ncdc noaa gov pub data ghcn daily o daily log h pwd cd ftp ncdc noaa gov pub data ghcn daily start with list of stations coordinates codes ghcnd stations txt list of stations and their metadata e g coordinates contents look like id lat long elev state name gsnflag hcnflag wmoid aj000037749 40 7000 47 7000 95 0 geokcay 37749 look up min max temps from stationid dly files ex aj000037749 dly line 1 looks like yearmo day1 day2 day3 aj000037749193602prcp 0 i 17 i 0 i 0 i 0 i 0 i 0 i 78 i 0 i 0 i 0 i 0 i 0 i 20 i 0 i 0 i 0 i 0 i 0 i 0 i 0 i 12 i 0 i 0 i 0 i 0 i 0 i 0 i 0 i 9999 9999 aj000037749200910tmax 240 s 224 s 252 s 256 s 250 s 250 s 239 s 212 s 220 s 221 s 225 s 239 s 9999 253 s 256 s 231 s 239 s 240 s 230 s 231 s 229 s 230 s 231 s 9999 243 s 9999 181 s 187 s 212 s 170 s 173 s list of stations with temperature data this takes some time delete files to freshen the lists if e h tempstations txt then awk f filename 1 tmax f filename print f h tempstations txt all dly fi which stations are in each sector format lat lon stationid where lat is rounded to the nearest integer awk filename ghcn lat 1 int 2 90 5 lon 1 int 3 180 5 filename tempsta id substr 1 5 11 station id 1 end for id in station print lat id 90 lon id 180 id h tempstations txt ghcnd stations txt h latlonids txt calculate daily average for each sector calculate monthly from dailies for each sector echo n h daily_1x1 txt for coord in awk print 1 2 h latlonids txt sort u do remove the comma between lat lon c coord echo c list of stations with these coordinates tstations grep c h latlonids txt awk print all 3 dly awk v coord coord st substr 0 0 11 yr substr 0 12 4 mo substr 0 16 2 1 tmax l length 0 d 1 for i 22 i l i 8 mx substr 0 i 5 10 0 if mx 100 mx 100 tmx yr mo d mx nx yr mo d if d yr mo d d yr day yr mo d 1 d 1 tmin l length 0 d 1 for i 22 i l i 8 mn substr 0 i 5 10 0 if mn 100 mn 100 tmn yr mo d mn nn yr mo d d end split coord c for day in d if tmn day d day gsub subsep d print c 1 c 2 d tmx day nx day tmn day nn day 2 0 tstations sort n h daily_1x1 txt done echo finished daily starting monthly monthly by sector awk split 3 day if day 1 1900 next date day 1 day 2 1 12 0 t 1 2 date 4 tn 1 2 date days date 1 latlon 1 2 1 end printf year for ll in latlon split ll info subsep printf d d info 1 info 2 printf n for d in days printf 1 3f d for ll in latlon if tn ll d printf 1 4f t ll d tn ll d else printf nan printf n h daily_1x1 txt sort n 0 1 h monthly_1x1 txt exit 1 comment posted in uncategorized posted by genezeien december 29 2009 style vs content feel free to leave an analysis suggestion or point out coding problems in the comments area on step by step contrary to typical blogging the step by step entry will continue to grow as additional analysis steps are completed rather than having new posts appear above the information that gives context to the later analysis if you have visited before and are looking for the latest progress just scroll down to the end most graphs are self explanatory though many analyses build upon previous steps code is primarily bash awk if you are familiar with c awk code is fairly easy to read the associative arrays make awk a natural fit for parsing files with content of unknown extent see also uhi arctic next up tobs aka time of observation bias as discussed in ncdc documentation 2 comments posted in uncategorized posted by genezeien december 28 2009 step by step is the climate changing author eugene zeien bs applied physics 1991 18 years experience in data analysis it support at the university of iowa abstract having heard so frequently that the data underlying the current consensus was robustly supportive i decided to take the time to find raw unadjusted data and undertake some simple analyses i was quite surprised by the results i am posting those here for comments and suggestions along with source code and links to the raw data the majority of climate researchers use the adjusted data in their work because cru giss and ncdc make the adjusted data easily accessible and easy to use since evidence has surfaced which suggests those three entities are not independent all three adjustment methods may be suspect let s take a look methods starting with a home pc i installed sun s virtualbox next since my experience is primarily linux unix based i installed ubuntu 9 10 on a virtual 40gb disk ghcn maintains a nice though hard to find ftp repository of raw climate station data which was downloaded and decompressed the documentation in readme txt was fairly easy to follow as a first pass all the stations available in ghcn were included temperature data was combined into a bin from all the stations within a geographical 1 1 degree sector this methodology allows for station relocation and overlap with minimal impact upon the results an annual temperature was computed for a sector with more than 240 daily readings within that year 240 was a completely arbitrary decision based upon the reasoning that the stations with high dropout rates are in the most inhospitable regions bin bash monthly data wget nr ftp ftp ncdc noaa gov pub data ghcn v2 o monthly log find ftp ncdc noaa gov pub data ghcn name z exec uncompress daily data wget nr exclude directories grid ftp ftp ncdc noaa gov pub data ghcn daily o daily log h pwd start with list of stations coordinates codes ghcnd stations txt list of stations and their metadata e g coordinates contents look like id lat long elev state name gsnflag hcnflag wmoid aj000037749 40 7000 47 7000 95 0 geokcay 37749 look up min max temps from stationid dly files ex aj000037749 dly line 1 looks like yearmo day1 day2 day3 aj000037749193602prcp 0 i 17 i 0 i 0 i 0 i 0 i 0 i 78 i 0 i 0 i 0 i 0 i 0 i 20 i 0 i 0 i 0 i 0 i 0 i 0 i 0 i 12 i 0 i 0 i 0 i 0 i 0 i 0 i 0 i 9999 9999 aj000037749200910tmax 240 s 224 s 252 s 256 s 250 s 250 s 239 s 212 s 220 s 221 s 225 s 239 s 9999 253 s 256 s 231 s 239 s 240 s 230 s 231 s 229 s 230 s 231 s 9999 243 s 9999 181 s 187 s 212 s 170 s 173 s build list of lat lon pairs to check for data cd ftp ncdc noaa gov pub data ghcn daily awk printf d d n 2 3 ghcnd stations txt sort u grep h coords txt going to populate a 180x360 degree grid anything from x 0 x 99 goes into x s box echo n h tavg_1x1 txt let lines cat h coords txt wc l let c 0 while c lt lines do let c coords awk nr c print 0 exit h coords txt lat coords 0 lon coords 1 select stations with this lat lon stations awk v lat lat v lon lon lat 0 0 2 2 lat 1 0 lon 0 0 3 3 lon 1 0 print 1 ghcnd stations txt if stations gt 0 then printf s s s n lat lon stations else printf s s n lat lon fi poll stations for years with temperature data 9999 is missing value tstations for s in stations do tstations tstations awk 1 tmax print filename exit all s dly done if tstations eq 0 then let lon continue fi printf t s s s n lat lon tstations would be best to pair up tmin tmax parsing is fun output file should be lat lon year tavg divided by 10 0 to remove builtin t 10 added a minimum n 240 to avoid regions with one temperature obs year awk st substr 0 0 11 yr substr 0 12 4 mo substr 0 16 2 1 tmax l length 0 d 1 for i 22 i l i 8 mx substr 0 i 5 10 0 if mx 100 mx 100 tmx yr mx nx yr if d yr mo d d yr day yr mo d 1 d 1 tmin l length 0 d 1 for i 22 i l i 8 mn substr 0 i 5 10 0 if mn 100 mn 100 tmn yr mn nn yr d end for i in tmx if tmn i d i 240 print lat lon i tmx i nx i tmn i nn i 2 0 tstations sort n h tavg_1x1 txt done awk temp 3 4 n 3 end for i in temp print i temp i n i h tavg_1x1 txt h tavg_globe txt exit in order to minimize the effect of temperature stations appearing and dropping out sectors were selected which had continuous annual temperature data from 1900 to 2009 the data from the 613 sectors has a fairly strong sinusoidal pattern the poorly correlated linear trend is probably related to the point in the wave s peaks and troughs where the data begins and ends do not copy this graph i have an idea correlation 0 43 using a 30 year sine wave combined with a 0 3 century drop in temperature this is a follow up from the previous processing script sift out sector that are have measures for all of 1900 1999 awk lat 1 lon 2 year 3 coords lat lon 1 t lat lon year 4 end for c in coords bad 0 for y 1900 y 1999 y if t c y bad 1 if bad 0 for y 1900 y 2009 y keep c y t c y n c y print year celsius for y 1900 y 2009 y for c in coords if n c y 0 annualt y keep c y n c y nsec y print y annualt y nsec y y nsec y h tavg_1x1 txt h continuous_tavg_globe txt the reason 1900 was chosen is due to the increase in the number of stations just prior to the turn of the century the next two graphs do not represent the 613 sectors used above i will post source code tomorrow intuitively the monthly average temperature oscillates with the northern seasons having established a basic overview of what the raw data looks like now it is time to look at the temperature anomalies starting with the 1 1 degree sector data those with continuous temperatures 1900 1999 were used to demonstrate the effect of transforming and averaging raw temperatures across the globe versus averaging anomalies base 1961 1990 this graph illustrates the wild fluctuations of land temperature anomalies from one year to the next and the 11 year moving average get sectors with complete 1900 1999 data compute anomalies base 1961 1990 11 yr average is easily done in spreadsheet so that s not here awk lat 1 lon 2 year 3 coords lat lon 1 t lat lon year 4 end for c in coords bad 0 for y 1900 y 1999 y if t c y bad 1 if bad 0 for y 1900 y 2009 y keep c y t c y n c y for y 1961 y 1990 y basesum c t c y basen c base c basesum c basen c print year celsius for y 1900 y h continuous_anomaly_globe txt how does an anomaly differ from real temperature minus a constant surprise a constant was chosen that left the two plots offset by 0 1 c how does the temperature data go from the chaotic variable state seen above to the relatively quiet plot posted by giss note the subtitle meteorological stations clearly this does not include ocean data other giss graphs are available here how do the giss computed anomalies compare with unadjusted anomalies the 1961 1990 mean from the giss anomalies was 0 09 whereas the 1961 1990 mean from my anomalies was 0 00 clearly the giss data is adjusted further after the conversion to anomalies clearly something needs to be done to reduce the variance in the raw temperature data let s take a look at temperature within 10 degree latitude bands i e latitude 80 is 75 to 84 99 using all sectors with 240 days of data within the year first pass average temp by latitude x10 awk if 1 0 lat int 1 5 10 else lat int 1 5 10 temp lat 3 4 n lat 3 years 3 1 lats lat 1 end printf year for l 90 l 90 l 10 printf lat d l printf n for y 1900 y 2009 y printf d y for l 9 l 9 l 1 printf f temp l y n l y 1 0 printf n h tavg_1x1 txt h tavg_bylat_globe txt two surprises there was no data in the range 45 to 84 99 latitude and none above 80 latitude the entire lat 80 band is data from 75 to 79 99 at last a warming trend has been found perhaps this graph is easier to read latitudes with no data have been eliminated the legend has been rearranged with equatorial latitudes at the top near their temperature plots the arctic trend will be considered in depth here https justdata wordpress com arctic trends uhi effects will be considered as well this one is too good to hide greater nyc area versus 40n latitude stations and 35n to 45n stations gather up all the sectors in the 40 0 to 40 99 latitude range odd side effect of my gridding method ny new york cntrl prk lat 40 7800 long 73 9700 ends up in the lat 40 long 74 sector awk begin print year 40n 35 45n nyc 1 40 2 74 avg 3 4 n 3 1 35 1 45 zavg 3 4 zn 3 1 40 2 74 nyt 3 4 nyn 3 end for y in avg print y avg y n y zavg y zn y nyt y nyn y tavg_1x1 txt sort n 40n_latitude3 txt 25 comments posted in uncategorized posted by genezeien december 13 2009 nothing to see here move along i am going to track down raw unadjusted climate data apply straight forward analysis methods present results code and data sources 5 comments posted in uncategorized categories uncategorized recent posts finding raw temperature data monthly gridded data in easy to use format style vs content step by step is the climate changing nothing to see here move along top posts anomalies finding raw temperature data monthly gridded data in easy to use format style vs content urban heating effects step by step is the climate changing arctic trends about nothing to see here move along create a free website or blog at wordpress com subscribe subscribed data analytics sign me up already have a wordpress com account log in now privacy data analytics subscribe subscribed sign up log in report this content view site in reader manage subscriptions collapse this bar loading comments write a comment email required name required website design a site like this with wordpress com get started