Sign inRegisterCheck connection

Weather Display Year-Climate HTML Is a Fragile Data Source

Weather Display climatedatayearout.html is a fragile HTML source. Why monthly and annual summaries are not logger dumps, and what to quality-control before graphing.

Back to PHP weather scripts and software guides

Weather Display can upload a file named climatedatayearout.html: a yearly climate summary rendered as a web page. Historical TNET Weather PHP at /scripts/climate1 scraped that page into an array so the operator could graph it. The scrape is not rehosted. The lesson is still current. HTML is a presentation format. Using it as a climate archive is a fragility that quality control has to offset before any chart is honest.

A sibling URL, /scripts/climate1/script.php, is about turning an already-parsed array into a published table. This page is the problem statement: why the year file exists, how it differs from a logger dump, and what to check before the numbers become a plot. A second WD export, climatedataout.html, is covered on climate2.

What the year-climate page was for

Weather Display builds climate and NOAA-style reports from the station’s own history, then FTPs HTML to the public host (WD web page help). climatedatayearout.html is one of those presentation files: monthly and annual statistics laid out for a browser. Operators wanted the same numbers in a charting library. Parsing the HTML looked easier than reading WD’s native logs, especially on a host that only had PHP and the uploaded webfiles directory.

That shortcut couples your science to the vendor’s layout. The historical TNET note on this URL already recorded the failure mode: the source HTML format changed, and the script stopped working. That is the expected life cycle of an HTML parser, not a rare accident.

Monthly and annual statistics are not raw samples

A logger dump is a sequence of observed values at a sampling interval: each row is a time and a set of sensor readings. A climate-year HTML page is already a derived product:

  • daily maxima and minima collapsed into monthly means, extremes, or totals;
  • rain accumulated by month and by year;
  • maybe degree-days, sunshine hours, or “departure from average”;
  • placeholders for months that have not occurred yet.

Those summaries hide sample size. A monthly mean temperature from 30 complete days is not the same statistic as a mean from 11 days after a logger outage. Official archives make that distinction visible. NOAA NCEI’s GHCNd applies automated quality assurance to daily values, and monthly and annual products such as GSOM/GSOY are computed from those controlled dailies, with extra validation after aggregation (GSOM/GSOY documentation). A WD HTML table does not ship those flags.

Before you graph a parsed year file, you are looking at WD’s idea of a climate summary, not at the anemometer. That can still be useful for a station diary. It is the wrong input if you need to recompute a mean with a different missing-data rule.

Why HTML is a brittle extract surface

Presentation HTML encodes layout: <table>, <td>, &nbsp;, colored fonts, “------” for unused days, unit strings baked into headers. A parser that counts table cells will break when WD:

  • adds a column (a new statistic);
  • changes English labels or translates them;
  • wraps a value in a new <span> or comment;
  • switches units in the header but not in every cell;
  • emits a different dash character for missing months.

Regular expressions against inner HTML are then matching typography. The historical climate1 workflow even offered a CSV download of whatever the parser thought it saw. CSV of a misaligned table is a second, neater-looking error.

Prefer, in this order, when the software allows it:

  1. Native logs or export formats WD still documents (structured text, not a skin).
  2. A dedicated CSV/XML climate export if the operator enabled one.
  3. HTML parsing only with a fixture test that fails when the layout moves.

Saratoga’s templates work from files such as clientraw.txt and tag files for current conditions (Saratoga WD setup), not from climate HTML. That split is correct: live conditions and climate summaries are different products.

Quality control before graphing

Treat every extracted cell as suspect until it passes station-level checks. You do not need NOAA’s full GHCNd suite to be more careful than a raw echo. Minimum checks:

  • Identity. Year, station name, and units in the HTML header match the file you think you parsed. A cached climatedatayearout.html from last year graphed as “this year” is a common FTP mistake.
  • Calendar. Twelve month slots. Confirm whether slot 1 is January. Confirm whether the annual column is a sum, a mean, or a WD-specific rollup. Do not assume a water year (see climate2 when two files disagree).
  • Completeness. Partial years need a sample-size annotation. Do not draw a smooth annual cycle through empty months.
  • Physical range. Monthly mean temperatures outside a plausible local envelope, negative rainfall, or humidity above 100% are parser or unit errors until proven otherwise.
  • Internal consistency. Monthly rain totals should sum toward the annual total within rounding. Mean temperature should lie between monthly mean maximum and minimum if all three exist.
  • Trace and zero. A trace is not 0.00 and is not missing. Official NWS climate services define a trace of liquid precipitation as less than 0.005 inch (NOWData FAQ). If WD printed T or 0.00 you must keep that distinction in the array, not coerce both to float zero.
  • Unit labels. in versus mm, °F versus °C, and inHg versus hPa. A header in mixed units with a body in one unit is a WD skin issue; the graph will not warn you.

Only after those checks is a chart a picture of the station climate. Until then it is a picture of the parser.

Observed, derived, and historical

Keep the wording exact when you publish:

  • Observed: a value the logger recorded (you will rarely get this from year HTML).
  • Derived: monthly mean, departure from average, degree-days, annual total—WD computed them.
  • Historical: a recovered table from a past site, which must carry its year and units if you quote it at all.

Do not describe a WD “departure from average” as an NCEI climate normal. Normals are a specific official product with a defined period. A station average inside Weather Display is whatever years that copy of WD has in its file.

The historical TNET Weather site used this path for a Mesa-area station’s year-climate scrape. The modern article does not replay those numbers. They would be an undated HTML extract without the quality flags this page argues for.

What to do instead of worshipping the scrape

If you still run Weather Display, export climate in the most structured form the version supports, keep the native data files, and treat HTML as a view. If you only have old HTML, parse it once into a table with explicit missing codes, archive the HTML beside the table, and never parse live on every page hit.

The scripts listing originally linked climate1 as “not a complete script”—it only obtained values. That limitation was honest. Obtaining values from presentation HTML is the risky half of the job. Publishing them without repeating the risk is the other half, on /scripts/climate1/script.php.

TNET’s own research documentation starts from named sources and quality controls rather than from scraped skins. That is the same discipline applied at a different scale: data sources, quality controls, and methodology. The legacy scripts hub indexes the related Weather Display utilities without offering the old PHP as a download.