Blog

Introduction

NIWA Collects Data - lots of it.  The data falls into two main categories: that collected automatically at fixed locations and monitoring stations, and that collected by ad-hoc field surveys. 

In the first category falls the major Climate, Meteorological and Hydrological sensor networks.  Climate data is stored in the National Climate Database while Hydrological data used to be stored in a TIDEDA database but is now moving to an Aquarius system.  In the second category falls all the biodiversity field surveys such as freshwater fish, invertebrates, aquatic plants etc.  The National River Water Quality Network (NRWQN) is effectively a time series of water quality measurements but they are managed as ad-hoc surveys at common locations.  This survey data is stored in NEMO - The NIWA Environmental Measurements and Observations database.  NEMO has been specially designed to accommodate a wide and varied range of different observation types and survey methods. 

So having collected all that data how can we make it available to interested parties and researchers?  In order to do so we need to solve several key problems for the end users:

  • How to discover that we hold data of interest.
  • How to access that data.
  • How to interpret it correctly. 

NIWA's strategy is first, wherever possible, make the data and metadata available through open interoperable standards. Then to use those standards ourselves to implement a data discovery and access portal, and finally to provide examples and tutorials to help others access and process the data. 

As a result NIWA now provides and continues to develop a range of Open Web Services (OWS) including OGC Metadata, Catalogue Services (CSW), Web mapping services (WFS, WMS) and Sensor Observation Services (SOS).  We will use these services to provide free and public access to a core selection of data sources along with paid access to a wider and deeper range of data sources for commercial users.

The EI Browser acts as a portal both to discover data, but also to view it online, download data sets and open connections to the web services. 

A tour of the EIB

http://ei.niwa.co.nz

The EIB provides a search and visualisation interface on a set of disparate data sources. In order to get started you can perform a meta level search on the home page that will give you an idea of the assets held for a given location and time frame. 

Home Page

You can select a location by entering a place or region name in the where field or selecting an area on the map.  If you do neither of these we will search the visible map area. 

By Selecting a date range you will limit results to those collected within or overlapping with the date range. Clicking on the date range field pops up a form that lets you rapidly select useful ranges such as last month or last year. 

Date Range Selection

Results

The results of the Meta level search appear in the bars over the map, and also on the menu bar.  This tells you how many hits you are likely to find in each of the main search areas.  Click on one of the bars to carry on the search in more detail.

 

Metadata Catalogue Search

NIWA Maintains a metadata catalogue based on the GeoNetwork software. This holds records for all many NIWA datasets and continues to grow.  This catalogue is available directly at http://dc.niwa.co.nz and you can also search it using the CSW protocol - more on that later.

If you open the page by following the link from the home page the previously found results are immediately shown.  

However you can refine your search by restricting the results to a location, time frame and keywords.

Click on the Metadata Record result to view the record in more detail. This opens in a new window so that you can open several records at once without losing your search results.

If the metadata references online resources such as WFS feeds, Maps etc. Then these will be listed here and you can use the URLS in your downstream applications. The metadata record will also tell you about any public/private status and access or usage restrictions for the data.

These metadata record pages are bookmark able and you can reference the URLS e.g.

National River Water Quality Network (NRWQN)

Stations Search

The Stations Search page allows uses to search the NIWA Stations Catalogue which lists metadata and management information for all NIWA monitoring stations as well as those of other organisations for which we have at some time held data. 

Again you can restrict the search by station type, organisation and any specific time series monitored at that location.  

Here we can see NIWA Climate stations in the central south island.  I changed the underlying google base map to make them more visible. 

By setting a date range up to the present we can ensure that we see only stations with recent data.  

 

Legend

If the station dot is surrounded by the blue square then that site provides access to the underlying time series data via the Sensor Observation System SOS.

Stations WFS

This search makes use of the NIWA Stations WFS Service. Click on the Show Current WFS Filter link in the panel heading to find out the URL and filter. 

To use this you will need some software that allows you to POST the WFS request to the server URL provided.  If you have the command line tool  CURL then you can do something like this:

server_url="paste url above here"
wfs_request="paste WFS request body here"
curl --header "Content-Type: text/xml" -d "$wfs_request" $server_url
 

 

There are more examples of how to use these requests in various programming languages here https://github.com/EODP-NZ/eib-ows-examples

Select a Station

Click on the map over a station marker. 

Clicking on the map will show a list of the nearby stations.  If there are several it may be because they are close to each other - or sometimes because the station has been known by a different name in the past. 

Click the line in the popup to open the station details page.

Station Page

The station page shows several panels - a Metadata details panel, An external links panel and if time series data is available a plots panel.

Station Metadata Details

This panel lists some of the metadata values found in the WFS response and shows the location of the station on a small map.

Station Plots

If the station has public online data available through the SOS protocol then this station plots panel will be shown.  Use the selection menu list to view the list of available time series observable properties.  You can turn on any or all of the time series and each will appear in a new chart panel.

Chart Panel

 

Each chart panel shows one of the available time series.   At this stage the amount of data is limited to 6 months in order to not overload the server. You can use the small time frame zoom bar under the chart to adjust the time extent of the main chart. 

The sidebar for the stations page will show any available links associated with the metadata. This includes a link to the NIWA Station Information Management System (SIMS)  and for each chart selected a link to the SOS request that generated the data for the chart.  This will open in a new window and show a large page of XML in the Observations and Measurements Format. 

You can connect this URL to any tool or programming language that understands SOS to retrieve and further analyse the data. 

 

Freshwater Biodiversity

The Freshwater Biodiversity (FBIS) Search page allows uses to search the NIWA NEMO Database for freshwater biodiversity data sets such as the NZ Freshwater fish database, Aquatic Plants, Invertebrates etc.   We hold a wide range of data sets from NIWA field surveys and other organisations such as DOC, EOS and Cawthron.

You can restrict the search location, species, animal type, and data source as well as selecting a time range and map polygon.  

Here we can see Fish of species Galaxias in the Auckland area sampled in the last five years.  - there are 68.  The points actually represent sampling events at which multiple samples may have been taken - some of which include one or more Galaxias.

Select a sampling Event

Click on a point to show the sampling events at that location.  Then click on the entry to see the full details of the samples at taken.

 

FBIS WFS

This search makes use of the NIWA FBIS WFS Service. Click on the Show Current WFS Filter link in the panel heading to find out the URL and filter. 

note the slightly different server endpoint - nemo instead of stations.

Sampling Event Report

The result is a sampling event report. Depending on the survey method this may list the location details, catch operations, habitat, and for each sample information such as the taxon, life stage, presence count and length etc. 

 

 

 

...

This report may be of limited interest on its own but it gives you access to all the observations made in the field.

Download Data

Currently we do not have an Open Data Standard protocol to retrieve this Sampling Event information. At some point in the future we will make this available through GBIF and possibly SOS links if we can work out how to map biodata onto O&M formats.

However you can still obtain a CSV formatted table of results for all the sampling events and samples matching the search criteria from the search page. 
First you have to sign in.
If you have not used any NIWA data or online services before you will first need to register and create an account.   You will then need to request access to the FBIS data service. 
Once signed in you have access to a range of downloads and reports. These will vary according to the type of samples selected. 
As there may be a large number of sampling events and samples as well as many observable properties for each sample the final CSV file may take several minutes to prepare.  Once complete the button will change to a download report link.
You should be able to save the resulting file and open it in Excel or read it into an R or Python program.

 

  "NIWA FBIS Report. We found 89 search results."
  fid, Sampling event id,Sample id, NZFFD Card number,Month,Year,NZFFD Catchment,Locality,Start time,Organisation,Easting,Northing,Altitude,Distance inland,Collection method,FWF Fishing Effort,Effort,Sample attempts,Species,Abundance,Presence Count,Presence percentage,Min length,Max length,NZ Reach id,Sample types,Start date,Longitude,Latitude,Datasource,Date uploaded,Meta data uuid,Geom
  fid--55599cc0_149a269c43c_1ed9,69842,238780,30964,3.0,2010.0,079.040 Henderson C,Opanuku Stream,,boffa miskell ltd,2655200,6479200,10,3.8,Electric fishing - combination of nets and electric fishing,50,Electric fishing passes: 1,1,Galaxias maculatus (Inanga),,21,,65,100,2005778,Fish,2010-03-

You must select either a region, or bounding box on the map in order to get some results in the downloaded CSV file.

 

Freshwater Chemistry

The freshwater chemistry search page operates much like the stations and biodiversity pages except that the search is restricted to stations in the National River Water Quality Network (NRWQN)  The observations made here take place on a regular basis and involve taking water samples and providing the results of chemical analysis. 

In order to access this data you need to be registered, subscribed to the FCHEM data source and signed in.

There is no keyword search here as all the stations sample the same set of parameters. 

Select a Station

Click on the map at one of the markers to view the station summary

Then click on the station link to view the station details page

Freshwater Chemistry Station

This page is somewhat similar to the climate and hydro stations page. There is a Station Metadata Panel, A Charts panel and in addition a table panel that shows the numeric values of the measurements taken at the station.  

 

Observed Properties

Data Table

You can use the selection list to control which parameters are shown in the data table. 

Download Data

Currently the FCHEM data is not available via SOS.

However you can download a CSV file of the station chemistry data from the left sidebar.  This file will contain all the available measurements.

 

Marine Biodiversity

The Marine Biodiversity page allows you to search a small but growing set of data sources. including the IPY CAML Biological Data Set, Coraline Algae and Plankton.  These samples come from the seas around New Zealand and the Antarctic Ross Sea.  

Much more NIWA data is available through IMOS - The integrated Marine Observing System and the AODN Australian Ocean Data Network http://imos.org.au/aodn.html  AODN is a similar system to EIBrowser built upon the same Open data web services.  Here you will find NIWA Bathymetry layers and CTD station data.

Sampling Events 

The sampling event selection and report are the same as for Freshwater Biodiversity

Downloads

As for Freshwater you can download a CSV file containing the selected samples.  Note that you must select either a region or bounding box on the map in order to get some results.

 

This November NIWA Systems Development Team released a new version of the EI Browser.  V2.02.

I'd like to give you a quick tour of the features and our online data strategy for Environmental Information. http://ei.niwa.co.nz