Welcome to the home page for the NIWA QGIS User Group. This is a publicly readable web page hosted by NIWA to support users of QGIS, particularly within NIWA, but any QGIS user is welcome!
Why are we here? Based on the understanding that the benefits from any Open Data initiative revolve around data mash-ups and re-use, providing access to a tool allowing people (including NIWA staff) to use all this free environmental data is an important part of any such initiative. QGIS is a core part of NIWA's information delivery strategy - for NIWA staff as well many of our other clients.
Much like the development of easily used and affordable word processing & spreadsheet software for personal computers enabled pretty much anyone to carry out what used to be specialised tasks, freely available desktop mapping and GIS software are enabling their democratisation and widespread general use. Note that we do not envisage QGIS as necessarily meeting all the needs of all GIS users, but NIWA and many other agencies in New Zealand and globally, are adopting tools like QGIS to enable more widespread use of institutional GIS resources, without the traditional expense associated with commercial GIS software.
So, here we are!
A new NIWA year begins, mid-2017...
With the new NIWA financial year starting in July, staff training clicks over another year. Several NIWA staff have identified QGIS and Postgres (with Postgis) as subjects they'd like to learn about.
My request for NIWA Wellington people to get in touch about these yielded the not unexpected result - different people wanting different things... Some want to start at a simple beginning, others have a more complex and detailed requirement... so I'm looking at holding several topic focused seminars, which one or more people can attend, and learn the bits that are relevant for them. Sort of like the local NIWA R-users group seminars organised by Andy McKenzie, but more interactive, so people can try what is being discussed. These will be held at NIWA and directed at NIWA staff, but if numbers permit, open to others to attend as well.
I think it is worth noting that staff also asked for generic GIS and also ArcGIS training. If we look at the common definitions of "doing GIS", it includes managing, querying, visualising, analysing, mapping and reporting on spatial data. This implies that GIS in NIWA includes all the tools staff use to do these activities, rather than just the tools that call themselves "GIS", which includes includes tools like R, Matlab, Python and IDL, as well as ArcMap, QGIS, Mapinfo, GMT, etc. I'll expand on this because I think it is important within NIWA.
NIWA one GIS (see: https://one.niwa.co.nz/display/GIS/one.NIWA+GIS+datasets)comprises an ESRI ArcServer setup, providing data for NIWA ArcMap users, primarily datasets (map layers or feature classes) of wide relevance to NIWA staff. Enter the OGC, and GDAL. NIWA is a member of the Open Geospatial Consortium, an international organisation which defines standards enabling data interoperability and reuse in the geospatial domain. GDAL is an open source library used for accessing many different formats of spatial data. Where these come together, is a couple of OGC standards WMS & WFS (Web Map Service and Web Feature Service) which GDAL supports. So if NIWA one GIS provides compatible WMS & WFS services of the data layers managed there, not only ArcGIS users can access them, but other NIWA GIS users can also. QGIS natively supports both, so QGIS users will be able to. There is also rgdal for R users, mexgdal for Matlab users, a GDAL Python bridge for IDL users, and Python GDAL/OGR tools supporting Python users. So the potential exists to have one NIWA GIS to provide spatial data to virtually everyone in NIWA doing GIS, irrespective of the tool they use, thanks to open source and open standards. Watch this space!!
One of the advantages of QGIS, is that by design it has always supported a high level of integration. It does not pretend to provide spatial data management capabilities, instead it tries to bring a range of spatial tools (Postgis, SAGA, GRASS, R, Python and others) together in a common framework, working directly with your existing spatial data, wherever and however it is currently managed.. In NIWA QGIS is used to directly access, query and map Specify data from the NIC stored in a non-spatial MySQL database, fisheries data from NIWA's research trawl database, remote sensing netCDF data, etc... so for people who prefer a tool that accesses and integrates well with existing data, I suggest QGIS is a good start, and is likely to meet all your needs. For R and Python users, QGIS has been well integrated with these almost since it first started being developed, and as another Open Source project, shares many libraries with these and other tools (in addition to GDAL), meaning the learning curve may well be simpler for those already familiar with some of the other tools.
One example of effective integration is Postgres, the open source relational database (with Postgis to manage, query and analyse spatial data). R is one of the programming language that can be used to write Postgres functions, so the result of an SQL query can be an R-rendered graphic representing the data, rather than just the data. Or an R user can easily access data from Postgres databases copied directly into their local R workspace, and QGIS supports embedded R functions as well as Postgres connectivity. Recent training workshops held overseas have included Postgres, QGIS and R as 3 components of an integrated suite, which is a useful approach for NIWA staff.
Accessing NSIDC data services, CCAMLR data and Specify specimen locations.
The NSIDC layers are available in a few CRS's - we will start with EPSG:3031, Antarctic Polar Stereographic. In QGIS, set the default CRS to this (EPSG:3031 WGS/84 Antarctic Polar Sterographic) using Settings -> Options -> CRS
The data for the Antarctic coastline, islands and ice shelf (for this demonstration) is retrieved from the NSIDC (US National Snow and Ice Data Centre) Atlas of the Cryosphere. It is provided via WFS/WMS services which can be accessed using QGIS. Details on the services are at: http://nsidc.org/data/atlas/ogc_services.html We want the actual data, not a map of the data, so we will
You can connect to the service in QGIS using the browser, the layer toolbar or the main menu, as you prefer. These notes use the main menu. Choose Layer -> Add Layer -> Add WFS layerand click New
Save and connect. You will get a list of the available datasets, each of which can be selected and loaded as a new layer into QGIS. You can add any of these you like, this demonstartion will use 5:
Antarctic ice shelves
Coastlines (excluding Antarctica)
Antarctic Polar front
You can select all these in a single operation, by using CTRL-click on each layer, then the Add button. QGIS will connect to NSIDC and download each dataset. This can take a couple of minutes, depending on your network connection. If you get a timeout error, use Settings -> Options -> Network to set the default timeout for network requests to 180000 (3 minutes) which should be enough for most cases. Move the layers up/down until the order makes sense & change colours/styles as you prefer.
You should now have a QGIS Antarctic map, but the Polar Stereographic projection is centred on the pole, so all directions are north - and the "up" direction on the map is 0o, which puts 180o down - along with the Ross Sea and New Zealand. We will now create a new custom projection, based on the Antarctic Polar Stereographic one, but with 180o up.
Choose Settings -> Custom CRS, click the green + button to add a new CRS. Click the right button to copy an existing CRS, Enter 3031 as a filter and choose WGS84 / Antarctic Polar Sereographic. Click OK to copy the parameters into the new CRS Parameters pane, and give the new CRS a relevant name, eg: NZ polar. You now need to edit the parameters to create the NZ up version. Change +lon_0=0 to +lon_0=180 to switch from 0o up to 180o up and save.
Now tell QGIS to use this CRS: Project -> Project Properties -> CRS. Enable "On the fly transformations" and enter NZ as a filter, then choose the new CRS as the project CRS. QGIS will now plot the map using this 180o up polar projection, and automatically reproject layers in other CRS's to this one for viewing on the map.
You should now have your Antarctic map looking correct for zooming in to the Ross Sea/New Zealand. Zoom in to the region of interest.
We will now save each of the NSIDC layers as a local shapefile, ready to re-use whenever needed. For each layer, right-click and choose "Save as". Ensure the format is "ESRI Shapefile", give it a suitable name & location and save it. You can keep the original EPSG:3031 Polar Stereographic CRS.
This is a very good time to save the project!
We will now download a copy of the Antarctic SSRU polygons from the CCAMLR website. If you want other CCAMLR datasets, grab them as well. In a web browser, connect to http://gis.ccamlr.org Click the Download button in the bottom left. This provides access to a wide range of Antarctic datasets, including the NSIDC ones. (For Antarctic biodata - also see the SCAR-MarBIN ANTABIF data portal at http://data.biodiversity.aq/). On the CCAMLR website, choose the Product: Management areas->SSRU's, choose Full dataset for Area of interest, and ESRI Shapefile format, then click Get to download, and unzip the zipped shapefile, then open in QGIS as a vector layer. Label this with ssrucode and style as desrired.
We now have a pretty full map of the region, so will now get some NIWA data to display on the map. This example is using the "virtual datasource" capability of QGIS. This entails creating an XML file which contains information:
where to get the data (type of database, server, database name and user details)
an SQL format query to retrieve the desired data
instructions on how to derive a geometry feature from these data.
The XML file we are using connects to the underlying NIC Specify database, which is a MySQL database, with columns of latitudes and longitudes in decimal degrees. The password can be provided on request, if you require access.
<OGRVRTDataSource> <OGRVRTLayer name="collectionObjects"> <SrcDataSource>MYSQL:niwainverttest,user=specreader,password=XXXXXXXXXX,host=specify6.niwa.co.nz,port=3306</SrcDataSource> <SrcSQL>select catalognumber, latitude1 AS y, case when longitude1 > 180 then -360+longitude1 else longitude1 end AS x,localityName as StationID, startDate, latitude1, longitude1, latitude2, longitude2, maxElevation, minElevation, taxon.fullname as TaxonName, prefT.fullname as PreferredName from collectionobject INNER JOIN collectingevent ON collectionobject.collectingeventid = collectingevent.collectingeventid INNER JOIN locality on collectingevent.localityid = locality.localityid LEFT JOIN determination on collectionobject.collectionobjectid = determination.collectionobjectid LEFT JOIN taxon on determination.taxonid = taxon.taxonid LEFT JOIN taxon prefT on determination.preferredtaxonid = prefT.taxonid WHERE Latitude1 is not null and longitude1 is not null and determination.iscurrent = 1 and catalognumber is not null ORDER BY Catalognumber</SrcSQL> <GeometryType>wkbPoint</GeometryType> <GeometryField encoding="PointFromColumns" x="x" y="y"/> <LayerSRS>EPSG:4326</LayerSRS> </OGRVRTLayer> </OGRVRTDataSource>
This is a virtual data source, as the file QGIS connects to does not contain the actual data, just pointers to the data. The most complicated part of this is the SQL query which requests the required content from the underlying tables in the Specify database. This example will return all longitudes in a +-180 space, so that a value of 185 will be automatically provided as -175. QGIS will automatically reproject these to the new custom CRS using the parameters defining it.
Edit this file to enter the correct password, save and open as a QGIS vector layer (type = VRT - Virtual Data Source). It should only take a few seconds to retrieve and plot the 96,000 records. You can use the usual query builder and style editor to select which records are displayed, and how they look in the map.
The last part of this exercise is to create a printable map from this using the QGIS print composer. While the map is plotted in Antarctic Polar Stereographic meter coordinates, our labels and ticks will be in decimal degrees. Open a new composer and add the map to the page. The next steps are about adding a frame and annotated lat/long grids. In the print composers's "Item properties", open the "Grid" tool, and click the green + to add a new grid. This is "Grid 1" and will be the plain frame around the map.
set Grid type to "Frame and annotations only"
set grid CRS to EPSG:4326
set frame style to "Line border"
Sroll back up the "Item properties" and add a second grid (Grid 2) which will contain the grid with labelled ticks. The settings for this gird are:
set Grid type to "Solid"
set grid CRS to EPSG:4326"
set frame style to "Exterior ticks"
set frame size to 2.0mm
set X and Y intervals to 10.0
set Left divisions to "Latitude(Y) only"
set Top divisions to "Longitude (X) only"
un-tick Bottom and Right sides
turn on "Draw coordinates"
set Format to "Decimal with suffix"
set Left to "Latitude only", "Outside frame" and either "horizontal" or "vertical" as you prefer
set Right to Disabled
set Top to "Longitude only", "Outside frame" and "Horizontal"
set Bottom to "Disabled"
set Coordinate precision to 0
save as an image
crop as desired (GIMP, etc)
You should have a map looking something like this:
For a formal comparison between ArcGIS and QGIS click here.
This study was undertaken to assess the implications of migrating a specific existing ESRI implementation to QGIS. As such,it compares the capabilities of QGIS compared with existing workflows developed over time around an ESRI installation. For those considering implementing a new GIS system, many of the compatibility issues are likely to be largely irrelevant, as are some other issues (and costs) such as data migration.
This approach obviously highlights the inability of QGIS to work with ESRI proprietary formats, and does not always investigate how a task may be accomplished with QGIS, but undertaken differently from the ESRI approach, although some such situations are covered.
It does compare some advanced analytical capabilities, and in some cases considers the use of plugins & the ability of QGIS to utilise other applications (notably GRASS. SAGA, GDAL and TauDEM) to undertake geoprocessing tasks which QGIS does not support natively.
It mentions the combination of QGIS/Postgis/Geoserver as a better comparison with ARC Server, but does not really consider other open source applications which can also be used to build a more complete open source GIS suite, such as Geonetwork for metadata manangement, or commercial tools like Geocat Bridge to enable a higher level of interoperability between ESRI and the open source/open standards based tools. The ability of Arc GIS to share a native Postgis datastore with QGIS is not described, and some approaches which might use Arc as a central server with desktop access for users via QGIS in a mixed model are also not investigated in any great depth, although the possibility is mentioned.
The report does generally describe the limitations of what it covers, and provides a useful comparison.
Abstract: NIWA manages several research databases for MPI Fisheries, which QGIS can be used to access. This demonstration will show how QGIS can be used to access & display research trawl survey data, eg:
one (or more) species from all trips
selected species from a particular trip
symbols sized by catch size
trawls with zero catch of a species
survey strata boundaries
These data can be overlaid with topo maps or marine charts, freely downloadable from LINZ.
Anyone familiar with the trawl data, stratified surveys or pretty much any species observation/distribution data should find this useful and relatively easy. The trawl database has several decades of species catch data, which can be accessed and viewed.
An additional feature is the use of the QGIS Action tool (originally developed by Gavin Macaulay of NIWA) to retrieve catch data from the database by clicking on a station on the map - this is particularly useful as it allows users to retrieve non-spatial data reference to a point on a map - the catch data (and underlying catch database table) contains no position data, but can still be retrieved in QGIS.
The demonstration is based around this pdf document - a How-To guide to access research trawl data.
Click to view for a more detailed view of Landcare Research's freely available NZ basemap! (internet mapping with QGIS)
Please feel free to add comments & suggestions.
Questions can go here, but the mailing list is preferable.
If anyone wants to help manage these wiki pages, please let me know!!!