nsw_lidar
Differences
This shows you the differences between two versions of the page.
Both sides previous revisionPrevious revisionNext revision | Previous revisionNext revisionBoth sides next revision | ||
nsw_lidar [2021/02/10 08:11] – bushwalking | nsw_lidar [2021/02/20 09:52] – bushwalking | ||
---|---|---|---|
Line 1: | Line 1: | ||
- | ====== | + | ====== |
NSW Spatial Services have undertaken a program to map all of NSW using lidar (light detecting and ranging) | NSW Spatial Services have undertaken a program to map all of NSW using lidar (light detecting and ranging) | ||
For details, see information on their [[http:// | For details, see information on their [[http:// | ||
Line 7: | Line 7: | ||
It can then processed with a GIS such as [[https:// | It can then processed with a GIS such as [[https:// | ||
- | ====== | + | ====== |
- | * [[https:// | + | |
- | * [[http:// | + | |
- | ====== Managing DEMs ====== | + | The original topics here are being progressively moved to their own pages |
- | ===== Downloading data ===== | + | ===== Managing DEMs ===== |
- | While data can be downloaded in an ad hoc manner, if you are regularly processing DEMs, it is better to have the DEM tiles already downloaded. | + | * [[nsw_dems|Merging DEMs]] - how to merge DEMs in QGIS |
+ | * [[nsw_dems|Managing large DEMs]] - while data can be downloaded in an ad hoc manner, if you are regularly processing | ||
- | The Sydney basin and Blue Mountains is around 50GB all up. | + | ===== Topographic maps ===== |
- | For example, to request | + | There are several primary |
- | {{:: | + | |
- | Below are the 1:100k map areas around Sydney: | + | * [[qgis_depressionless_dem|Hydrologically correct DEM]] - prior to further operations, it is usually important to make sure that your DEM is free of depressions. Otherwise these will mess up streams and possibly contours. |
- | |8833_gulgong|8933_merriwa|9033_muswellbrook|9133_camberwell|9233_dungog|9333_buladelah|9433_forster| | + | Once you have a depressionless DEM, the following items can be generated: |
- | |8832_mudgee|8932_mt_pomany|9032_howes_valley|9132_cessnock|9232_newcastle|9332_port_stephens| | | + | * [[qgis_contours|Contours]] |
- | |8831_bathurst|8931_wallerawang|9031_st_albans|9131_gosford|9231_lake_macquarie| | | | + | * [[qgis_hydrology|Hydrology (Stream Network)]] |
- | |8830_oberon|8930_katoomba|9030_penrith|9130_sydney| | | | | + | * [[qgis_slope|Slopes]] - while not a standard feature of topographic maps, this can be a simple way to identify cliffs |
- | |8829_taralga|8929_burragorang|9029_wollongong|9129_port_hacking| | | | | + | * Clifflines |
- | |8828_goulburn|8928_moss_vale|9028_kiama| | | | | | + | |
- | |8827_braidwood|8927_ulladulla|9027_jervis_bay| | | | | | + | |
- | ===== Pre-processing data ===== | + | ===== Styles |
- | The following may be useful for Windows users. | + | |
- | Below is a Windows Powershell script that will | + | |
- | | + | |
- | * extract the raw data from the remaining zip files | + | |
- | * convert all of the .ASC files to GeoTIFF | + | |
- | * move the old zip files to a current sub-folder | + | |
- | * zip the current and archive sub-folders to temp.zip | + | |
- | * create a virtual raster (.vrt) file of all of the GeoTiffs | + | |
- | You will need to replace the Environment variables (Path, GDAL_PATH, GDAL_DRIVER_PATH, | + | ===== Automation ===== |
- | Usage is: | + | * [[qgis_basic_automation|Basic map creation]] - a set of PyQGIS scripts that will create (and optionally save) a basic topographic map. This map can be used in QField |
- | < | + | |
- | eg < | + | |
- | The .\ is required if you are running the script from your current directory. | + | ===== Mobile apps ===== |
- | <file powershell buildvrt.ps1> | + | |
- | $zipFile=$args[0] | + | |
- | $targetFolder=$args[1] | + | |
- | + | ||
- | # Unzip files from all subdirectories to new folder | + | |
- | Expand-Archive -LiteralPath $zipFile -DestinationPath $targetFolder | + | |
- | Get-ChildItem -Path " | + | |
- | Get-ChildItem -Path $targetFolder -Directory | Remove-Item -Recurse | + | |
- | + | ||
- | # Create hash of zip files, by name (location, date) | + | |
- | $zipFileList = @{} | + | |
- | Get-ChildItem -Path " | + | |
- | $zipFileList.add($_, | + | |
- | $_ -match ' | + | |
- | $zipFileList[$_][' | + | |
- | $zipFileList[$_][' | + | |
- | $_ -match ' | + | |
- | $zipFileList[$_][' | + | |
- | } | + | |
- | # $zipFileList | + | |
- | + | ||
- | # Create hash of location (date, name) | + | |
- | $locationList = @{} | + | |
- | $zoneCount = @{} | + | |
- | $zipFileList.keys | ForEach-Object { | + | |
- | if($locationList[$zipFileList[$_][' | + | |
- | $t = $locationList[$zipFileList[$_][' | + | |
- | $t.add($zipFileList[$_][' | + | |
- | } else { | + | |
- | $t = @{} | + | |
- | $t.add($zipFileList[$_][' | + | |
- | $locationList.add($zipFileList[$_][' | + | |
- | } | + | |
- | if($zoneCount[$zipFileList[$_][' | + | |
- | $zoneCount[$zipFileList[$_][' | + | |
- | } else { | + | |
- | $zoneCount[$zipFileList[$_][' | + | |
- | } | + | |
- | } | + | |
- | # $locationList | ConvertTo-Json | + | |
- | # $zoneCount | ConvertTo-Json | + | |
- | + | ||
- | # Create archive folder | + | |
- | $archiveFolder = " | + | |
- | If(!(test-path $archiveFolder)) | + | |
- | { | + | |
- | New-Item -ItemType Directory -Force -Path $archiveFolder | + | |
- | } | + | |
- | + | ||
- | # Sort each location by date desc, and move old files to /archive | + | |
- | $locationList.keys | ForEach-Object { | + | |
- | $i=0 | + | |
- | $locationList[$_].GetEnumerator() | sort key -des | ForEach-Object { | + | |
- | if ($i -eq 0) { | + | |
- | $i++ | + | |
- | return} | + | |
- | else { | + | |
- | #$_ | ConvertTo-Json | + | |
- | $s = $_.Value | + | |
- | Move-Item -Path " | + | |
- | } | + | |
- | } | + | |
- | } | + | |
- | + | ||
- | # You can't build a VRT with files from a different projection, so | + | |
- | # delete files from outside main zone | + | |
- | # This could probably be altered to include a step to reproject those files | + | |
- | $mainZone = '' | + | |
- | $zoneCount.GetEnumerator() | sort value -des | select -first 1 | ForEach-Object { | + | |
- | $mainZone = $_.Name | + | |
- | } | + | |
- | + | ||
- | $zipFileList.keys | ForEach-Object { | + | |
- | if($zipFileList[$_][' | + | |
- | Remove-Item -Path " | + | |
- | } | + | |
- | } | + | |
- | + | ||
- | Get-ChildItem -Path " | + | |
- | $Env:Path += "; | + | |
- | $Env: | + | |
- | $Env: | + | |
- | $Env: | + | |
- | + | ||
- | Get-ChildItem -Path " | + | |
- | $srcFile = $_.FullName | + | |
- | $destFile = $_.FullName -replace ' | + | |
- | &" | + | |
- | } | + | |
- | + | ||
- | Get-ChildItem -Path " | + | |
- | Get-ChildItem -Path " | + | |
- | Get-ChildItem -Path " | + | |
- | Get-ChildItem -Path " | + | |
- | + | ||
- | # Create current folder | + | |
- | $currentFolder = " | + | |
- | If(!(test-path $currentFolder)) | + | |
- | { | + | |
- | New-Item -ItemType Directory -Force -Path $currentFolder | + | |
- | } | + | |
- | + | ||
- | # Move zip files to current folder | + | |
- | Move-Item -Path " | + | |
- | + | ||
- | # Zip /archive & /current to new zip folder | + | |
- | Compress-Archive -Path " | + | |
- | + | ||
- | # Create 2m vrt | + | |
- | New-Item " | + | |
- | Get-ChildItem -Path " | + | |
- | &" | + | |
- | </ | + | |
- | + | ||
- | It is possible to then build a larger virtual raster from the individual 1:100k virtual rasters. | + | |
- | + | ||
- | ===== Loading data ===== | + | |
- | + | ||
- | Loading up a large virtual raster into QGIS can be very slow, as can manipulating it. However, you can quickly load up a smaller section of the map using the following steps: | + | |
- | * create a polygon using https:// | + | |
- | * download the polygon using the **Export drawn data to GeoJSON** function | + | |
- | * load up the Python console (// | + | |
- | * run the following command (replace file locations with your own) | + | |
- | + | ||
- | < | + | |
- | resultClip = processing.runAndLoadResults(" | + | |
- | </ | + | |
====== Topographic maps ====== | ====== Topographic maps ====== | ||
Line 187: | Line 48: | ||
The steps below are works in progress to determine effective (the best?) ways to extract the various items out of the DEM data for use in topographic maps. Any feedback/ | The steps below are works in progress to determine effective (the best?) ways to extract the various items out of the DEM data for use in topographic maps. Any feedback/ | ||
- | |||
- | ===== Merge DEMs ===== | ||
- | |||
- | The NSW DEM data is supplied in 2km squares. The squares need to be merged into a single DEM for further operations. | ||
- | |||
- | Most of the eastern ranges, where a lot of bushwalking happens, are 2m DEMs. The coast is typically 1m, and the western slopes and plains are 5m (with major rivers 1m!). Be careful if you need to merge DEMs with different resolutions - see below for more details. | ||
- | |||
- | ==== Virtual Raster ==== | ||
- | |||
- | While merging can be done in theory using a Virtual Raster (Raster- > Miscellaneous -> Build Virtual Raster... ), I have had poor performance with this (including recent version 3.12 Bucuresti). Any operation seems to result in screen redrawing, so moving around and zooming in and out is slow and painful. | ||
- | |||
- | That said, if you are just using the Virtual Raster for future steps, then the limitations from the screen redrawing may not be important. | ||
- | |||
- | QGIS uses [[https:// | ||
- | |||
- | |||
- | ==== Merge Raster ==== | ||
- | |||
- | I generally use the the Raster- > Miscellaneous -> Merge... function | ||
- | |||
- | |||
- | QGIS uses [[https:// | ||
- | |||
- | ===== Fill Sinks ===== | ||
- | |||
- | From the initial DEM, first step is to Fill Sinks. Otherwise you will get sinks in the middle of watercourses, | ||
- | |||
- | There are various related tools in the Processing Toolbox that will do this, including: | ||
- | * SAGA : Terrain Analysis - Hydrology : Fill Sinks | ||
- | * SAGA : Terrain Analysis - Hydrology : Fill Sinks (Wang and Liu) | ||
- | * SAGA : Terrain Analysis - Hydrology : Fill Sinks XXL (Wang and Liu) | ||
- | |||
- | The results from all will be similar, but the Wang and Liu versions should be faster. | ||
- | |||
- | There are other approaches that deepen channels rather than fill sinks in order to get a hydrologically sound drainage network. For example | ||
- | * SAGA : Terrain Analysis - Hydrology : Sink Removal | ||
- | has an option for this. | ||
- | |||
- | ===== Contours ===== | ||
- | ==== Basic Processing ==== | ||
- | There are various contour extraction algorithms in QGIS, for example: | ||
- | * GDAL : Raster Extraction : Contour (same as Raster -> Extraction -> Contour...) | ||
- | |||
- | Below is an example of contours created without and with sink removal. The contours on the right have been derived from a DEM where the sinks (in yellow on the left) have been filled. | ||
- | {{: | ||
- | {{: | ||
- | |||
- | Even with sink removal, small | ||
- | |||
- | ==== Simplifying ==== | ||
- | |||
- | Vectors can be compressed by using something like: | ||
- | * Vector geometry : Simplify | ||
- | A tolerance of 1(m) seems reasonable for 1:25000 mapping. Smaller tolerances may be appropriate for larger scale maps (eg 1:10000, 1:5000). | ||
- | |||
- | For more options in compression, | ||
- | * GRASS : [[https:// | ||
- | V.generalize can also be used to smooth contours - possibly best done prior to simplificiation | ||
- | |||
- | ==== Cleaning ==== | ||
- | |||
- | Once simplified, it is worth removing small closed loops, such as those in the image below. | ||
- | {{: | ||
- | |||
- | Here is one approach, which involves adding a length attribute to each contour, and removing those that fall below a certain length. It may cause issues if you have short sections of contour near the edge of the map that you need. | ||
- | |||
- | * Open Attribute Table (F6) | ||
- | * Open field calculator (Ctrl+I) | ||
- | * Add new attribute length, calculated as $length | ||
- | {{:: | ||
- | * Select all features and filter on length < 25 (or whatever length is appropriate for your scale) | ||
- | {{: | ||
- | |||
- | ==== Contour Labelling ==== | ||
- | |||
- | See separate page on [[[qgis_contour_labelling|QGIS Contour Labelling]] | ||
- | |||
- | ===== Hydrology (Stream Network) ===== | ||
- | |||
- | The starting point for hydrology is a hydrologically sound DEM, as above. Use a fill sinks or channel deepening algorithm. | ||
- | |||
- | ==== Catchment Areas ==== | ||
- | |||
- | Next step is to create Catchment Areas. Again, there is a Catchment Area tool (in fact several), and six methods within the tool. For the purpose of delineating watercourses in steep terrain, the choice of method probably makes little difference. | ||
- | |||
- | * SAGA : Terrain Analysis - Hydrology : Catchment Area | ||
- | |||
- | This gives an output that is best viewed in log scale. You can do this via | ||
- | * Raster -> Raster Calculator... | ||
- | * log10 ( " | ||
- | |||
- | Use the log scale version to determine the cutoff for what streams you want to see and which ones are too small. 10000 seems to give comparable results to the existing 1:25000 maps. | ||
- | |||
- | Note that if you don't have the entirety of the catchment, you may get erroneous results. | ||
- | |||
- | ==== Channel Network ==== | ||
- | |||
- | The following tool can be used to create channels (streams) - there are other options: | ||
- | * SAGA : Terrain Analysis - Channels : Channel Network | ||
- | |||
- | Use | ||
- | * Elevation = Filled DEM | ||
- | * Initiation Grid = Catchment Area | ||
- | * Initiation Type = Greater Than | ||
- | * Initiation Threshold = 10000 (or whatever number you have determined) | ||
- | |||
- | {{: | ||
- | |||
- | ==== Classification ==== | ||
- | |||
- | For 1:25000 maps, I've had reasonable results from using the following formula in the Raster Calculator to classify the streams into categories. Different scales may need different bounds, and this doesn' | ||
- | |||
- | '' | ||
- | |||
- | * Intermittent: | ||
- | * Minor: 6.15-7.4 | ||
- | * Major: 7.4+ | ||
- | |||
- | ==== Convert to Vector and Simplify ==== | ||
- | |||
- | Convert to vector using r.to.vect | ||
- | |||
- | {{: | ||
- | |||
- | The raw stream data is very jagged. Smooth using | ||
- | * v.generalize | ||
- | * Algorithm = Hermite (there are other options which can be used, but Hermite has the smoothed line passing through the points of the original) | ||
- | * Maximal tolerance value = 20 (in m, obviously scale dependent) | ||
- | |||
- | Simplify using using: | ||
- | * Vector geometry : Simplify | ||
- | Tolerance:? | ||
- | |||
===== Clifflines ===== | ===== Clifflines ===== | ||
Line 388: | Line 116: | ||
===== Dumping Ground / WIP ===== | ===== Dumping Ground / WIP ===== | ||
==== Resources ==== | ==== Resources ==== | ||
+ | * [[https:// | ||
+ | * [[http:// | ||
[[https:// | [[https:// |
nsw_lidar.txt · Last modified: 2024/04/05 19:00 by bushwalking