Differences

This shows you the differences between two versions of the page.

--- nsw_lidar [2019/02/10 11:29] – bushwalking
+++ nsw_lidar [2021/02/09 23:02] – bushwalking
@@ Line 9: / Line 9: @@
 ====== Resources ======
   * [[https://www.tandfonline.com/doi/full/10.1080/23729333.2017.1300998|A design of contour generation for topographic maps with adaptive DEM smoothing]]
-  * [[http://www.jennessent.com/downloads/tpi-poster-tnc_18x22.pdf|Poster on TPI]]
+  * [[http://www.jennessent.com/downloads/tpi-poster-tnc_18x22.pdf|Poster on TPI by Andrew D. Weiss]]
+====== Managing DEMs ======
+===== Downloading data =====
+While data can be downloaded in an ad hoc manner, if you are regularly processing DEMs, it is better to have the DEM tiles already downloaded. Download tiles by 1:100k map area, which is 0.5 x 0.5 degree squares. Each 1:100k map area ranges from around 2GB to 6GB of data, depending on the number of 2m DEMs vs 1m DEMs, and other factors.
+The Sydney basin and Blue Mountains is around 50GB all up.
+For example, to request data for the Katoomba 1:100k map area (-33.5,150,-34,150.5), see the screenshot below, and Select All DEMs:
+{{::elvis_download.png?nolink|}}
+Below are the 1:100k map areas around Sydney:
+|8833_gulgong|8933_merriwa|9033_muswellbrook|9133_camberwell|9233_dungog|9333_buladelah|9433_forster|
+|8832_mudgee|8932_mt_pomany|9032_howes_valley|9132_cessnock|9232_newcastle|9332_port_stephens| |
+|8831_bathurst|8931_wallerawang|9031_st_albans|9131_gosford|9231_lake_macquarie| | |
+|8830_oberon|8930_katoomba|9030_penrith|9130_sydney| | | |
+|8829_taralga|8929_burragorang|9029_wollongong|9129_port_hacking| | | |
+|8828_goulburn|8928_moss_vale|9028_kiama| | | | |
+|8827_braidwood|8927_ulladulla|9027_jervis_bay| | | | |
+===== Pre-processing data =====
+The following may be useful for Windows users.
+Below is a Windows Powershell script that will
+  * move any old DEMs and DEMs from a different zone (you can't mix zones in a virtual raster) to an archive sub-folder
+  * extract the raw data from the remaining zip files
+  * convert all of the .ASC files to GeoTIFF
+  * move the old zip files to a current sub-folder
+  * zip the current and archive sub-folders to temp.zip
+  * create a virtual raster (.vrt) file of all of the GeoTiffs
+You will need to replace the Environment variables with your own - lines starting with $Env.
+Usage is: buildvrt.ps1 <zipFile> <targetFolder>
+eg buildvrt.ps1 <zipFile> <targetFolder>
+<file powershell buildvrt.ps1>
+$zipFile=$args[0]
+$targetFolder=$args[1]
+# Unzip files from all subdirectories to new folder
+Expand-Archive -LiteralPath $zipFile -DestinationPath $targetFolder
+Get-ChildItem -Path "$targetFolder\*.zip" -Recurse | Move-Item -Destination $targetFolder
+Get-ChildItem -Path $targetFolder -Directory | Remove-Item -Recurse
+# Create hash of zip files, by name (location, date)
+$zipFileList = @{}
+Get-ChildItem -Path "$targetFolder\*.zip" -Name | ForEach-Object {
+	$zipFileList.add($_, @{})
+	$_ -match '(\d{7})_(\d{2})'
+	$zipFileList[$_]['location'] = $Matches.1
+	$zipFileList[$_]['zone'] = $Matches.2
+	$_ -match '[^_\d](\d{6})'
+	$zipFileList[$_]['date'] = $Matches.1
+}
+# $zipFileList | ConvertTo-Json
+# Create hash of location (date, name)
+$locationList = @{}
+$zoneCount = @{}
+$zipFileList.keys | ForEach-Object {
+	if($locationList[$zipFileList[$_]['location']]) {
+		$t = $locationList[$zipFileList[$_]['location']]
+		$t.add($zipFileList[$_]['date'],$_)
+	} else {
+		$t = @{}
+		$t.add($zipFileList[$_]['date'],$_)
+		$locationList.add($zipFileList[$_]['location'],$t)
+	}
+	if($zoneCount[$zipFileList[$_]['zone']]) {
+		$zoneCount[$zipFileList[$_]['zone']]++
+	} else {
+		$zoneCount[$zipFileList[$_]['zone']] = 1
+	}
+}
+# $locationList | ConvertTo-Json
+# $zoneCount | ConvertTo-Json
+# Create archive folder
+$archiveFolder = "$targetFolder\archive"
+If(!(test-path $archiveFolder))
+{
+      New-Item -ItemType Directory -Force -Path $archiveFolder
+}
+# Sort each location by date desc, and move old files to /archive
+$locationList.keys | ForEach-Object {
+	$i=0
+	$locationList[$_].GetEnumerator() | sort key -des | ForEach-Object {
+		if ($i -eq 0) {
+			$i++
+			return}
+		else {
+			#$_ | ConvertTo-Json
+			$s = $_.Value
+			Move-Item -Path "$targetFolder\$s" -Destination "$targetFolder\archive\$s"
+		}
+	}
+}
+# You can't build a VRT with files from a different projection, so
+# delete files from outside main zone
+# This could probably be altered to include a step to reproject those files
+$mainZone = ''
+$zoneCount.GetEnumerator() | sort value -des | select -first 1 | ForEach-Object {
+	$mainZone = $_.Name
+}
+$zipFileList.keys | ForEach-Object {
+	if($zipFileList[$_]['zone'] -ne $mainZone) {
+		Remove-Item -Path "$targetFolder\$_"
+	}
+}
+Get-ChildItem -Path "$targetFolder\*.zip" | Expand-Archive -DestinationPath $targetFolder
+$Env:Path += ";C:\OSGEO4~1\apps\Python27\Scripts;C:\OSGEO4~1\bin"
+$Env:GDAL_DATA="C:\OSGEO4~1\share\gdal"
+$Env:GDAL_DRIVER_PATH="C:\OSGEO4~1\bin\gdalplugins"
+$Env:PROJ_LIB="C:\OSGEO4~1\share\proj"
+Get-ChildItem -Path "$targetFolder\*.asc" | ForEach-Object {
+	$srcFile = $_.FullName
+	$destFile = $_.FullName -replace '.asc', '.tif'
+	&"gdal_translate.exe" -of "GTiff" $srcFile $destFile
+}
+Get-ChildItem -Path "$targetFolder\*.asc" | Remove-Item -Recurse
+Get-ChildItem -Path "$targetFolder\*.html" | Remove-Item -Recurse
+Get-ChildItem -Path "$targetFolder\*.prj" | Remove-Item -Recurse
+Get-ChildItem -Path "$targetFolder\*.xml" | Remove-Item -Recurse
+# Create current folder
+$currentFolder = "$targetFolder\current"
+If(!(test-path $currentFolder))
+{
+      New-Item -ItemType Directory -Force -Path $currentFolder
+}
+# Move zip files to current folder
+Move-Item -Path "$targetFolder\*.zip" -Destination "$currentFolder"
+# Zip /archive & /current to new zip folder
+Compress-Archive -Path "$targetFolder\current", "$targetFolder\archive" -DestinationPath "$targetFolder\temp.zip"
+# Create 2m vrt
+New-Item "$targetFolder\input.txt"
+Get-ChildItem -Path "$targetFolder\*.tif" | Add-Content "$targetFolder\input.txt"
+&"gdalbuildvrt.exe" -resolution lowest -input_file_list "$targetFolder\input.txt" "$targetFolder\temp.vrt"
+</file>
+It is possible to then build a larger virtual raster from the individual 1:100k virtual rasters.
+===== Loading data =====
+Loading up a large virtual raster into QGIS can be very slow, as can manipulating it. However, you can quickly load up a smaller section of the map using the following steps:
+  * create a polygon using https://maps.ozultimate.com/
+  * download the polygon using the **Export drawn data to GeoJSON** function
+  * load up the Python console (//Plugins->Python Console// or //Ctrl+Alt+P// on Windows)
+  * run the following command (replace file locations with your own)
+<code>
+resultClip = processing.runAndLoadResults("gdal:cliprasterbymasklayer", { 'ALPHA_BAND' : False, 'CROP_TO_CUTLINE' : True, 'DATA_TYPE' : 0, 'EXTRA' : '', 'INPUT' : 'E:/geodata/8930_katoomba/area.vrt', 'KEEP_RESOLUTION' : False, 'MASK' : 'C:/Users/Downloads/data.geojson', 'MULTITHREADING' : False, 'NODATA' : None, 'OPTIONS' : '', 'OUTPUT' : 'TEMPORARY_OUTPUT', 'SET_RESOLUTION' : False, 'SOURCE_CRS' : None, 'TARGET_CRS' : None, 'X_RESOLUTION' : None, 'Y_RESOLUTION' : None })
+</code>
 ====== Topographic maps ======
@@ Line 24: / Line 189: @@
 The NSW DEM data is supplied in 2km squares. The squares need to be merged into a single DEM for further operations.
-While this can be done in theory using a virtual raster, I have had poor performance with this. Any operation seems to result in screen redrawing, so moving around and zooming in and out is quite slow and painful.
+Most of the eastern ranges, where a lot of bushwalking happens, are 2m DEMs. The coast is typically 1m, and the western slopes and plains are 5m (with major rivers 1m!). Be careful if you need to merge DEMs with different resolutions - see below for more details.
+==== Virtual Raster ====
+While merging can be done in theory using a Virtual Raster (Raster- > Miscellaneous -> Build Virtual Raster... ), I have had poor performance with this (including recent version 3.12 Bucuresti). Any operation seems to result in screen redrawing, so moving around and zooming in and out is slow and painful.
+That said, if you are just using the Virtual Raster for future steps, then the limitations from the screen redrawing may not be important.
+QGIS uses [[https://gdal.org/programs/gdalbuildvrt.html|gdalbuildvrt]]. Uncheck "Place each input file into a separate band", and if you are using files with different resolutions (eg 1m and 2m), change the Resolution to Highest or Lowest as appropriate. You can also change the Resolution to User, and set the optional -tr switch (eg -tr 1 1 or -tr 2 2)
+==== Merge Raster ====
+I generally use the the Raster- > Miscellaneous -> Merge... function
-Instead, I generally use the the Raster- > Miscellaneous -> Merge... function
+QGIS uses [[https://gdal.org/programs/gdal_merge.html|gdal_merge]], which defaults to using the resolution of the first file. This is not always desired. It can be controlled by using the optional -ps (pixel size) switch. For example, if you have a combination of 1m and 2m DEMs, you can use -ps 1 1 to force them to a merged 1m DEM, or -ps 2 2 to force them to merge to a 2m DEM.
 ===== Fill Sinks =====
@@ Line 62: / Line 241: @@
 For more options in compression, look at:
   * GRASS : [[https://grasswiki.osgeo.org/wiki/V.generalize_tutorial|v.generalize]]
+V.generalize can also be used to smooth contours - possibly best done prior to simplificiation
 ==== Cleaning ====
@@ Line 112: / Line 292: @@
 {{:2019-02-08_12_41_50-channel_network.png?600|}}
-The raster channel network can then be classified and converted to vector.
+==== Classification ====
+For 1:25000 maps, I've had reasonable results from using the following formula in the Raster Calculator to classify the streams into categories. Different scales may need different bounds, and this doesn't account for significantly larger rivers.
+''( log10 ( "Catchment Area@1" ) >= x) * ( log10 ( "Catchment Area@1" ) < y) * ("Channel Network@1" != 0)''
+  * Intermittent: 4-6.15 (x-y)
+  * Minor: 6.15-7.4
+  * Major: 7.4+
+==== Convert to Vector and Simplify ====
+Convert to vector using r.to.vect
+{{:qgis_raw_stream.png?600|}}
+The raw stream data is very jagged. Smooth using
+  * v.generalize
+  * Algorithm = Hermite (there are other options which can be used, but Hermite has the smoothed line passing through the points of the original)
+  * Maximal tolerance value = 20 (in m, obviously scale dependent)
+Simplify using using:
+  * Vector geometry : Simplify
+Tolerance:?
 ===== Clifflines =====
-The steps below have been tested in the Blue Mountains, a region that has a significant number of relatively vertical sandstone cliffs. It may be less effective in different terrain.
+The steps below are being developed for use in the Blue Mountains, a region that has a significant number of relatively vertical sandstone cliffs. It may be less effective in different terrain.
+This is more a set of ideas than a fully fledged process. The main aims are to get a set of steps that can largely be automated, and that create cliffline vectors that are running in the correct direction. There is still some way to go on this!
 ==== Initial analysis of slope, aspect ====
@@ Line 126: / Line 332: @@
 using DEM and [1] Maximum Triangle Slope (Tarboton (1997)). I haven't tested any other algorithms.
-Cliff areas can be identified using a range of 60-90 and 70-90 degrees on the Slope file. Using 60-90 degrees helps connect logical cliffs and avoid small breaks.
+Cliff areas can be identified using a range of say 60-90 and 70-90 degrees on the Slope file. Using 60-90 degrees helps connect logical cliffs and avoid small breaks.
 ==== Initial Cleaning ====
 Next convert data to 1 bit (1,2 not 0,1, as Sieve ignores 0s) using Raster Calculator.
-Formula is: (Slope > 0) + 1
+Formula is: (Slope > 60) + 1
 Then Sieve resulting data using a Threshold of 100 and 8-connectedness to get rid of small non-connected cliffs. Note above that Sieve doesn't like 0s.
@@ Line 176: / Line 382: @@
 Run v.clean: Cleaning Tool = rmdangle, Threshold = 5,10
+===== Dumping Ground / WIP =====
+==== Resources ====
+[[https://grass.osgeo.org/grass78/manuals/addons/r.geomorphon.html|GRASS - r.geomorphon function information page]]. This is a different approach that could be taken for landform classification. Yet to be tested.
+[[https://docs.qgis.org/3.4/en/docs/training_manual/grass/grass_setup.html|Training lession for QGIS 3.4 on GRASS Setup and basic use]]. Specific GRASS setup is required to use any GRASS functions in QGIS.
+[[https://grasswiki.osgeo.org/wiki/Introduction_to_GRASS_GIS_with_terrain_analysis_examples|GRASS GIS example of terrain analysis using r.geomorphon]]
+[[http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.706.8909&rep=rep1&type=pdf|Geomorphons - a pattern recognition approach to classification and mapping of landforms]] paper.
+[[https://whiteboxgeospatial.wordpress.com/2014/12/25/a-new-form-of-multi-scale-topographic-position-visualization/|Multiscale topographic position - WhiteboxTools]] blog post.
+[[https://jblindsay.github.io/wbt_book/available_tools/geomorphometric_analysis.html#multiscaletopographicpositionimage|Multiscale topographic position image - WhiteboxTools function - user manual entry]]
+==== Method ====
+The below snip of Breakfast Creek makes use of TPI calculated from a LIDAR derived DEM. Only positive values for TPI as displayed, which indicate cliff-like features. It is then combined with contours and aerial imagery to convey the terrain of the area.
+{{:breakfast_creek_snip.jpg?400|TPI from LIDAR DEM combined with contours and imagery to show clifflines and terrain}}