Differences

This shows you the differences between two versions of the page.

--- nsw_dems [2021/05/04 18:25] – [Pre-processing data] bushwalking
+++ nsw_dems [2023/10/08 21:46] – bushwalking
@@ Line 27: / Line 27: @@
 |8827_braidwood|8927_ulladulla|9027_jervis_bay| | | | |
-===== Pre-processing data =====
-The following may be useful for Windows users.
-Below is a Windows Powershell script that will
-  * move any old DEMs and DEMs from a different zone (you can't mix zones in a virtual raster) to an //archive// sub-folder
-  * extract the raw files from the remaining zip files
-  * convert all of the .ASC files to GeoTIFF
-  * move the old zip files to a //current// sub-folder
-  * zip the //current// and //archive// sub-folders to //temp.zip//
-  * create a virtual raster (.vrt) file of all of the GeoTiffs
-You will need to replace the Environment variables (Path, GDAL_PATH, GDAL_DRIVER_PATH, PROJ_LIB) in the script with your own - see the lines starting with **$Env**.
-Usage is:
-<code>.\buildvrt.ps1 <zipFile> <targetFolder></code>
-eg <code>.\buildvrt.ps1 E:\geodata_raw\DATA_11279.zip E:\geodata\kiama</code>
-The .\ is required if you are running the script from your current directory.
-<file powershell buildvrt.ps1>
-$zipFile=$args[0]
-$targetFolder=$args[1]
-# Unzip files from all subdirectories to new folder
-Expand-Archive -LiteralPath $zipFile -DestinationPath $targetFolder
-Get-ChildItem -Path "$targetFolder\*.zip" -Recurse | Move-Item -Destination $targetFolder
-Get-ChildItem -Path $targetFolder -Directory | Remove-Item -Recurse
-# Create hash of zip files, by name (location, date)
-$zipFileList = @{}
-Get-ChildItem -Path "$targetFolder\*.zip" -Name | ForEach-Object {
-	$zipFileList.add($_, @{})
-	$_ -match '(\d{7})_(\d{2})'
-	$zipFileList[$_]['location'] = $Matches.1
-	$zipFileList[$_]['zone'] = $Matches.2
-	$_ -match '[^_\d](\d{6})'
-	$zipFileList[$_]['date'] = $Matches.1
-}
-# $zipFileList | ConvertTo-Json
-# Create hash of location (date, name)
-$locationList = @{}
-$zoneCount = @{}
-$zipFileList.keys | ForEach-Object {
-	if($locationList[$zipFileList[$_]['location']]) {
-		$t = $locationList[$zipFileList[$_]['location']]
-		$t.add($zipFileList[$_]['date'],$_)
-	} else {
-		$t = @{}
-		$t.add($zipFileList[$_]['date'],$_)
-		$locationList.add($zipFileList[$_]['location'],$t)
-	}
-	if($zoneCount[$zipFileList[$_]['zone']]) {
-		$zoneCount[$zipFileList[$_]['zone']]++
-	} else {
-		$zoneCount[$zipFileList[$_]['zone']] = 1
-	}
-}
-# $locationList | ConvertTo-Json
-# $zoneCount | ConvertTo-Json
-# Create archive folder
-$archiveFolder = "$targetFolder\archive"
-If(!(test-path $archiveFolder))
-{
-      New-Item -ItemType Directory -Force -Path $archiveFolder
-}
-# Sort each location by date desc, and move old files to /archive
-$locationList.keys | ForEach-Object {
-	$i=0
-	$locationList[$_].GetEnumerator() | sort key -des | ForEach-Object {
-		if ($i -eq 0) {
-			$i++
-			return}
-		else {
-			#$_ | ConvertTo-Json
-			$s = $_.Value
-			Move-Item -Path "$targetFolder\$s" -Destination "$targetFolder\archive\$s"
-		}
-	}
-}
-# You can't build a VRT with files from a different projection, so
-# delete files from outside main zone
-# This could probably be altered to include a step to reproject those files
-$mainZone = ''
-$zoneCount.GetEnumerator() | sort value -des | select -first 1 | ForEach-Object {
-	$mainZone = $_.Name
-}
-$zipFileList.keys | ForEach-Object {
-	if($zipFileList[$_]['zone'] -ne $mainZone) {
-		Remove-Item -Path "$targetFolder\$_"
-	}
-}
-Get-ChildItem -Path "$targetFolder\*.zip" | Expand-Archive -DestinationPath $targetFolder
-$Env:Path += ";C:\OSGEO4~1\apps\Python27\Scripts;C:\OSGEO4~1\bin"
-$Env:GDAL_DATA="C:\OSGEO4~1\share\gdal"
-$Env:GDAL_DRIVER_PATH="C:\OSGEO4~1\bin\gdalplugins"
-$Env:PROJ_LIB="C:\OSGEO4~1\share\proj"
-Get-ChildItem -Path "$targetFolder\*.asc" | ForEach-Object {
-	$srcFile = $_.FullName
-	$destFile = $_.FullName -replace '.asc', '.tif'
-	&"gdal_translate.exe" -of "GTiff" $srcFile $destFile
-}
-Get-ChildItem -Path "$targetFolder\*.asc" | Remove-Item -Recurse
-Get-ChildItem -Path "$targetFolder\*.html" | Remove-Item -Recurse
-Get-ChildItem -Path "$targetFolder\*.prj" | Remove-Item -Recurse
-Get-ChildItem -Path "$targetFolder\*.xml" | Remove-Item -Recurse
-# Create current folder
-$currentFolder = "$targetFolder\current"
-If(!(test-path $currentFolder))
-{
-      New-Item -ItemType Directory -Force -Path $currentFolder
-}
-# Move zip files to current folder
-Move-Item -Path "$targetFolder\*.zip" -Destination "$currentFolder"
-# Zip /archive & /current to new zip folder
-Compress-Archive -Path "$targetFolder\current", "$targetFolder\archive" -DestinationPath "$targetFolder\temp.zip"
-# Create 2m vrt
-New-Item "$targetFolder\input.txt"
-Get-ChildItem -Path "$targetFolder\*.tif" | Add-Content "$targetFolder\input.txt"
-&"gdalbuildvrt.exe" -resolution user -tr 2 2 -input_file_list "$targetFolder\input.txt" "$targetFolder\temp.vrt"
-</file>
-You can rename temp.vrt to anything you like, but it needs to stay relative to the GeoTIFF files.
-It is possible to then build a larger virtual raster from the individual 1:100k virtual rasters. Note that larger rasters need to have the same projection. Also keep in mind that not all data is at 2m resolution.
-===== Loading data =====
-Loading up a large virtual raster into [[https://www.qgis.org/en/site/index.html|QGIS]] can be very slow, as can manipulating it. However, you can quickly load up a smaller section of the map using the following steps:
-  * create the boundary of your desired area using the polygon tool at https://maps.ozultimate.com/
-  * download the polygon using the **Export drawn data to GeoJSON** function
-  * load up the Python console (//Plugins->Python Console// or //Ctrl+Alt+P// on Windows) in QGIS
-  * run the following command (replace file locations with your own - the INPUT should be your virtual raster, and the MASK should be your GeoJSON boundary polygon)
-<code>
-resultClip = processing.runAndLoadResults("gdal:cliprasterbymasklayer", { 'ALPHA_BAND' : False, 'CROP_TO_CUTLINE' : True, 'DATA_TYPE' : 0, 'EXTRA' : '', 'INPUT' : 'E:/geodata/8930_katoomba/area.vrt', 'KEEP_RESOLUTION' : False, 'MASK' : 'C:/Users/Downloads/data.geojson', 'MULTITHREADING' : False, 'NODATA' : None, 'OPTIONS' : '', 'OUTPUT' : 'TEMPORARY_OUTPUT', 'SET_RESOLUTION' : False, 'SOURCE_CRS' : None, 'TARGET_CRS' : None, 'X_RESOLUTION' : None, 'Y_RESOLUTION' : None })
-</code>