User Tools

Site Tools


nsw_dems

This is an old revision of the document!


Managing large DEMs

Downloading data

While data can be downloaded in an ad hoc manner, if you are regularly processing DEMs, it is better to have the DEM tiles already downloaded.

Download tiles by 1:100k map area, which is 0.5 x 0.5 degree squares. Each 1:100k map area ranges from around 2GB to 6GB of data, depending on the number of 2m DEMs vs 1m DEMs, and other factors.

The Sydney basin and Blue Mountains is around 50GB all up.

For example, to request data from ELVIS for the Katoomba 1:100k map area (bounded by -33.5,150,-34,150.5), see the screenshot below, and Select All DEMs:

For reference, below are the 1:100k map areas around Sydney:

8833_gulgong8933_merriwa9033_muswellbrook9133_camberwell9233_dungog9333_buladelah9433_forster
8832_mudgee8932_mt_pomany9032_howes_valley9132_cessnock9232_newcastle9332_port_stephens
8831_bathurst8931_wallerawang9031_st_albans9131_gosford9231_lake_macquarie
8830_oberon8930_katoomba9030_penrith9130_sydney
8829_taralga8929_burragorang9029_wollongong9129_port_hacking
8828_goulburn8928_moss_vale9028_kiama
8827_braidwood8927_ulladulla9027_jervis_bay

Pre-processing data

The following may be useful for Windows users.

Below is a Windows Powershell script that will

  • move any old DEMs and DEMs from a different zone (you can't mix zones in a virtual raster) to an archive sub-folder
  • extract the raw files from the remaining zip files
  • convert all of the .ASC files to GeoTIFF
  • move the old zip files to a current sub-folder
  • zip the current and archive sub-folders to temp.zip
  • create a virtual raster (.vrt) file of all of the GeoTiffs

You will need to replace the Environment variables (Path, GDAL_PATH, GDAL_DRIVER_PATH, PROJ_LIB) in the script with your own - see the lines starting with $Env.

Usage is:

.\buildvrt.ps1 <zipFile> <targetFolder>

eg

.\buildvrt.ps1 E:\geodata_raw\DATA_11279.zip E:\geodata\kiama

The .\ is required if you are running the script from your current directory.

buildvrt.ps1
$zipFile=$args[0]
$targetFolder=$args[1]
 
# Unzip files from all subdirectories to new folder
Expand-Archive -LiteralPath $zipFile -DestinationPath $targetFolder
Get-ChildItem -Path "$targetFolder\*.zip" -Recurse | Move-Item -Destination $targetFolder
Get-ChildItem -Path $targetFolder -Directory | Remove-Item -Recurse
 
# Create hash of zip files, by name (location, date)
$zipFileList = @{}
Get-ChildItem -Path "$targetFolder\*.zip" -Name | ForEach-Object {
	$zipFileList.add($_, @{}) 
	$_ -match '(\d{7})_(\d{2})'
	$zipFileList[$_]['location'] = $Matches.1
	$zipFileList[$_]['zone'] = $Matches.2
	$_ -match '[^_\d](\d{6})'
	$zipFileList[$_]['date'] = $Matches.1
}
# $zipFileList | ConvertTo-Json
 
# Create hash of location (date, name)
$locationList = @{}
$zoneCount = @{}
$zipFileList.keys | ForEach-Object {
	if($locationList[$zipFileList[$_]['location']]) {
		$t = $locationList[$zipFileList[$_]['location']]
		$t.add($zipFileList[$_]['date'],$_)
	} else {
		$t = @{}
		$t.add($zipFileList[$_]['date'],$_)
		$locationList.add($zipFileList[$_]['location'],$t)
	}
	if($zoneCount[$zipFileList[$_]['zone']]) {
		$zoneCount[$zipFileList[$_]['zone']]++
	} else {
		$zoneCount[$zipFileList[$_]['zone']] = 1
	}
}
# $locationList | ConvertTo-Json
# $zoneCount | ConvertTo-Json
 
# Create archive folder
$archiveFolder = "$targetFolder\archive"
If(!(test-path $archiveFolder))
{
      New-Item -ItemType Directory -Force -Path $archiveFolder
}
 
# Sort each location by date desc, and move old files to /archive
$locationList.keys | ForEach-Object {
	$i=0
	$locationList[$_].GetEnumerator() | sort key -des | ForEach-Object {
		if ($i -eq 0) {
			$i++
			return}
		else {
			#$_ | ConvertTo-Json
			$s = $_.Value
			Move-Item -Path "$targetFolder\$s" -Destination "$targetFolder\archive\$s"
		}
	}
}
 
# You can't build a VRT with files from a different projection, so
# delete files from outside main zone
# This could probably be altered to include a step to reproject those files
$mainZone = ''
$zoneCount.GetEnumerator() | sort value -des | select -first 1 | ForEach-Object {
	$mainZone = $_.Name
}
 
$zipFileList.keys | ForEach-Object {
	if($zipFileList[$_]['zone'] -ne $mainZone) {
		Remove-Item -Path "$targetFolder\$_"
	}
}
 
Get-ChildItem -Path "$targetFolder\*.zip" | Expand-Archive -DestinationPath $targetFolder
$Env:Path += ";C:\OSGEO4~1\apps\Python27\Scripts;C:\OSGEO4~1\bin"
$Env:GDAL_DATA="C:\OSGEO4~1\share\gdal"
$Env:GDAL_DRIVER_PATH="C:\OSGEO4~1\bin\gdalplugins"
$Env:PROJ_LIB="C:\OSGEO4~1\share\proj"
 
Get-ChildItem -Path "$targetFolder\*.asc" | ForEach-Object {
	$srcFile = $_.FullName
	$destFile = $_.FullName -replace '.asc', '.tif'
	&"gdal_translate.exe" -of "GTiff" $srcFile $destFile
}
 
Get-ChildItem -Path "$targetFolder\*.asc" | Remove-Item -Recurse
Get-ChildItem -Path "$targetFolder\*.html" | Remove-Item -Recurse
Get-ChildItem -Path "$targetFolder\*.prj" | Remove-Item -Recurse
Get-ChildItem -Path "$targetFolder\*.xml" | Remove-Item -Recurse
 
# Create current folder
$currentFolder = "$targetFolder\current"
If(!(test-path $currentFolder))
{
      New-Item -ItemType Directory -Force -Path $currentFolder
}
 
# Move zip files to current folder
Move-Item -Path "$targetFolder\*.zip" -Destination "$currentFolder"
 
# Zip /archive & /current to new zip folder
Compress-Archive -Path "$targetFolder\current", "$targetFolder\archive" -DestinationPath "$targetFolder\temp.zip"
 
# Create 2m vrt
New-Item "$targetFolder\input.txt"
Get-ChildItem -Path "$targetFolder\*.tif" | Add-Content "$targetFolder\input.txt"
&"gdalbuildvrt.exe" -resolution lowest -input_file_list "$targetFolder\input.txt" "$targetFolder\temp.vrt"

It is possible to then build a larger virtual raster from the individual 1:100k virtual rasters.

Loading data

Loading up a large virtual raster into QGIS can be very slow, as can manipulating it. However, you can quickly load up a smaller section of the map using the following steps:

  • create the boundary of your desired area using the polygon tool at https://maps.ozultimate.com/
  • download the polygon using the Export drawn data to GeoJSON function
  • load up the Python console (Plugins→Python Console or Ctrl+Alt+P on Windows) in QGIS
  • run the following command (replace file locations with your own - the INPUT should be your virtual raster, and the MASK should be your GeoJSON boundary polygon)
resultClip = processing.runAndLoadResults("gdal:cliprasterbymasklayer", { 'ALPHA_BAND' : False, 'CROP_TO_CUTLINE' : True, 'DATA_TYPE' : 0, 'EXTRA' : '', 'INPUT' : 'E:/geodata/8930_katoomba/area.vrt', 'KEEP_RESOLUTION' : False, 'MASK' : 'C:/Users/Downloads/data.geojson', 'MULTITHREADING' : False, 'NODATA' : None, 'OPTIONS' : '', 'OUTPUT' : 'TEMPORARY_OUTPUT', 'SET_RESOLUTION' : False, 'SOURCE_CRS' : None, 'TARGET_CRS' : None, 'X_RESOLUTION' : None, 'Y_RESOLUTION' : None })
nsw_dems.1612926232.txt.gz · Last modified: 2021/02/10 14:03 by bushwalking

Donate Powered by PHP Valid HTML5 Valid CSS Driven by DokuWiki