Useful Pages
-
- Managing DEMs
- Topographic maps
- Styles
- Automation
- Mobile apps
While data can be downloaded in an ad hoc manner, if you are regularly processing DEMs, it is better to have the DEM tiles already downloaded.
Once downloaded, pre-process the tiles into GeoTIFFs, build a virtual raster for each 1:100k grid square. You can then build a virtual raster of virtual rasters!
Download tiles by 1:100k map area, which is 0.5 x 0.5 degree squares. Each 1:100k map area ranges from around 2GB to 6GB of data, depending on the number of 2m DEMs vs 1m DEMs, and other factors.
The Sydney basin and Blue Mountains is around 50GB all up.
For example, to request data from ELVIS for the Katoomba 1:100k map area (bounded by -33.5,150,-34,150.5), see the screenshot below, and Select All DEMs:
For reference, below are the 1:100k map areas around Sydney:
8833_gulgong | 8933_merriwa | 9033_muswellbrook | 9133_camberwell | 9233_dungog | 9333_buladelah | 9433_forster |
8832_mudgee | 8932_mt_pomany | 9032_howes_valley | 9132_cessnock | 9232_newcastle | 9332_port_stephens | |
8831_bathurst | 8931_wallerawang | 9031_st_albans | 9131_gosford | 9231_lake_macquarie | ||
8830_oberon | 8930_katoomba | 9030_penrith | 9130_sydney | |||
8829_taralga | 8929_burragorang | 9029_wollongong | 9129_port_hacking | |||
8828_goulburn | 8928_moss_vale | 9028_kiama | ||||
8827_braidwood | 8927_ulladulla | 9027_jervis_bay |
The following may be useful for Windows users.
Below is a Windows Powershell script that will
You will need to replace the Environment variables (Path, GDAL_PATH, GDAL_DRIVER_PATH, PROJ_LIB) in the script with your own - see the lines starting with $Env.
Usage is:
.\buildvrt.ps1 <zipFile> <targetFolder>
eg
.\buildvrt.ps1 E:\geodata_raw\DATA_11279.zip E:\geodata\kiama
The .\ is required if you are running the script from your current directory.
$zipFile=$args[0] $targetFolder=$args[1] # Unzip files from all subdirectories to new folder Expand-Archive -LiteralPath $zipFile -DestinationPath $targetFolder Get-ChildItem -Path "$targetFolder\*.zip" -Recurse | Move-Item -Destination $targetFolder Get-ChildItem -Path $targetFolder -Directory | Remove-Item -Recurse # Create hash of zip files, by name (location, date) $zipFileList = @{} Get-ChildItem -Path "$targetFolder\*.zip" -Name | ForEach-Object { $zipFileList.add($_, @{}) $_ -match '(\d{7})_(\d{2})' $zipFileList[$_]['location'] = $Matches.1 $zipFileList[$_]['zone'] = $Matches.2 $_ -match '[^_\d](\d{6})' $zipFileList[$_]['date'] = $Matches.1 } # $zipFileList | ConvertTo-Json # Create hash of location (date, name) $locationList = @{} $zoneCount = @{} $zipFileList.keys | ForEach-Object { if($locationList[$zipFileList[$_]['location']]) { $t = $locationList[$zipFileList[$_]['location']] $t.add($zipFileList[$_]['date'],$_) } else { $t = @{} $t.add($zipFileList[$_]['date'],$_) $locationList.add($zipFileList[$_]['location'],$t) } if($zoneCount[$zipFileList[$_]['zone']]) { $zoneCount[$zipFileList[$_]['zone']]++ } else { $zoneCount[$zipFileList[$_]['zone']] = 1 } } # $locationList | ConvertTo-Json # $zoneCount | ConvertTo-Json # Create archive folder $archiveFolder = "$targetFolder\archive" If(!(test-path $archiveFolder)) { New-Item -ItemType Directory -Force -Path $archiveFolder } # Sort each location by date desc, and move old files to /archive $locationList.keys | ForEach-Object { $i=0 $locationList[$_].GetEnumerator() | sort key -des | ForEach-Object { if ($i -eq 0) { $i++ return} else { #$_ | ConvertTo-Json $s = $_.Value Move-Item -Path "$targetFolder\$s" -Destination "$targetFolder\archive\$s" } } } # You can't build a VRT with files from a different projection, so # delete files from outside main zone # This could probably be altered to include a step to reproject those files $mainZone = '' $zoneCount.GetEnumerator() | sort value -des | select -first 1 | ForEach-Object { $mainZone = $_.Name } $zipFileList.keys | ForEach-Object { if($zipFileList[$_]['zone'] -ne $mainZone) { Remove-Item -Path "$targetFolder\$_" } } Get-ChildItem -Path "$targetFolder\*.zip" | Expand-Archive -DestinationPath $targetFolder $Env:Path += ";C:\OSGEO4~1\apps\Python27\Scripts;C:\OSGEO4~1\bin" $Env:GDAL_DATA="C:\OSGEO4~1\share\gdal" $Env:GDAL_DRIVER_PATH="C:\OSGEO4~1\bin\gdalplugins" $Env:PROJ_LIB="C:\OSGEO4~1\share\proj" Get-ChildItem -Path "$targetFolder\*.asc" | ForEach-Object { $srcFile = $_.FullName $destFile = $_.FullName -replace '.asc', '.tif' &"gdal_translate.exe" -of "GTiff" $srcFile $destFile } Get-ChildItem -Path "$targetFolder\*.asc" | Remove-Item -Recurse Get-ChildItem -Path "$targetFolder\*.html" | Remove-Item -Recurse Get-ChildItem -Path "$targetFolder\*.prj" | Remove-Item -Recurse Get-ChildItem -Path "$targetFolder\*.xml" | Remove-Item -Recurse # Create current folder $currentFolder = "$targetFolder\current" If(!(test-path $currentFolder)) { New-Item -ItemType Directory -Force -Path $currentFolder } # Move zip files to current folder Move-Item -Path "$targetFolder\*.zip" -Destination "$currentFolder" # Zip /archive & /current to new zip folder Compress-Archive -Path "$targetFolder\current", "$targetFolder\archive" -DestinationPath "$targetFolder\temp.zip" # Create 2m vrt New-Item "$targetFolder\input.txt" Get-ChildItem -Path "$targetFolder\*.tif" | Add-Content "$targetFolder\input.txt" &"gdalbuildvrt.exe" -resolution user -tr 2 2 -input_file_list "$targetFolder\input.txt" "$targetFolder\temp.vrt"
You can rename temp.vrt to anything you like, but it needs to stay relative to the GeoTIFF files.
It is possible to then build a larger virtual raster from the individual 1:100k virtual rasters. Note that larger rasters need to have the same projection. Also keep in mind that not all data is at 2m resolution.
Loading up a large virtual raster into QGIS can be very slow, as can manipulating it. However, you can quickly load up a smaller section of the map using the following steps:
resultClip = processing.runAndLoadResults("gdal:cliprasterbymasklayer", { 'ALPHA_BAND' : False, 'CROP_TO_CUTLINE' : True, 'DATA_TYPE' : 0, 'EXTRA' : '', 'INPUT' : 'E:/geodata/8930_katoomba/area.vrt', 'KEEP_RESOLUTION' : False, 'MASK' : 'C:/Users/Downloads/data.geojson', 'MULTITHREADING' : False, 'NODATA' : None, 'OPTIONS' : '', 'OUTPUT' : 'TEMPORARY_OUTPUT', 'SET_RESOLUTION' : False, 'SOURCE_CRS' : None, 'TARGET_CRS' : None, 'X_RESOLUTION' : None, 'Y_RESOLUTION' : None })