Differences

This shows you the differences between two versions of the page.

--- stitching_etopos [2023/07/08 09:55] – created bushwalking
+++ stitching_etopos [2023/08/31 08:55] (current) – Add link bushwalking
@@ Line 1: / Line 1: @@
 ====== Stitching e-Topos ======
-I recently went to considerable effort to stitch together NSW 1:25k e-Topo (PDF) maps, for use in OziExplorer. I figured it would be worthwhile to document the steps, with a bit of background. The same process may be useful for stitching other georeferenced maps with collars.
+I recently went to considerable effort to stitch together NSW 1:25k e-Topo (PDF) maps into one big raster, for use in OziExplorer. I figured it would be worthwhile to document the steps, with a bit of background. The same process may be useful for stitching other georeferenced maps with collars.
+Note that it is skewed towards Windows users, and assumes a certain level of technical knowledge (eg running commands on the command line), and that you have GDAL installed and the path to the binaries in your PATH.
+The following answer from GIS Stack Exchange has more details on resampling and compression options:
+https://gis.stackexchange.com/questions/464734/convert-geopdf-to-8-bit-geotiff-using-gdalwarp/464842#464842
+===== 0. Background on UTM and map dimensions =====
 First thing is, it's worth remembering that the UTM system slices the world up into a series of 6 degree zones, which are shaped like sections of an orange. These are flattened, and any grid is overlaid on this. For NSW (and many other places), individual maps are a fixed number of degrees wide. In the case of standard NSW 1:25k maps, they are 0.25 degrees wide. But 6 degrees at the equator is much wider than 6 degrees at 80 degrees south. Or, in the case of NSW, 0.25 degrees at Murwillumbah (24.5km) in the north of the state is wider than 0.25 degrees at Eden (22.2km) in the south.
@@ Line 13: / Line 20: @@
 The upshot of all of this is that maps are not nice rectangles that can just have the borders stripped off and dropped into a grid. However, there are some tools in the GDAL toolkit that will help us.
-First step is to strip off the border of the PDF and convert to a raster format. We can do this for a single image using gdalwarp, by setting a few options, including:
+===== 1. Download eTopos =====
-  * -t_srs <target spatial reference> - I've used EPSG:28356, which is MGA94 Zone 56 (eastern NSW). All maps need to be in the same co-ordinate system in order to merge them in the next step.
+These can be downloaded from [[https://portal.spatial.nsw.gov.au/portal/apps/webappviewer/index.html?id=06e3c2e0de1e4efda863854048c613c6|SIX Maps]]. Currently the 2017 versions.
+While there is a 5 map per day limit, this can be naughtily bypassed by grabbing a copy of the {{ :20170212_nsw_map_boundaries.xlsx |Excel spreadsheet}} on the [[downloads|Downloads]] page, which has direct links to all of the topos. You can drop a bunch of links into your favourite downloader, or just click on them one by one.
+While you're there, also grab the {{ :20160506_nsw_map_bounds.geojson |GeoJSON version of the NSW Map Boundaries}}, as you'll need a copy of it in the next stages.
+There is also a [[https://portal.spatial.nsw.gov.au/portal/apps/webappviewer/index.html?id=06e3c2e0de1e4efda863854048c613c6|newer site]] that has a range of versions from 2011 to 2022.
+===== 2. Remove borders =====
+Once you have the PDFs, the next step is to strip off the border of the PDF and convert to a raster format. We can do this for a single image using gdalwarp, by setting a few options, including:
+  * -t_srs <target spatial reference> - I've used EPSG:28356, which is MGA94 Zone 56 (eastern NSW). All maps need to be in the same co-ordinate system in order to merge them in the next step. This may not be needed(?)
   * -r <resampling_method> - cubic seems to be the recommended option given that we aren't trying to resize
   * -cutline - applies a mask to the raster (confusingly, this doesn't actually crop the image - you also need "crop_to_cutline")
@@ Line 20: / Line 38: @@
   * -crop_to_cutline - crops to the mask applied under "cutline"
   * -dstalpha - sets an output alpha band so that the pixels that are outside the cutline but inside the bounding box are transparent. This is needed when merging in the next step
+  * -co COMPRESS=LZW (optional) - will reduce the size of the intermediate GeoTiffs
 Sample command in full - you'll need to set your own directories, both for input and output file, and for the location of the NSW map boundaries GeoJSON:
@@ Line 36: / Line 55: @@
 </code>
-You now have a directory of GeoTiffs. In Windows, you can run the following at the command line to create a list of all the files in a text document:
+===== 3. Generate file list =====
+You now have a directory of GeoTiffs. In Windows, you can run the following at the command line to create a list of all the .TIF files in a text document:
 <code>dir /b /s "E:\data\maps\maps\nsw\*.tif" > "E:\data\maps\maps\nsw\tiff_list.txt"</code>
+Unfortunately, the NSW PDFs have spaces in the filenames by default, and the list actually needs all of the filenames quoted. The slightly more complicated command below handles this:
+<code>(for /f "delims=" %A in ('dir /b /s E:\data\maps\maps\nsw\*.tif') do @echo "%A") > "E:\data\maps\maps\nsw\tiff_list.txt"</code>
+===== 4. Merge files =====
 Finally, you can run gdal_merge, using the list from the previous step to create one huge GeoTiff. Note that normal GeoTiffs max out at 4GB, so you need to be selective about how many files you include in the merge. If you try to include too many, you will get errors, and a partially completed output. I got away with 56 1:25k maps, but the actual number will depend on various factors. There is a BIGTIFF=YES option that can be added to the -co option, but I don't know how this works in downstream processes/systems.
@@ Line 43: / Line 69: @@
 <code>gdal_merge -o "E:\data\maps\maps\nsw\sydney_basin.tif" -of GTiff -co COMPRESS=LZW --optfile "E:\data\maps\maps\nsw\tiff_list.txt"</code>
-For OziExplorer, the TIF can be used directly (use File->Import Map->Single DRG Map - and you will need the GeoTiff support library DLL), but it is faster and more space-efficient if converted to OZFX4 format. This can be done using Img2Ozf, downloadable from the OziExplorer website. Default settings are reasonable. As an example, a 4GB GeoTiff was converted to a 400MB OZFX4 file - a 90% reduction in size.
+The GeoTiff file can be used as is in many applications.
+There are other compression options (''-co COMPRESS=JPEG'', ''-co COMPRESS=WEBP'') that will result in a smaller file, but may limit where the file can subsequently be used.
+The ''-co tiled=YES'' option may be useful for improving the speed of the resulting GeoTiff.
+===== 5. Create OZFX4 file for OziExplorer =====
+For OziExplorer, the TIF can be used directly (use File->Import Map->Single DRG Map - and you will need the GeoTiff support library DLL), but it is faster and more space-efficient if converted to OZFX4 format. This can be done using [[https://www.oziexplorer4.com/img2ozf/img2ozf.html|Img2Ozf]], downloadable from the OziExplorer website. Default settings are reasonable. As an example, a 4GB GeoTiff was converted to a 400MB OZFX4 file - a 90% reduction in size.