The steps below are being developed for use in the Blue Mountains, a region that has a significant number of relatively vertical sandstone cliffs. It may be less effective in different terrain.

This is more a set of ideas than a fully fledged process. The main aims are to get a set of steps that can largely be automated, and that create cliffline vectors that are running in the correct direction. There is still some way to go on this!

SAGA → Terrain Analysis - Morphometry → Slope, Aspect, Curvature

Extract

Slope, Aspect 

using DEM and [1] Maximum Triangle Slope (Tarboton (1997)). I haven't tested any other algorithms.

Cliff areas can be identified using a range of say 60-90 and 70-90 degrees on the Slope file. Using 60-90 degrees helps connect logical cliffs and avoid small breaks.

Next convert data to 1 bit (1,2 not 0,1, as Sieve ignores 0s) using Raster Calculator. Formula is: (Slope > 60) + 1

Then Sieve resulting data using a Threshold of 100 and 8-connectedness to get rid of small non-connected cliffs. Note above that Sieve doesn't like 0s.

Also good to rerun Sieve with smaller Threshold (1-10) and 4-connectedness to a) get rid of some small dangles. b) fill small holes.

Additional smoothing can be done using a User Defined Filter with the following matrix. This will apply some smoothing by allowing you to reclassify the pixel values, and remove single pixel indentations like this:

000    000
101 -> 111
111    111

and single pixel protrusions like this:

000    000
010 -> 000
111    111

The main problem is that the matrix has to be defined each time in QGIS. There doesn't seem to be an option to load it. Possibly this can be done outside QGIS.

Matrix is:

0.0 0.5 0.0
0.5 0.5 0.5
0.0 0.5 0.0

If the original matrix is 0/1 then the cutoff will be 1.5

If the original matrix is 1/2 then the cutoff will be 3.5

This step could be run multiple times - some testing would need to be done to determine how many times.

Other options for cleaning the data include a plugin called LecoS, but this doesn't work on QGIS 3. Another possibility is Shrink and Expand - radius 1? But this also creates some new holes that didn't previously exist, so not ideal.

Convert back to 0/1 data using Raster Calculator

Use Translate: set Output Data Type = Byte, set NoData = 0

Run r.thin - r.thin is quite picky about the input file format. Needs to be NULL/non-NULL (not float or int). The Translate process above provides this. The previous two steps could be combined into one. Also, this file may need to be explicitly saved (not just a temporary file?!)

Run r.to.vect: set Feature Type = line

Run v.clean: Cleaning Tool = rmdangle, Threshold = 5,10