renderdem generates invalid bounds → OOM during odm_dem stage

[paultenfjord]

log_x86.txt
log_arm.txt
odm_356.txt

Affected latest ODM on git (682148b0bd64, 54543d0c15ef) - tested both x86 and ARM.

Using the Brighton dataset (git clone https://github.com/pierotofy/drone_dataset_brighton_beach.git)

Running:

docker run -it --rm \
    -v "$(pwd)/images:/code/images" \
    -v "$(pwd)/odm_orthophoto:/code/odm_orthophoto" \
    -v "$(pwd)/odm_texturing:/code/odm_texturing" \
    opendronemap/odm --dtm

Yields OOM after:

running renderdem "/code/odm_georeferencing/odm_georeferenced_model.laz" --outdir "/code/odm_dem" --output-type idw --radiuses

Cause:

Reading points from /code/odm_georeferencing/odm_georeferenced_model.laz
Number of points: 1466984
Classification dimension: Classification
Point cloud bounds are [minx: **1.79769313486e+308**, maxx: **2.22507385851e-308**,
                        miny: **1.79769313486e+308,** maxy: **2.22507385851e-308**]
DEM resolution is (**2147483648**, **2147483648**), max tile size is 4096,
will split DEM generation into 0 tiles
Killed

renderdem (invoked from odm_dem) incorrectly computes the point cloud bounds when filtering by classification. OOM a result of >4 trillion points, perhaps due to a classification filter returning an empty point cloud?

When using version 3.5.6 this is the relevant output:

Point cloud bounds are [minx: 576646.15, maxx: 576776.36,
                        miny: 5188108.22, maxy: 5188239.14]
DEM resolution is (2605, 2619), max tile size is 4096, will split DEM generation into 1 tiles

My host environment is (although probably not relevant for this bug):
macOS (M4, 16 GB RAM, Docker Desktop)

[paultenfjord]

Tried on different datasets. But reproduced on brighton dataset for bug report.

[smathermather]

Probably related to this:

[INFO]    Classifying /code/odm_georeferencing/odm_georeferenced_model.laz using Simple Morphological Filter (1/2)
[INFO]    running pdal translate -i /code/odm_georeferencing/odm_georeferenced_model.laz -o /code/odm_georeferencing/odm_georeferenced_model.laz smrf --filters.smrf.scalar=1.25 --filters.smrf.slope=0.15 --filters.smrf.threshold=0.5 --filters.smrf.window=18.0 --writers.las.forward=scale,offset
PDAL: Input and output filenames are equal and no --overwrite option was provided!

usage: pdal translate [options] input output [filter]
standard options:
  --developer-debug  Enable developer debug (don't trap exceptions)
  --label            A string to label the process with
  --driver           Override reader driver
  --help             Print help and exit
options:
  --input, -i     Input filename
  --output, -o    Output filename
  --filter, -f    Filter type
  --json          PDAL pipeline from which to extract filters.
  --pipeline, -p  Pipeline output
  --metadata, -m  Dump metadata output to the specified file
  --reader, -r    Reader type
  --writer, -w    Writer type
  --nostream      Run in standard mode
  --stream        Run in stream mode. Error if not possible.
  --dims          Dimensions to store
  --overwrite     Overwrite existing input

For more information, see the full documentation for PDAL at http://pdal.io/

[smathermather]

Partial fix (overwrite now works by throwing the --overwrite flag), but as a result, I've now exposed likely gdal / numpy build issue:

[INFO]    running gdaldem hillshade "/code/odm_dem/dtm.preview.tif" "/code/odm_dem/dtm.previewhillshade.tif" -z 1.0 -s 1.0 -az 315.0 -alt 45.0
0...10...20...30...40...50...60...70...80...90...100 - done.
[INFO]    running "/code/venv/bin/python3" "/code/opendm/tiles/hsv_merge.py" "/code/odm_dem/dtm.previewcolor.tif" "/code/odm_dem/dtm.previewhillshade.tif" "/code/odm_dem/dtm.previewcolored_hillshade.tif"
/code/venv/lib/python3.12/site-packages/osgeo/gdal.py:312: FutureWarning: Neither gdal.UseExceptions() nor gdal.DontUseExceptions() has been explicitly called. In GDAL 4.0, exceptions will be enabled by default.
  warnings.warn(
Traceback (most recent call last):
  File "/code/opendm/tiles/hsv_merge.py", line 198, in <module>
    rScanline = rBand.ReadAsArray(0, i, hillband.XSize, 1, hillband.XSize, 1)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/code/venv/lib/python3.12/site-packages/osgeo/gdal.py", line 5283, in ReadAsArray
    from osgeo import gdal_array
  File "/code/venv/lib/python3.12/site-packages/osgeo/gdal_array.py", line 10, in <module>
    from . import _gdal_array
ImportError: cannot import name '_gdal_array' from 'osgeo' (/code/venv/lib/python3.12/site-packages/osgeo/__init__.py)
[WARNING] Cannot generate colored hillshade: Child returned 1
[INFO]    running gdal_translate -outsize 1400 0 -of png "None" "/code/opensfm/stats/dtm.png" --config GDAL_CACHEMAX 47.4%
ERROR 4: None: No such file or directory

===== Dumping Info for Geeks (developers need this to fix bugs) =====
Child returned 1
Traceback (most recent call last):
  File "/code/stages/odm_app.py", line 82, in execute
    self.first_stage.run()
  File "/code/opendm/types.py", line 470, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 470, in run
    self.next_stage.run(outputs)
  File "/code/opendm/types.py", line 470, in run
    self.next_stage.run(outputs)
  [Previous line repeated 8 more times]
  File "/code/opendm/types.py", line 449, in run
    self.process(self.args, outputs)
  File "/code/stages/odm_report.py", line 214, in process
    system.run("gdal_translate -outsize {} 0 -of png \"{}\" \"{}\" --config GDAL_CACHEMAX {}%".format(image_target_size, colored_hillshade_dem, osfm_dem, get_max_memory()))
  File "/code/opendm/system.py", line 112, in run
    raise SubprocessException("Child returned {}".format(retcode), retcode)
opendm.system.SubprocessException: Child returned 1

@NathanMOlson -- if you get a chance to look, I think this is related to #1904 -- maybe numpy / gdal issues?

I'll take a look again early next week as well, but just wanted to test and then put a pin in the problem today upon seeing it.

Edit: I will be building an oats testing machine shortly so we can start to build some tests flags against real datasets.