1


2

 1. Introduction
 2. Grids 1.0
 3. Example outputs
 4. Further work

3


4

 Projected from Lat/Lon 3 arc second resolution grid into a Lambert
Azimuthal Equal Area projection
 Ncols 49724
 Nrows 16373
 Cellsize 82.4544

5


6

 From 60° North to 60° South
 Ncols 432000
 Nrows 220800
 Cellsize 3 / 60 / 60 (3 arc seconds)

7

 SRTM DEM data is being produced at a 1 arc second resolution
 North America is available
 Soon after we will have higher resolution…
 We have 10 meter resolution DEM for the UK…
 Again hundreds of thousands of rows and columns
 Soon we will have more higher spatial resolution

8

 DesertLinks and tempQsim
 Need to focus on processing European  especially the Mediterranean
region of the European Union
 Hydrological DEMs
 Hydrological variables
 Flow accumulation
 Wetness index
 Geomorphometrics
 Physical indicators
 Salinity and Erosion risks
 Based on 3 arc second SRTM data
 We want something much better than HYDRO1K and ArcGIS9

9


10

 There are 4 packages:
 core
 exchange
 process
 utilities

11

 So what are geomorphometrics?
 Measures of the state and change in surface geometry of earth’s
physical horizons
 Topographic measures
 Terrain analysis aids
 Spatial variables
 Components of desertification indexes?

12

 Grids
 Basic unit is a cell
 A value attributed to a point of a lattice that covers a 2D plane
 double
 int
 Basic arrangement
 Chunks of cells arranged in rows and columns
 Chunk are rectangular and a fixed size for a grid

13

 No chunk may contain more than java.lang.Integer.MAX_VALUE (2147483647)
number of rows or columns
 There can also be no more than java.lang.Integer.MAX_VALUE number of rows
or columns of chunks
 Need further testing to specify exactly

14

 Advantages
 each chunk can be stored optimally using any of a number of data
structures; and,
 each chunk can be readily cached and reloaded as needs be
 Lightweight, but some overhead
 The usual practice is to use chunks that are relatively small but larger
than 64 cells

15

 64CellMap
 2DArray
 JAI
 Map
 RAF

16

 The most sophisticated data structure?
 Data stored in a fast, lightweight implementation of the java.util
Collections API
 gnu.trove
 TdoublelongHashMap
 TintlongHashMap
 The long is gives the mapping of the value to the 64 cells in the chunk

17

 for chunks that contain a single cell value there is a single mapping in
the HashMap
 for chunks with 64 different cell values there are 64 mappings in the
HashMap
 Iterating over (going through) the keys in the HashMap is necessary to
get and set cell values, so generally this works faster for smaller
numbers of mappings.

18

 A mapping of keys (cell values) and values (cell identifiers) is a
general way of storing grid data.
 Efficient in terms of memory use where a default value can be set, and
if there are only a small number of nondefault mappings in the chunk
(compared to the number of cells in the chunk)
 Offer the means to generating some statistics about a chunk very
efficiently
 the diversity (number of different values)
 mode

19

 Each chunk and grid is associated with a factory and an iterator
 Factories keep things tidy, production can be done in one place and an a
controlled way
 Iterators offer the fastest and most efficient way of going through all
the values in a grid or chunk
 These can be ordered and unordered

20

 Attached to every grid is a statistics object
 Implemented by every statistics object and every chunk and grid is a
statistics interface
 Abstract classes provide a generic way of returning statistics
 Specific chunks and grids can override these methods to provide faster
implementations

21

 Two basic types attached to a grid
 Updated
 Statistics initialised and kept up to date as underlying data changes
 Better if statistics are wanted
 Not updated
 Statistics not initialised or kept up to date as underlying data
changes
 Better is statistics are not wanted

22

 nonNoDataValueCountBigInteger
 number of cells with non noDataValues
 sumBigDecimal
 the sum of all non noDataValues
 minBigDecimal
 the minimum of all non noDataValues
 minCountBigInteger
 the number of min values as a BigInteger
 maxBigDecimal
 the maximum of all non noDataValues

23

 /**
 * This method does something
 * @param args an Object[]
containing the arguments needed for processing
 * @param handleOutOfMemoryError
If true then OutOfMemoryErrors are caught
 * in this method then caching
operations are initiated prior to retrying.
 * If false then OutOfMemoryErrors are
caught and thrown.
 */
 public Object[] methodToProcessl( Object[] args, boolean
handleOutOfMemoryError ) {
 try {
 return
anotherMethodToProcess( args, handleOutOfMemoryErrorFalse );
 } catch (
java.lang.OutOfMemoryError oome0 ) {
 if ( handleOutOfMemoryError
) {
 clearMemoryReserve();
 clearChunk( cacheChunk()
);
 initMemoryReserve();
 return methodToProcessl(
args, handleOutOfMemoryError );
 } else {
 throw oome0;
 }
 }
 }

24

 Want to avoid trying to free up memory by caching a chunk that is needed
 methods to stop this happening
 cacheChunksExcept( ChunkID[] chunks )
 When processing there can be multiple grids and it might be better to
cache coincident chunks, rather than all chunks in one grid then the
next etc…

25

 3 classes
 processor
 GWS extension
 DEM extension
 Logging
 Memory handling
 Workspace
 Methods
 addToGrid
 aggregation
 value replacement
 arithmetic operators
 rescaling

26

 Weighting
 Normalisation
 Cross scale
 2 main types:
 Univariate
 First order
 Second order
 moments (proportions, variance, skewness)
 Bivariate
 difference
 normalized difference
 correlation

27

 Methods
 Hollow or pit detection
 Hollow filling
 Flow accumulation
 Distributive
 Based on all downslope cells not just maximum
 Geomorphometrics
 E.g. Slope and aspect
 Regional based and weighted like GWS

28


29


30

 Slope is the gradient and aspect is the angle to a particular
orientation over a given region
 Circular region
 Vector addition
 Distance weighting
 Scales

31


32


33


34


35


36


37


38


39


40

 Differencing to focus on midscales can be done directly by applying a
more complex spatial weighting scheme
 Only relevant for some measures
 Not aspect
 What about upness and downness?

41


42


43


44

 Rotation invariant metrics
 Axis based metrics

45

 For each metric compare each orthogonal set within a region
 What do is observed?
 hhhh, hhhl, hhll, hlhl, hlll, llll
 Based on this observation we add to particular metrics
 Conditioners
 max, min, sum (and absolute variants)

46

 How to treat cells with the same height?
 How to treat cells with no data value?
 What conditioners are appropriate?
 Two types of weight
 Distance weighting
 Difference in elevation weighting
 What is best?

47


48


49


50


51

 Profile or Contour
 Concavity or Convexity
 Convergence or Divergence

52

 What other interesting metrics?
 minimum type profile concavity
 maximum type profile convexity
 Better for identifying features?
 Probably best in combination with the other metrics
 Sorry, no examples!

53

 Release Grids 1.0
 Organise the development of grids as an open source project
 Potentially integration with GeoTools
 Provide examples for the Mediterranean
 Get other users to play around and analyse output
 Compare functionality with respect to proprietary GIS outputs

54

