Skip to content

Latest commit

 

History

History
1036 lines (851 loc) · 68.7 KB

CHANGELOG.md

File metadata and controls

1036 lines (851 loc) · 68.7 KB
Raw

Changelog

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[Unreleased]

Added

  • GeoTrellisPath assumes file scheme when none provided #3191

Changed

  • Fix PolygonRasterizer failure on some inputs #3160
  • Fix GeoTiff Byte and UByte CellType conversions #3189
  • Fix incorrect parsing of authority in GeoTrellisPath #3191
  • GeoTrellisPath.zoomLevel is now Option[Int] -> Int to better indicate that it is a required parameter #3191
  • Fix the length of one degree at the equator in metres #3197
  • Fix Fix DelaunayRasterizer #3202
  • RasterSources resolutions should be consistent across implementations #3210
  • Bump gdal-warp-bindings version up to 1.0.0 #3211
  • Fixed GDALRasterSource.resample method behavior to respect the passed resampleMethod #3211
  • Fix Jackson dependencies #3212

Removed

  • Removed duplicate JSON codecs for TileLayout #3181
  • Removed private restriction for TiffTag read overload #3181

[3.2.0] - 2019-11-19

Changed

  • Preserve NODATA values for double cell types when resampling with Max or Min resampler #3144
  • Update dependency versions for Scala 2.12 cross build #3132
  • Fix eager evaluation of band min-max in GDALDataset #

[3.1.0] - 2019-10-25

Changed

  • Fix StreamingByteReader over-allocation when reading past EOF #3138
  • Tile, ArrayTile, ConstantTile, DelegatingTile, MultibandTile, MultibandArrayTile, RasterRegion, RasterSource, MosaicRasterSource converted from trait to abstract class #3136

[3.0.0] - 2019-10-18

Added

  • RasterSources API #3053
  • geotrellis.layer package to group functionality related to tiled layers.
    • SpatialKey, SpaceTimeKey, TemporalKey, Bounds, Boundable, EmptyBounds, KeyBounds, TemporalProjectedExtent, EmptyBoundsError, SpatialComponent, TemporalComponent, LayoutDefinition, LayoutScheme, LayoutLevel, LocalLayoutScheme, FloatingLayoutScheme, ZoomedLayoutScheme, MapKeyTransform, etc.
  • geotrellis.store package to group interfaces related to saving and reading tiled layers. Implementations save moved to:
    • geotrellis.store.accumulo
    • geotrellis.store.cassandra
    • geotrellis.store.hbase
    • geotrellis.store.hadoop
    • geotrellis.store.s3
  • geotrellis.vectortile.MVTFeature which properly conforms to the MVT 2.0 spec. Specifically, MVTFeature adds id: Option[Long] property.
  • RangeReader SPI produced an instance of the reader from URI #2998
  • geotrellis.raster.mapalgebra.focal.ZFactor to improve slope calculations #3014
  • A sparseStitch method on geotrellis.layer.stitch.SpatialTileLayoutCollectionStitchMethods. Note that SpatialTileLayoutCollectionStitchMethods now has the additional constraint geotrellis.raster.prototype.TilePrototypeMethods on type V. This should be transparent for users of the geotrellis.raster.Tile and geotrellis.raster.MultibandTile types #3017
  • geotrellis.util.np package which contains linspace and percentilemethods that match NumPy functionality. An implicit class was also added to geotrellis.raster which provides the percentile method for geotrellis.raster.Tile #3067
  • Conversion from raster to features of cells #3117
  • io.circe JSON encoders and decoders for GeoTiffInfo and related classes #3128

Changed

  • geotrellis.raster.summary.polygonal.[Multi]TilePolygonalSummaryHandler replaced with geotrellis.raster.summary.polygonal.PolygonalSummary. Users should expect to implement concrete subclasses of geotrellis.raster.summary.GridVisitor and pass those to the new polygonalSummary methods. There are a number of default implementations provided for simple operations in geotrellis.raster.summary.visitors
  • Polygonal summaries on raster RDDs of RDD[(SpatialKey, T <: Grid[Int])] with Metadata[TileLayerMetadata[SpatialKey]] can now be performed with far less boilerplate using the same visitor pattern as the new raster polygonal summary API. See RDDPolygonalSummary.scala for additional details.
  • geotrellis.raster.Grid.dimensions is now Dimensions[N] instead of (N, N) #3124
  • GeoTiffInfo moved out of GeoTiffReader into containing package #3128
  • Replaced use of internal Mergable trait with cats' Semigroup
  • The slope focal method now requires a ZFactor instance.
  • Use the AWS S3 SDK v2 instead of v1 #2911.
  • The implicit/package structure of the geotrellis.raster package has changed such that it's now possible to import almost all features/extensions with import geotrellis.raster._ #2891
  • geotrellis.pro4j.WKT object conversion methods to and from EPSG codes have changed. All now return Options rather than silently unwrap. Methods that convert from EPSG code to WKT string are now prefixed consistently with fromEpsg and methods that convert from WKT string to EPSG code are now prefixed consistently with toEpsg.
  • geotrellis.util.CRS.fromWKT now returns Option[CRS] instead of CRS after silently unwrapping an optional internally
    • The geotrellis.vectortile.{Layer, VectorTile} interfaces now uses MVTFeature instead of geotrellis.vector.Feature.
  • Change from scala-logging to log4s #3116
  • Update dependencies #2904.
  • Update dependencies #3053.
  • Bump ScalaPB version up with some API enhancements #2898.
  • Artifacts in Viewshed have been addressed, the pixels/meter calculation has also been improved #2917.
  • Fix map{X|Y}ToGrid function behavior that could give a bit incorrect results #2953.
  • Fix COG layer update bug related to COGLayerMetadata zoom ranges ordering #2922.
  • Use original ZoomRanges on COG layer update #2956.
  • ArrayTile equality will now check the cols, rows, and cellType of the two tiles #2991.
  • Fix incorrect Deflate compressor usage in GeoTiff reading #2997.
  • Refactor IO thread pool usage #3007.
  • S3RangeReader will now optionally read the HEADER of an object #3025.
  • FileRangeReaderProvider can now handle more types of URIs #3034.
  • Bump proj4 version to fix multiple performance issues #3039.
  • Fix HttpRangeReader swallows 404 error #3073
  • Add a ToSpatial function for the collections API #3082.
  • Fix TIFFTagsReader to skip unsupported tags #3088.
  • reprojectExtentAsPolygon should be more deterministic #3083.
  • AmazonS3URI.getKey now returns an empty string instead of null if no key was provided #3096.
  • Improved handling of null prefix for S3AttributeStore. Prefer use of S3AttributeStore.apply to take advantage of this improved handling #3096.
  • Fix Tiled TIFF BitCellType Segments conversion #3102
  • Updated computeResolution in IterativeCostDistance to match Iterative Viewshed #3106

Removed

  • Scala wrappers of JTS Geometry in geotrellis.vector #2932
  • geotrellis.etl package has been removed. The code has been archived at https://github.com/geotrellis/spark-etl #2969.
  • The S3Client wrapper that was used to support AWS SDK v1 has been removed in favor of directly using AWS SDK S3Client #2911.
  • geotrellis.s3-testkit has been removed; testing now depends on an external min.io server which mocks the S3 API. #2911.
  • geotrellis.slick has been removed. Slick support will now reside in geotrellis-contrib #2902.

[2.3.0] - 2019-04-18

Changed

  • Fix Accumulo and HBase AttributeStore performance (#2899).
    • Fix Cassandra AttributeStore performance (#2901).
    • Fix createAlignedGridExtent function actually to align to GridExtents (#2878).
    • Fix to rasterRegionReproject function (#2880).

[2.2.0] - 2019-01-11

Changed

  • Extract Proj4J to org.locationtech.proj4j (#2846).
  • Fix {Backend}LayerCopy exception handling (#2860).
  • Fix cache level in Ingest objects (#2854).
  • Catch a possible precision loss on reprojection to the same projection (#2849).
  • Make histogram.merge methods more specific (#2852).
  • Fixed elliptical radius check in IDW algorithm (#2844).
  • Replace all JavaConversions imports by JavaConverters (#2843).
  • Made implicit conversions to/from Raster and ProjectedRaster deprecated (#2834).
  • Fix vectorizer bug (#2839).
  • Added mapTile to ProjectedRaster (#2830).
  • Downgrade Avro version (#2837).
  • Fix error in ReprojectRasterExtent (#2825).
  • Dependency updates (#2813).
  • Bump Spark version up to 2.4 (#2823).
  • Use a local[2] mode inside TestEnvironment trait (#2821).
  • Make IO code agnostic about effects (#2818).
  • Make IO code agnostic about effects (#2818).

[2.1.0] - 2018-10-02

Added

  • geotrellis.spark.etl.TemporalIngest main method (#2709).
  • Add bbox field to all GeoJSON Features (#2811).

Changed

  • TileRDDReproject now works on RDD of TileFeature[T, D] (#2803).
  • TileRDDReproject now uses Reproject.Options.errorThreshold value (#2803).
  • geotrellis.spark.testkit.TestEnvironment now includes SparkSession (#2808).
  • geotrellis.raster.RasterRegionReproject defaults to approximate resampling for MultibandBandTile (#2803).
  • Stitcher type class instance is now defined for TileFeature[T, D] (#2803).
  • Fix GeoTiffSegment conversion to UByteCellType and UShortCellType (#2794).
  • Fix GeoTiff[T] interpretation of AutoHigherResolution OverviewStrategy (#2802).
  • GeoTiff[T].getClosestOverview method is now public (#2804).
  • GeoTiffOptions.storageMethod now defaults to Tiled instead of Striped (#2810).
  • GeoTiff TiffTags class can now read inverted extents (#2812).

[2.0.0] 2019-01-11

Added

  • All focal operations now except an optional partitioner parameter.
  • BufferTiles.apply methods and the bufferTiles methods now except an optional partitioner parameter.
  • CollectionLayerReader now has an SPI interface.
  • ZoomResample can now be used on MultibandTileLayerRDD.
  • Partitioner can be specified in the reproject methods of TileLayerRDD.
  • Compression level of GeoTiffs can be specified in the DeflateCompression constructor.
  • resampleMethod parameter has been added to COGLayerWriter.Options.
  • A new type called LayerType has been created to help identify the nature of a layer (either Avro or COG).
  • LayerHeader now have an additional parameter: layerType.
  • AttributeStore now has four new methods: layerType, isCOGLayer, readCOGLayerAttributes, and writeCOGLayerAttributes.
  • Kryo serialization of geometry now uses a binary format to reduce shuffle block size.
  • Alter geotrellis.spark.stitch.StitchRDDMethods to allow RDD[(K, V)] to be stitched when not all tiles are of the same dimension.
  • Introduce Pyramid class to provide a convenience wrapper for building raster pyramids.
  • Both COGValueReader and OverzoomingCOGValueReader now have the readSubsetBands method which allows users to read in a select number of bands in any order they choose.
  • COGLayerReader now has the readSubsetBands and querySubsetBands methods which allow users to read in layers with the desired bands in the order they choose.
  • KeyBounds now has the rekey method that will rekey the bounds from a source layout to a target layout.
  • Kryo serialization of geometry now uses a binary format to reduce shuffle block size.
  • GeoTiffMultibandTile now has another crop method that takes a GridBounds and an Array[Int] that represents the band indices.
  • MultibandTile now has a new method, cropBands that takes an Array of band indices and returns a cropped MultibandTile with the chosen bands.
  • GeoTiff[MultibandTile] can be written with BandInterleave, only PixelInterleave previously supported. (#2767)
  • Input.partitionBytes and it is set to 134217728 by default to correspond HadoopGeoTiffRDD default behaviour.
  • Output.bufferSize option to set up a custom buffer size for the buffered reprojection.

Changed

  • The length of the key (the space-filling curve index or address) used for layer reading and writing has been extended from a fixed length of 8 bytes to an arbitrary length. This change affects not only the geotrellis.spark package, but all backends (excluding geotrellis.geowave and geotrellis.geomesa).
  • Reprojection has improved performance due to one less shuffle stage and lower memory usage. TileRDDReproject loses dependency on TileReprojectMethods in favor of RasterRegionReproject
  • The Ascii draw methods are now method extensions of Tile.
  • Replace geotrellis.util.Functor with cats.Functor.
  • Specifying the maxTileSize for a COGLayer that's to be written is now done via COGLayerWriter.Options which can be passed directly to the write methods.
  • Specifying the compression for a COGLayer that's to be written is now done via COGLayerWriter.Options which can be passed directly to the write methods.
  • The attribute name for COGLayerStorageMetadata is now metadata instead of cog_metadata.
  • Scalaz streams were replaced by fs2 streams.
  • Refactored HBaseInstance, now accepts a plain Hadoop Configuration object.
  • Refactored CassandraInstance, now accepts a getCluster function.
  • Use pureconfig to handle all work with configuration files.
  • Change TilerMethods.tileToLayout functions that accept TileLayerMetadata as an argument to return RDD[(K, V)] with Metadata[M] instead of RDD[(K, V)].
  • Expose attributeStore parameter to LayerReader interface.
  • Added exponential backoffs in S3RDDReader.
  • Changed SinglebandGeoTiff and MultibandGeoTiff crop function behavior to work properly with cases when extent to crop by doesn't intersect tiff extent.
  • All classes and objects in the geowave package now use the spelling: GeoWave in their names.
  • TileLayerMetadata.fromRdd method has been renamed to TileLayerMetadata.fromRDD.
  • KeyBounds.fromRdd method has been renamed to KeyBounds.fromRDD.
  • Removed implicit conversion from Raster[T] to T (#2771).
  • Removed decompress option from GeoTiffReader functions.
  • geotrellis.spark-etl package is deprecated since GeoTrellis 2.0.
  • Input.maxTileSize is 256 by default to correspond HadoopGeoTiffRDD default behavior.
  • geotrellis.slick.Projected has been moved to geotrellis.vector.Projected
  • StreamingHistogram.binCount now returns non-zero counts (#2590)
  • HilbertSpatialKeyIndex index offset. Existing spatial layers using Hilbert index will need to be updated (#2586)
  • Fixed CastException that sometimes occurred when reading cached attributes.
  • Uncompressed GeoTiffMultibandTiles will now convert to the correct CellType.
  • Calculating the Slope of a Tile when targetCell is Data will now produce the correct result.
  • Introduce new hooks into AttributeStore trait to allow for better performance in certain queries against catalogs with many layers.
  • GeoTiffReader can now read tiffs that are missing the NewSubfileType tag.
  • Pyramiding code will once again respect resampling method and will now actually reduce shuffle volume by resampling tiles on map side of pyramid operation.
  • COGLayer attributes can be accessed via the various read attribute methods in AttributeStore (ie readMetadata, readHeader, etc)
  • The regex used to match files for the HadoopLayerAttributeStore and FileLayerAttributeStore has been expanded to include more characters.
  • HadoopAttributeStore.availableAttributes has been fixed so that it'll now list all attribute files.
  • Allow for simple features to be generated with a specified or random id with geometry stored in the standard field, the_geom
  • Update version of Amazon SDK API to remove deprecation warnings.
  • Fixed a bug in incorrect metadata fetch by COGLayerReader that could lead to an incorrect data querying.
  • Cropping RDDs with clamp=false now produces correct result.
  • Fixed tiff reads in case RowsPerStrip tiff tag is not defined.
  • Change aspect result to azimuth, i.e. start from due north and be clockwise.
  • COG overviews generated in the COGLayer.fromLayerRDD method will now use the passed in ResampleMethod.
  • Reading a GeoTiff with streaming will now work with files that are larger than java.lang.Integer.MAX_VALUE.
  • GeoTiffMultibandTile.crop will now work with GeoTiffs that have tiled segments and band interleave.
  • GeoTiffMultibandTile.crop will now return ArrayMultibandTile(s) with the correct number of bands.
  • Improved performance of COGValueReader.readSubsetBands when reading from S3.

Removed

  • LayerUpdater with its functionality covered by LayerWriter (#2663).

[1.2.1] - 2018-01-03

Changed

[1.2.0] - 2017-11-06

This release cycle saw a regular contributor Simeon Fitch elevated to official Committer status within GeoTrellis.

The team would like to thank him, along with our newest contributors Aaron Santos, Austin Heyne, Christos Charmatzis, Jessica Austin, and @mteldridge for helping make this release possible.

Changed

  • geotrellis.raster
    • Deprecation: GridBounds.size in favor of GridBounds.sizeLong.
    • Deprecation: GridBounds.coords in favor of GridBounds.coordsIter.
    • New: GridBounds.offset and GridBounds.buffer for creating a modified GridBounds from an existing one.
    • New: ColorRamps.greyscale: Int => ColorRamp, which will generate a ramp when given some number of stops.
    • New: ConstantTile.fromBytes to create any type of ConstantTile from an Array[Byte].
    • New: Tile.rotate90: Int => Tile, Tile.flipVertical: Tile and Tile.flipHorizontal: Tile.
  • geotrellis.vector
    • New: Geometry.isEmpty: Boolean. This incurs much less overhead than previous ways of determining emptiness.
    • New: Line.head and Line.last for efficiently grabbing the first or last Point in the Line.
  • geotrellis.spark
    • Deprecation: The LayerUpdater trait hierarchy. Use LayerWriter.update or LayerWriter.overwrite instead.
    • Deprecation: Every cache provided by geotrellis.spark.util.cache. These will be removed in favor of a pluggable cache in 2.0.
    • New: SpatialKey.extent: LayoutDefinition => Extent
    • New: ValueReader.attributeStore: AttributeStore
    • New: TileLayerRDD.toSpatialReduce: ((V, V) => V) => TileLayerRDD[SpatialKey] for smarter folding of 3D tile layers into 2D tile layers.
    • The often-used apply method overloads in MapKeyTransform have been given more descriptive aliases.
    • Change: Querying a layer will now produce a result whose metadata will have an Extent and KeyBounds of the queried region and not of the whole layer.
  • geotrellis.vectortile (experimental)
    • New: VectorTile.toGeoJson and VectorTile.toIterable.
    • Library simplified by assuming the codec backend will always be protobuf.

Added

Rasterizing Geometry Layers

Finally, the full marriage of the vector, raster, and spark packages! You can now transform an RDD[Geometry] into a writable GeoTrellis layer of (SpatialKey, Tile)!

val geoms: RDD[Geometry] = ...
val celltype: CellType = ...
val layout: LayoutDefinition = ...
val value: Double = ...  /* Value to fill the intersecting pixels with */

val layer: RDD[(SpatialKey, Tile)] with Metadata[LayoutDefinition] =
  geoms.rasterize(value, celltype, layout)

Clipping Geometry Layers to a Grid

In a similar vein to the above, you can now transform an arbitrarily large collection of Geometries into a proper GeoTrellis layer, where the sections of each Geometry are clipped to fit inside their enclosing Extents.

Clipping Geometry to Grid

Here we can see a large Line being clipped into nine sublines. It's one method call:

import geotrellis.spark._

val layout: LayoutDefinition = ...  /* The definition of your grid */
val geoms: RDD[Geometry] = ...      /* Result of some previous work */

/* There are likely many clipped Geometries per SpatialKey... */
val layer: RDD[(SpatialKey, Geometry)] = geoms.clipToGrid(layout)

/* ... so we can group them! */
val grouped: RDD[(SpatialKey, Iterable[Geometry])] = layer.groupByKey

If clipping on the Extent boundaries is not what you want, there are ways to customize this. See the ClipToGrid entry in our Scaladocs.

Sparkified Viewshed

A Viewshed shows "visibility" from some set vantage point, given an Elevation raster. Prior to GeoTrellis 1.2 this was possible at the individual Tile level but not the Layer (RDD) level. Now it is.

First, we need to think about the Viewpoint type:

import geotrellis.spark.viewshed._

val point: Viewpoint(
  x = ...,                    // some coordinate.
  y = ...,                    // some coordinate.
  viewHeight = 4000,          // 4 kilometres above the surface.
  angle = Math.PI / 2,        // direction that the "camera" faces (in radians). 0 == east.
  fieldOfView = Math.PI / 2,  // angular width of the "view port".
  altitude = 0                // the height of points you're interested in seeing.
)

In other words:

  • x, y, viewHeight: where are we?
    • angle: what direction are we looking?
    • fieldOfView: how wide are we looking?
    • altitude: how high/low is the "target" of our viewing?

Given a Seq[Viewpoint] (the algorithm supports multiple simultaneous view points), we can do:

// Recall this common alias:
//   type TileLayerRDD[K] = RDD[(K, Tile)] with Metadata[TileLayerMetadata[K]]

val layer: TileLayerRDD[SpatialKey] = ...  /* Result of previous work */

val viewshed: TileLayerRDD[SpatialKey] = layer.viewshed(Seq(point))

Sparkified Euclidean Distance

We use Euclidean Distance to render a collection of points into a heat-map of proximities of some area. Say, of two roads crossing:

Euclidean Distance Diagram

Prior to GeoTrellis 1.2, this was possible at the individual Tile level but not the Layer (RDD) level. Now it is.

/* Result of previous work. Potentially millions of points per SpatialKey. */
val points: RDD[(SpatialKey, Array[Coordinate])] = ...
val layout: LayoutDefinition = ...  /* The definition of your grid */

val layer: RDD[(SpatialKey, Tile)] = points.euclideanDistance(layout)

Polygonal Summaries over Time

The following was possible prior to GeoTrellis 1.2:

val layer: TileLayerRDD[SpatialKey] = ...
val polygon: Polgyon = ...

/* The maximum value within some Polygon overlaid on a Tile layer */
val summary: Double = layer.polygonalMaxDouble(polygon)

The above is also now possible for layers keyed by SpaceTimeKey to form a "time series":

val layer: TileLayerRDD[SpaceTimeKey] = ...
val polygon: MultiPolygon = ...

/* The maximum value within some Polygonal area at each time slice */
val summary: Map[ZonedDateTime, Double] = layer.maxSeries(polygon)

OverzoomingValueReader

A GeoTrellis ValueReader connects to some layer catalog and lets you read individual values (usually Tiles):

import geotrellis.spark.io.s3._

val store: AttributeStore = ...
val reader: Reader[SpatialKey, Tile] = S3ValueReader(store).reader(LayerId("my-catalog", 10))

val tile: Tile = reader.read(SpatialKey(10, 10))

However .reader is limited to zoom levels that actually exist for the given layer. Now you can use .overzoomingReader to go as deep as you like:

import geotrellis.raster.resample._

val reader: Reader[SpatialKey, Tile] =
  S3ValueReader(store).overzoomingReader(LayerId("my-catalog", 20), Average)

val tile: Tile = reader.read(SpatialKey(1000, 1000))

Regridding a Tile Layer

Have you ever wanted to "redraw" a grid over an established GeoTrellis layer? Say, this 16-tile Layer into a 4-tile one, both of 1024x1024 total pixels:

Regrid Diagram

Prior to GeoTrellis 1.2, there was no official way to do this. Now you can use .regrid:

/* The result of some previous work. Say each Tile is 256x256. */
val layer: TileLayerRDD[SpatialKey] = ...

/* "Recut" the tiles so that each one is now 512x512.
 * No pixels are gained or lost, save some NODATA on the bottom
 * and right edges that may appear for padding purposes.
 */
val regridded: TileLayerRDD[SpatialKey] = layer.regrid(512)

You can also regrid to non-rectangular sizes:

val regridded: TileLayerRDD[SpatialKey] = layer.regrid(tileCols = 100, tileRows = 300)

Robust Layer Querying

It's common to find a subset of Tiles in a layer that are touched by some given Polygon:

val poly: Polygon = ???

val rdd: TileLayerRDD[SpatialKey] =
 layerReader
    .query[SpatialKey, Tile, TileLayerMetadata[SpatialKey]](Layer("name", zoom))
    .where(Intersects(poly))
    .result

Now you can perform this same operation with Line, MultiLine, and even (Polygon, CRS) to ensure that your Layer and Geometry always exist in the same projection.

Improved Tile ASCII Art

Sometimes you just want to visualize a Tile without going through the song-and-dance of rendering it to a .png. The existing Tile.asciiDraw method kind of does that, except its output is all in numbers.

The new Tile.renderAscii: Palette => String method fulfills your heart's desire:

import geotrellis.raster._
import geotrellis.raster.io.geotiff._
import geotrellis.raster.render.ascii._

val tile: Tile = SinglebandGeoTiff("path/to/tiff.tiff").tile

// println(tile.renderAscii())  // the default
println(tile.renderAscii(AsciiArtEncoder.Palette.STIPLED))
            ▚▖
            ▚▚▜▚▚
            ▚▖▚▜▚▖▚▚
           ▜▚▚▚▜▚█▚▜▚█▚
           █▚▜▖▜▖▚▚█▚▚▜▚█▖
           ▚▚█▚▜▚▚▚▚▚▚▚▜▚▚▚▚▚
          ▚▚▖▚▚▚▚▚█▜▚▚▜▚▚▖▚▖▚▖▚
          ▚▚▚▚█▚▚▚▚▚██▚▚▚▜▖▖██▚▚▜▚
          ▚▚█▚▚▚▚▚▚▚▜▚▚▚▚▚▚▜▚█▚▚▚▚▚▚▚
         █▚▚▖▚█▚▜▚▚▚▚▖▚▚▚▚▚▚▚▚▚▚▜▚▚▚▚▚▚▖
         █▚▚▚▜▚▖▚▚▚▚▚▚▚▚▚▚▚▚▚▚▚▚▚██▖▜▚█▚▚▚
         █▚▚██▚▚▚▚▚▚▚▚▖▚▚▚▚▚▚▚▚█▚▚▚▚▚▚▖▖▖▚▚▚▚
        █▜▚▚██▜▚▚▚▜▖▚▚▜▚█▜▚▚▚▜▚▖▚▜▚█▚▚▖▚▚▖▚▚▖▖▚▚
        ▚▚█▚▚▚█▚██▚▚▚▚▚▚▚▚▜▚▚█▜▚▖█▚▚▚▜▚▚▚▚▚▚▜▚█▚█
        █▚▜▚▜▚█▚▜▚▚▜▚█▚▚▚▚▚▚▚▚▚▚▚▖▚▖▚▚▖▚█▚█▚▚▚▖█▚
        ████▚███▚▚▚▚██▚▚▚█▜▚▚▖▚▚▚▖▖▚▚▚▚▚▚▚▚█▚▜▖█
       ▖█▜▚█▚██▜▖▜▜█▜▜█▜▚▚▚▚▚█▖▚▚▚▚█▚▚▚▚▚▚▜▚▚█▖▜
       ▚▖██▚▜▚█▚▚▜▜█▜▜▜██▚▚▚▚█▚▚▚▜▖▚▚█▚▖▚▜▚▚▚▖▚█
       █▚▚▚▚▜▚██▖██▜▚▚█▚▚▖▚▚▜▚▖▚▖▚▚▚▚▚▖▚▚▖▖▖▚▖▚
      ▚▚▚█▚▚▚▚▚█▜▚▚████▚█▚▚▚█▚▖▚▚▚▖▚▚█▚▚▖▚▚▚▖▖▖
      ▚▚▚█▚▚▚▖▖▚▜█▜██▜██▚▚▖██▜▚▜▚█▚▚▚▚▚▚▚▚▖▖▜██
      ▚▚▚▚▜█▚▚▚▚▚█████▚▜██▚██▚▚▚▚▜▚▖▚█▚▚▖▚▖▚▚█
     ▚▚▜▚▚▚▚▜▚▜▚▚▚▚▜▚█▚▜█▚██▚██▚▚▚▚▖▚▚▚▚▖▖▚▚▖█
     ▚▜▚▜▚▚▚▚▚▚█▚▚▚▚▚██▜▜▜███▖▚▚▜█▚▚▖▚█▚▚█▚▖▚
     ▚▜▚▚▚▚▚▚▚▚▚▚▜▜▜▚▚▖▚▖▚▚▜▜██▜▚██▚▚▚▚▚▚▖▜█▚
    ▚▚▖▚▚█▚█▚▚▚█▚▖▚▚▚█▚▚▚▚▚▜██▚█▜▚█▚▜▚▚███▜█▜
    ▚▚▚▜▚▚▚▚▚▚▚▚▚▚▚▖█▚█▚▚▜█▜█▜█▜▚▖▚▚▚██▜▜█▚▜
    ▚▚▚▚▜▚▚▚▚▚▚▜▚▚▚▚▚▚▖▚█▜▖▖█▚▖▜▖▚▖█▚▖█▚▚▜▚█
    ▚▚█▚▚█▚▚▜▚▚▚▚▜▚▚▚▚▚▜▚▖▚█▜█▚▜▜▚█▖▜███▜▚▚
   ▚▚▚▚▚▚▖▜▚█▚▚▚▖▚▚▚▚▚▚▚▚▚▚▚▜█▖▜▜▜█▚▚▚▖▚█▚█
   ▜▚▚▚█▚▖▚█▚█▚▚█▚▚▚▚▚▚▚▖▚▚▚▜▚▚▚▜▚▖▚▖▚▚▚▚▜▚
   ▚▚▚▚▖▚█▖█▜▚▚▚▚▚▚▚▚▖▚▚▖▖█▚▜▚▖▚▚▚▚▖▖▚█▚▚▚
  ▚▚▚▚▚▚▚▚▚█▚▚▚▖▚▚▚█▚▜▚█▚▚▖▜██▚▖▚▚▚▚▚▚▚▚▚▖
  ▚▚▚▚▚▚▚▖▚▚██▚▚▚▚▚▚▚▚▜▚▚█▚██▚▚▚▚▖▚▚▖▚▚█▜▖
  ▚▚▚▚▚▚▚▚▚▚▚▚▚█▚▜▚▚▚▜▚▚▖▚▚▚▚▚▜▚▚▚▚▖▚▚▚▚▚
 ▚██▖▚▚▚▚▚▚▚▚▜▚▚█▚▚▚▚▜▚▚▚▚█▜▖▚▚█▜▜█▜█▚▖▚▖
 ▚▚▚▖▚▚█▚▚▜███▚▚▚▜▚▚▚▚▚█▚▖▖█▖▚████▜███▚██
 ▚█▚▚▚▚██▜▚▜▚▜▜▜█▜▚█▚▜▖▜▚▚▚█▚▜█▚▜▚▚▚▚▚▖▖
    █▜█▚▚▜▚▜▚▜▜▜▚▚▚▚██▖▖▖▚██▖█▚▜▜▚▚▚▚▚▚▖
       ▚█▜▜▜▜▜██▚▜▚▚▚▚▚▚▖▜▚▜▚▚▚▜▚█▚▚▖▖▖
          ██▚▚▚▚▚▚▚▜▚▜▖▚██▜▜▚▖▚▚█▚▚▚▖▜▜
             ▜▚▚▖▚▚▚▖▚▜▜██▜▜▚█▚▚▜▚▚▜██▚
                ▚▚█▚▜▚▚█▖▜▚▚▚▖█▚▚█▚▚█▚
                   █▜▜▚▚▜▜▚▚▚▜█▚▚▚▜█▜█
                      ▚▚▖▚█▖▚▖▜▚▖▚▖▜▚
                         ███▖██▚▖▚▚▚▚
                            ▜▚▚█▚▚▖▖█
                              ▚▖▜█▜▚
                                 ▖█▚

Gorgeous.

Storage on Azure via HDFS

By adding some additional configuration, you can now use our HDFS Layer Backend to read and write GeoTrellis layers to Microsoft Azure's blob storage.

S3 Configurability

It's now possible to customize how our S3 backend communicates with S3.

Configuring JTS Precision

GeoTrellis uses the Java Topology Suite for its vector processing. By default, JTS uses a "floating" PrecisionModel. When writing code that needs to be numerically robust, this default can lead to Topology Exceptions.

You can now use Typesafe Config to configure this to your application's needs. See here for the specifics.

Added

Changed

[1.1.0] - 2017-06-22

Added

Changes

While we are trying to stick strictly to SemVer, there are slight API changes in this release. We felt that while this does break SemVer in the strictest sense, the change were not enough to warrant a 2.0 release. Our hope is in the future to be more cognizant of API changes for future releases.

  • Made EPSG capitalization consistent in method names:

    • In geotrellis.proj4.CRS, changed getEPSGCode to getEpsgCode
    • In geotrellis.proj4.io.wkt.WKT, changed fromEPSGCode to fromEpsgCode and getEPSGCode to getEpsgCode

  • Changed some internal but publicly visible classes dealing with GeoTiff reading

    • Changed size to length in ArraySegmentBytes
    • Replaced foreach on SegmentBytes with getSegments, which the caller can iterate over themselves
    • Changed getDecompressedBytes to decompressGeoTiffSegment

  • Changed some internal but publicly visible implicit classes and read methods around TiffTagReader

    • Added as an implicit parameter to multiple locations, most publicly in TiffTagReader.read(byteReader: ByteReader, tagsStartPosition: Long)(implicit ttos: TiffTagOffsetSize). Also changed that method from being generic to always taking a Long offset.

  • Moved some misplaced implicit JsonFormats

    • Moved CellTypeFormat and CellSizeFormat from geotrellis.spark.etl.config.json in the spark-etl subproject to geotrellis.raster.io.json.Implicits in the raster subproject.

  • Changed LazyLogger from the com.typesafe.scalalogging version to our own version

    • This shouldn't break any code, but technically is an API change.

[1.0.0] - 2017-05-17

Added

  • GeoTools support
    • Add Support for GeoTools SimpleFeature #1495
    • Conversions between GeoTools GridCoverage2D and GeoTrellis Raster types #1502
  • Streaming GeoTiff reading #1559
  • Windowed GeoTiff ingests into GeoTrellis layers, allowing users to ingest large GeoTiffs #1763
    • Reading TiffTags via MappedByteBuffer #1541
    • Cropped Windowed GeoTiff Reading #1559
    • Added documentation to the GeoTiff* files #1560
    • Windowed GeoTiff Docs #1616
  • GeoWave Raster/Vector support (experimental)
    • Create GeoWave Subproject #1542
    • Add vector capabilities to GeoWave support #1581
    • Fix GeoWave Tests #1665
  • GeoMesa Vector support (experimental)
    • Create GeoMesa subproject #1621
  • Moved to a JSON-configuration ETL process
    • ETL Refactor #1553
    • ETL Improvements and other issues fixes #1647
  • Vector Tile reading and writing, file-based and as GeoTrellis layers in RDDs. #1622
  • File Backends
  • Collections API #1606
    • Collections polygonal summary functions #1614
    • Collections mapalgebra focal functions #1619
  • Add TileFeature Type #1429
  • Added Focal calculation target type #1601
  • Triangulation
    • Voronoi diagrams and Delaunay triangulations #1545, #1699
    • Conforming Delaunay Triangulation #1848
  • Euclidean distance tiles #1552
  • Spark, Scala and Java version version support
    • Move to Spark 2; Scala 2.10 deprecation #1628
    • Java 7 deprecation #1640
  • Color correction features:
    • Histogram Equalization #1668
    • Sigmoidal Contrast #1681
    • Histogram matching #1769
  • CollectNeighbors feature, allowing users to group arbitrary values by the neighbor keys according to their SpatialComponent #1860
  • Documentation: We moved to ReadTheDocs, and put a lot of work into making our docs significantly better. See them here.

Minor Additions

  • Documentation improvements: Quickstart, Examples
    • Added example for translating from SpaceTimeKey to SpatialKey #1549
    • doc-examples subproject; example for tiling to GeoTiff #1564
    • Added example for focal operation on multiband layer. #1577
    • Projections, Extents, and Layout Definitions doc #1608
    • Added example of turning a list of features into GeoJSON #1609
    • Example: ShardingKeyIndex[K] #1633
    • Example: VoxelKey #1639
  • Introduce ADR concept
    • ADR: HDFS Raster Layers #1582
    • ADR: Readers / Writers Multithreading #1613
  • Fixed some markdown docs #1625
  • parseGeoJson lives in geotrellis.vector.io #1649
  • Parallelize reads for S3, File, and Cassandra backends #1607
  • Kernel Density in Spark
  • k-Nearest Neighbors
  • Updated slick
  • Added GeoTiff read/write support of TIFFTAG_PHOTOMETRIC via GeoTiffOptions. #1667
  • Added ability to read/write color tables for GeoTIFFs encoded with palette photometric interpretation #1802
  • Added ColorMap to String conversion #1512
  • Add split by cols/rows to SplitMethods #1538
  • Improved HDFS support #1556
  • Added Vector Join operation for Spark #1610
  • Added Histograms Over Fractions of RDDs of Tiles #1692
  • Add interpretAs and withNoData methods to Tile #1702
  • Changed GeoTiff reader to handle BigTiff #1753
  • Added BreakMap for reclassification based on range values. #1760
  • Allow custom save actions on ETL #1764
  • Multiband histogram methods #1784
  • DelayedConvert feature, allowing users to delay conversions on tiles until a map or combine operation, so that tiles are not iterated over unnecessarily #1797
  • Add convenience overloads to GeoTiff companion object #1840

Fixes / Optimizations

  • Fixed GeoTiff bug in reading NoData value if len = 4 #1490
    • Add detail to avro exception message #1505
    • Fix: The toSpatial method gives metadata of type TileLayerMetadata[SpaceTimeKey]
      • Custom Functor Typeclass #1643
    • Allow Intersects(polygon: Polygon) in layer query #1644
    • Optimize ColorMap #1648
    • Make regex for s3 URLs handle s3/s3a/s3n #1652
    • Fixed metadata handling on surface calculation for tile layer RDDs #1684
    • Fixed reading GeoJson with 3d values #1704
    • Fix to Bicubic Interpolation #1708
    • Fixed: Band tags with values of length > 31 have additional white space added to them #1756
    • Fixed NoData bug in tile merging logic #1793
    • Fixed Non-Point Pixel + Partial Cell Rasterizer Bug #1804

New Committers

  • metasim
  • lokifacio
  • aeffrig
  • jpolchlo
  • jbouffard
  • vsimko
  • longcmu
  • miafg

[0.10.3]

  • PR #1611 Any RDD of Tiles can utilize Polygonal Summary methods. (@fosskers)
  • PR #1573 New foreach for MultibandTile which maps over each band at once. (@hjaekel)
  • PR #1600 New mapBands method to map more cleanly over the bands of a MultibandTile.

[0.10.2]

  • PR #1561 Fix to polygon sequence union, account that it can result in NoResult. (1)
  • PR #1585 Removed warnings; add proper subtyping to GetComponent and SetComponent identity implicits; fix jai travis breakage. (1)
  • PR #1569 Moved RDDLayoutMergeMethods functionality to object. (1)
  • PR #1494 Add ETL option to specify upper zoom limit for raster layer ingestion (@mbertrand)
  • PR #1571 Fix scallop upgrade issue in spark-etl (@pomadchin)
  • PR #1543 Fix to Hadoop LayerMover (@pomadchin)

Special thanks to new contributor @mbertrand!

[0.10.1]

  • PR #1451 Optimize reading from compressed Bit geotiffs (@shiraeeshi)
  • PR #1454 Fix issues with IDW interpolation (@lokifacio)
  • PR #1457 Store FastMapHistogram counts as longs (@jpolchlo)
  • PR #1460 Fixes to user defined float/double CellType parsing (@echeipesh)
  • PR #1461 Pass resampling method argument to merge in CutTiles (1)
  • PR #1466 Handle Special Characters in proj4j (@jamesmcclain)
  • PR #1468 Fix nodata values in codecs (@shiraeeshi)
  • PR #1472 Fix typo in MultibandIngest.scala (@timothymschier)
  • PR #1478 Fix month and year calculations (@shiraeeshi)
  • PR #1483 Fix Rasterizer Bug (@jamesmcclain)
  • PR #1485 Upgrade dependencies as part of our LocationTech CQ process (1)
  • PR #1487 Handle entire layers of NODATA (@fosskers)
  • PR #1493 Added support for int32raw cell types in CellType.fromString (@jpolchlo)
  • PR #1496 Update slick (@adamkozuch, @moradology)
  • PR #1498 Add ability to specify number of streaming buckets (@moradology)
  • PR #1500 Add logic to ensure use of minval/avoid repetition of breaks (@moradology)
  • PR #1501 SparkContext temporal GeoTiff format args (@echeipesh)
  • PR #1510 Remove dep on cellType when specifying layoutExtent (@fosskers)
  • PR #1529 LayerUpdater fix (@pomadchin)

Special thanks to new contributors @fosskers, @adamkozuch, @jpolchlo, @shiraeeshi, @lokifacio!

[0.10.0]

The long awaited GeoTrellis 0.10 release is here!

It’s been a while since the 0.9 release of GeoTrellis, and there are many significant changes and improvements in this release. GeoTrellis has become an expansive suite of modular components that aide users in the building of geospatial application in Scala, and as always we’ve focused specifically on high performance and distributed computing. This is the first official release that supports working with Apache Spark, and we are very pleased with the results that have come out of the decision to support Spark as our main distributed processing engine. Those of you who have been tuned in for a while know we started with a custom built processing engine based on Akka actors; this original execution engine still exists in 0.10 but is in a deprecated state in the geotrellis-engine subproject. Along with upgrading GeoTrellis to support Spark and handle arbitrarily-sized raster data sets, we’ve been making improvements and additions to core functionality, including adding vector and projection support.

It’s been long enough that release notes, stating what has changed since 0.9, would be quite unwieldy. Instead I put together a list of features that GeoTrellis 0.10 supports. This is included in the README on the GeoTrellis Github, but I will put them here as well. It is organized by subproject, with more basic and core subprojects higher in the list, and the subprojects that rely on that core functionality later in the list, along with a high level description of each subproject.

geotrellis-proj4

  • Represent a Coordinate Reference System (CRS) based on Ellipsoid, Datum, and Projection.
  • Translate CRSs to and from proj4 string representations.
  • Lookup CRS's based on EPSG and other codes.
  • Transform (x, y) coordinates from one CRS to another.

geotrellis-vector

  • Provides a scala idiomatic wrapper around JTS types: Point, Line (LineString in JTS), Polygon, MultiPoint, MultiLine (MultiLineString in JTS), MultiPolygon, GeometryCollection
  • Methods for geometric operations supported in JTS, with results that provide a type-safe way to match over possible results of geometries.
  • Provides a Feature type that is the composition of a geometry and a generic data type.
  • Read and write geometries and features to and from GeoJSON.
  • Read and write geometries to and from WKT and WKB.
  • Reproject geometries between two CRSs.
  • Geometric operations: Convex Hull, Densification, Simplification
  • Perform Kriging interpolation on point values.
  • Perform affine transformations of geometries

geotrellis-vector-testkit

  • GeometryBuilder for building test geometries
  • GeometryMatcher for scalatest unit tests, which aides in testing equality in geometries with an optional threshold.

geotrellis-raster

  • Provides types to represent single- and multi-band rasters, supporting Bit, Byte, UByte, Short, UShort, Int, Float, and Double data, with either a constant NoData value (which improves performance) or a user defined NoData value.
  • Treat a tile as a collection of values, by calling "map" and "foreach", along with floating point valued versions of those methods (separated out for performance).
  • Combine raster data in generic ways.
  • Render rasters via color ramps and color maps to PNG and JPG images.
  • Read GeoTiffs with DEFLATE, LZW, and PackBits compression, including horizontal and floating point prediction for LZW and DEFLATE.
  • Write GeoTiffs with DEFLATE or no compression.
  • Reproject rasters from one CRS to another.
  • Resample of raster data.
  • Mask and Crop rasters.
  • Split rasters into smaller tiles, and stitch tiles into larger rasters.
  • Derive histograms from rasters in order to represent the distribution of values and create quantile breaks.
  • Local Map Algebra operations: Abs, Acos, Add, And, Asin, Atan, Atan2, Ceil, Cos, Cosh, Defined, Divide, Equal, Floor, Greater, GreaterOrEqual, InverseMask, Less, LessOrEqual, Log, Majority, Mask, Max, MaxN, Mean, Min, MinN, Minority, Multiply, Negate, Not, Or, Pow, Round, Sin, Sinh, Sqrt, Subtract, Tan, Tanh, Undefined, Unequal, Variance, Variety, Xor, If
  • Focal Map Algebra operations: Hillshade, Aspect, Slope, Convolve, Conway's Game of Life, Max, Mean, Median, Mode, Min, MoransI, StandardDeviation, Sum
  • Zonal Map Algebra operations: ZonalHistogram, ZonalPercentage
  • Operations that summarize raster data intersecting polygons: Min, Mean, Max, Sum.
  • Cost distance operation based on a set of starting points and a friction raster.
  • Hydrology operations: Accumulation, Fill, and FlowDirection.
  • Rasterization of geometries and the ability to iterate over cell values covered by geometries.
  • Vectorization of raster data.
  • Kriging Interpolation of point data into rasters.
  • Viewshed operation.
  • RegionGroup operation.

geotrellis-raster-testkit

  • Build test raster data.
  • Assert raster data matches Array data or other rasters in scalatest.

geotrellis-spark

  • Generic way to represent key value RDDs as layers, where the key represents a coordinate in space based on some uniform grid layout, optionally with a temporal component.
  • Represent spatial or spatiotemporal raster data as an RDD of raster tiles.
  • Generic architecture for saving/loading layers RDD data and metadata to/from various backends, using Spark's IO API with Space Filling Curve indexing to optimize storage retrieval (support for Hilbert curve and Z order curve SFCs). HDFS and local file system are supported backends by default, S3 and Accumulo are supported backends by the geotrellis-s3 and geotrellis-accumulo projects, respectively.
  • Query architecture that allows for simple querying of layer data by spatial or spatiotemporal bounds.
  • Perform map algebra operations on layers of raster data, including all supported Map Algebra operations mentioned in the geotrellis-raster feature list.
  • Perform seamless reprojection on raster layers, using neighboring tile information in the reprojection to avoid unwanted NoData cells.
  • Pyramid up layers through zoom levels using various resampling methods.
  • Types to reason about tiled raster layouts in various CRS's and schemes.
  • Perform operations on raster RDD layers: crop, filter, join, mask, merge, partition, pyramid, render, resample, split, stitch, and tile.
  • Polygonal summary over raster layers: Min, Mean, Max, Sum.
  • Save spatially keyed RDDs of byte arrays to z/x/y files into HDFS or the local file system. Useful for saving PNGs off for use as map layers in web maps or for accessing GeoTiffs through z/x/y tile coordinates.
  • Utilities around creating spark contexts for applications using GeoTrellis, including a Kryo registrator that registers most types.

geotrellis-spark-testkit

  • Utility code to create test RDDs of raster data.
  • Matching methods to test equality of RDDs of raster data in scalatest unit tests.

geotrellis-accumulo

  • Save and load layers to and from Accumulo. Query large layers efficiently using the layer query API.

geotrellis-cassandra

Save and load layers to and from Casandra. Query large layers efficiently using the layer query API.

geotrellis-s3

  • Save/load raster layers to/from the local filesystem or HDFS using Spark's IO API.
  • Save spatially keyed RDDs of byte arrays to z/x/y files in S3. Useful for saving PNGs off for use as map layers in web maps.

geotrellis-etl

  • Parse command line options for input and output of ETL (Extract, Transform, and Load) applications
  • Utility methods that make ETL applications easier for the user to build.
  • Work with input rasters from the local file system, HDFS, or S3
  • Reproject input rasters using a per-tile reproject or a seamless reprojection that takes into account neighboring tiles.
  • Transform input rasters into layers based on a ZXY layout scheme
  • Save layers into Accumulo, S3, HDFS or the local file system.

geotrellis-shapefile

  • Read geometry and feature data from shapefiles into GeoTrellis types using GeoTools.

geotrellis-slick

  • Save and load geometry and feature data to and from PostGIS using the slick scala database library.
  • Perform PostGIS ST_ operations in PostGIS through scala.