Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add aggregation support for geo_shape fields #50834

Closed
wants to merge 141 commits into from
Closed
Show file tree
Hide file tree
Changes from 122 commits
Commits
Show all changes
141 commits
Select commit Hold shift + click to select a range
00f5a60
introduce EdgeTree writer and reader (#40810)
talevy May 21, 2019
e81ef33
Introduce GeometryTree writer/reader (#42331)
talevy May 22, 2019
9b5e5f3
Merge remote-tracking branch 'upstream/master' into geoshape-doc-values
talevy May 22, 2019
b864b4a
Merge remote-tracking branch 'upstream/master' into geoshape-doc-values
talevy May 23, 2019
6938994
Merge remote-tracking branch 'upstream/master' into geoshape-doc-values
talevy May 24, 2019
119bfc6
Merge remote-tracking branch 'upstream/master' into geoshape-doc-values
talevy May 30, 2019
c393d05
Merge remote-tracking branch 'upstream/master' into geoshape-doc-values
talevy Jun 3, 2019
c10da7f
Merge remote-tracking branch 'upstream/master' into geoshape-doc-values
talevy Jun 4, 2019
6eeabc7
fix changes
talevy Jun 4, 2019
bf6b3fe
Merge remote-tracking branch 'upstream/master' into geoshape-doc-values
talevy Jun 5, 2019
5791a96
Merge remote-tracking branch 'upstream/master' into geoshape-doc-values
talevy Jun 6, 2019
49975f2
make GeometryTree and EdgeTree writers implement Writeable (#42910)
talevy Jun 7, 2019
16c8a4d
Merge remote-tracking branch 'upstream/master' into geoshape-doc-values
talevy Jun 10, 2019
92b24c3
Clean up Extent logic in Geo Tree readers (#42968)
talevy Jun 10, 2019
2d06ac2
use encoded lat and lon integers
talevy Jun 10, 2019
7fde5e7
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
Jun 13, 2019
3496f1e
Merge remote-tracking branch 'upstream/master' into geoshape-doc-values
talevy Jun 20, 2019
417c48c
Add GeometryTree support for point/multipoint (#43432)
talevy Jun 21, 2019
c9a8046
Merge remote-tracking branch 'upstream/master' into geoshape-doc-values
talevy Jun 21, 2019
cd62ddd
Merge remote-tracking branch 'upstream/master' into geoshape-doc-values
talevy Jul 2, 2019
f76240e
Merge remote-tracking branch 'upstream/master' into geoshape-doc-values
talevy Jul 3, 2019
db446db
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Jul 8, 2019
aac8862
fix bug in point2d select algorithm
talevy Jul 8, 2019
bd3a935
fix point2d offbyone bug
talevy Jul 9, 2019
74c3f5d
add EdgeTree support for lines/multilines (#43949)
talevy Jul 9, 2019
86f7544
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Jul 10, 2019
ed052e3
wrap MultiGeoPointValues into generic type (#42863)
talevy Jul 10, 2019
bf1001f
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Jul 22, 2019
a3a9ac0
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Jul 23, 2019
019a6d7
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Jul 25, 2019
a0196e2
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Jul 26, 2019
90fe361
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Jul 29, 2019
0bb6727
fix lat/lon encoding from Geometries to the trees
talevy Jul 29, 2019
c0d6490
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Aug 1, 2019
e62d1cb
fix merge with spatial's shape indexer
talevy Aug 2, 2019
cdc9a50
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Aug 2, 2019
67430d4
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Aug 5, 2019
9778e76
Add support for polygons with holes in GeometryTree (#45068)
talevy Aug 5, 2019
0e28db9
add polygon comb test
talevy Aug 5, 2019
9776596
move geo-encoding of points to GeometryTree (#45221)
talevy Aug 6, 2019
70fefef
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Aug 6, 2019
ad4e6f8
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Aug 7, 2019
b12ab7c
update Extent to store enough info to handle dateline wrapping (#45301)
talevy Aug 8, 2019
1a7f432
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Aug 8, 2019
1eed9da
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Aug 8, 2019
8d396ab
support proper ValuesSourceConfig resolution of GEO and GEO_SHAPE. (#…
talevy Aug 9, 2019
527449b
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Aug 9, 2019
7dd1651
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Aug 12, 2019
2fcfbbf
add geo_shape support to geo_bounds agg (#45413)
talevy Aug 13, 2019
6b881cf
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Aug 13, 2019
6e8c6a8
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Aug 13, 2019
74cb1d0
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Aug 20, 2019
66673b9
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Aug 28, 2019
4df40ee
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Aug 30, 2019
04224e9
generalize encoding/decoding of coordinates to helper classes (#46154)
talevy Aug 30, 2019
0a36422
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Sep 3, 2019
741a456
Add geo_shape support to geo_centroid aggregation (#46299)
talevy Sep 10, 2019
ab674e1
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Sep 10, 2019
b37f263
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Sep 19, 2019
0fd63b8
geo-grid aggregation support for geo-shapes (#46513)
talevy Sep 20, 2019
4c0b935
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Sep 20, 2019
7901d22
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Sep 24, 2019
60e8c6e
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Oct 2, 2019
08dc844
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Oct 7, 2019
4ccee94
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Oct 23, 2019
5ecb3eb
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Oct 24, 2019
71a5c84
fix geoshapefieldmappertests to expect docvalues
talevy Oct 24, 2019
b6488a8
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Oct 25, 2019
1fcce0c
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Oct 28, 2019
63cab7d
fix compile issue in ValuesSourceConfigTests: type removal
talevy Oct 28, 2019
6ca6a14
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Oct 30, 2019
54c68b5
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Nov 1, 2019
19959d1
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Nov 11, 2019
c631a61
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Nov 13, 2019
0ad1b36
fix rightOffset bug in EdgeTreeReader
talevy Nov 13, 2019
668838a
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Nov 14, 2019
27724ce
Add randomized shape GeometryTree tests (#49112)
imotov Nov 14, 2019
2131daa
fix bug in EdgeTreeReader's intersect logic (#49127)
talevy Nov 15, 2019
f472b7f
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Nov 15, 2019
84e69dd
Add disjointness check for parallel lines in EdgeTreeReader (#49188)
talevy Nov 15, 2019
cd0b6b1
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Nov 21, 2019
65c23ee
re-introduce custom GEO logic for missing values
talevy Nov 21, 2019
72d0295
fix resolution of GEO field type in CoreValuesSourceType
talevy Nov 21, 2019
b560af1
remove unused geoshapeField
talevy Nov 21, 2019
5fed23d
Rasterize shape during geotilegrid computation (#49065)
talevy Nov 22, 2019
ab3296c
use sloppy math in geotile-grid utils (#49350)
talevy Nov 22, 2019
63ce350
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Nov 22, 2019
5d9b86b
Introduce variable encoding to EdgeTreeReader (#49349)
talevy Nov 22, 2019
cbc691f
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Nov 25, 2019
ffb3d27
Add support for geometrycollections to GeometryTreeReader and Writer …
imotov Nov 27, 2019
b39f712
Merge branch 'master' into updateGeoshapeBranch
iverase Nov 28, 2019
32ba24b
Add new encoding strategy based on Geometry tessellation. (#47903)
iverase Nov 28, 2019
2cb612e
Merge branch 'master' into geoshape-doc-values
iverase Dec 2, 2019
3d5de44
[Geo] Do not create a tree reader for each document (#49720)
iverase Dec 2, 2019
6d0c493
Merge branch 'master' into geoshape-doc-values
iverase Dec 3, 2019
0a90477
Improve resiliency and performance of geogrid aggregation over geosha…
iverase Dec 3, 2019
eca3328
fixes assertion of shape-count in GeometryTreeReader (#49810)
talevy Dec 3, 2019
87344b1
use ShapeType ordinals when deserializing shape trees by type (#49811)
talevy Dec 4, 2019
85f1e75
remove usage of Extent in ShapeTreeReader#relate logic (#49816)
talevy Dec 4, 2019
fbfee32
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Dec 5, 2019
0d4957f
Add full extent to the triangle tree to support geo_bounds aggregatio…
iverase Dec 5, 2019
a4402b3
Use TriangleTree for GeoShape Doc Values (#49840)
talevy Dec 10, 2019
abfb764
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Dec 10, 2019
bd1d0c2
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Dec 11, 2019
cabcfe5
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Dec 11, 2019
6f5919a
Build triangle tree from geo_shape indexed fields (#50012)
iverase Dec 12, 2019
21d4b49
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Dec 13, 2019
c560f38
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Dec 17, 2019
3337858
add shape-type metadata to geo_shape's doc-value (#50104)
talevy Dec 18, 2019
57da334
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Dec 19, 2019
4994a54
compress extent depending of how many values are set. (#50349)
iverase Dec 19, 2019
3794990
remove Extent.fromPoints tests
talevy Dec 19, 2019
9d5fa9f
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Jan 3, 2020
f686a0b
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Jan 6, 2020
2e15ea1
Implement weighted geo_shape centroid support (#50297)
talevy Jan 8, 2020
92f3426
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Jan 10, 2020
e60cd24
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Jan 14, 2020
15b029c
fix whitespace in AbstractGeometryFieldMapper loop
talevy Jan 14, 2020
aae5d74
use primitive double for constant in GeoTileUtils
talevy Jan 14, 2020
8ea0f66
use left-shift operator in place of Math.pow in GeoGridTiler
talevy Jan 14, 2020
4afc03e
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Jan 23, 2020
21142f9
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Jan 27, 2020
87c3e92
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Feb 6, 2020
a9d6a58
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Feb 10, 2020
7ef5fa5
remove usage of Lucene's GeoRelationUtils (#52165)
talevy Feb 10, 2020
47cb48f
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Feb 11, 2020
c95396d
GeoShape Doc Values: Fix and document tiling semantics for shapes (#5…
talevy Feb 11, 2020
6da87aa
fix encoding bug when clipping geo-tile latitude mask (#52274)
talevy Feb 13, 2020
0a57b59
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Feb 20, 2020
124e384
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Feb 20, 2020
6210b68
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Feb 24, 2020
004df2a
track the offset of the BytesRef passed to the tree reader (#52704)
iverase Feb 24, 2020
54483c9
reduce bytes used when serializing Extent (#52549)
talevy Feb 24, 2020
cc289f1
Fix geo_shape centroid calculation for downgraded shapes (#52500)
talevy Feb 24, 2020
4c410f0
add `doc_values` mapping option to geo_shape field mapping (#47519)
talevy Feb 24, 2020
d960712
ignore infinite centroid values in CentroidCalculator (#52782)
talevy Feb 26, 2020
4b9591c
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Feb 27, 2020
b136fa7
Add bounds support for geogrid agg on shapes (#51973)
talevy Mar 2, 2020
b773b46
add explicit tests for geo_shape with doc values on old indices (#52861)
talevy Mar 5, 2020
916f896
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Mar 10, 2020
95671be
Merge remote-tracking branch 'elastic/master' into geoshape-doc-values
talevy Mar 11, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -53,6 +53,20 @@ public class Geohash {
/** Bit encoded representation of the latitude of north pole */
private static final long MAX_LAT_BITS = (0x1L << (PRECISION * 5 / 2)) - 1;

// Below code is adapted from the spatial4j library (GeohashUtils.java) Apache 2.0 Licensed
private static final double[] precisionToLatHeight, precisionToLonWidth;
static {
precisionToLatHeight = new double[PRECISION + 1];
precisionToLonWidth = new double[PRECISION + 1];
precisionToLatHeight[0] = 90*2;
precisionToLonWidth[0] = 180*2;
boolean even = false;
for(int i = 1; i <= PRECISION; i++) {
precisionToLatHeight[i] = precisionToLatHeight[i-1] / (even ? 8 : 4);
precisionToLonWidth[i] = precisionToLonWidth[i-1] / (even ? 4 : 8);
even = ! even;
}
}

// no instance:
private Geohash() {
Expand Down Expand Up @@ -97,6 +111,16 @@ public static Rectangle toBoundingBox(final String geohash) {
}
}

/** Array of geohashes 1 level below the baseGeohash. Sorted. */
public static String[] getSubGeohashes(String baseGeohash) {
String[] hashes = new String[BASE_32.length];
for (int i = 0; i < BASE_32.length; i++) {//note: already sorted
char c = BASE_32[i];
hashes[i] = baseGeohash+c;
}
return hashes;
}

/**
* Calculate all neighbors of a given geohash cell.
*
Expand Down Expand Up @@ -201,6 +225,13 @@ public static final String getNeighbor(String geohash, int level, int dx, int dy
}
}

/**
* Encode a string geohash to the geohash based long format (lon/lat interleaved, 4 least significant bits = level)
*/
public static final long longEncode(String hash) {
return longEncode(hash, hash.length());
}

/**
* Encode lon/lat to the geohash based long format (lon/lat interleaved, 4 least significant bits = level)
*/
Expand Down Expand Up @@ -297,7 +328,6 @@ private static long encodeLatLon(final double lat, final double lon) {
return BitUtil.interleave(latEnc, lonEnc) >>> 2;
}


/** encode latitude to integer */
public static int encodeLatitude(double latitude) {
// the maximum possible value cannot be encoded without overflow
Expand All @@ -316,6 +346,16 @@ public static int encodeLongitude(double longitude) {
return (int) Math.floor(longitude / LON_DECODE);
}

/** approximate width of geohash tile for a specific precision in degrees */
public static double lonWidthInDegrees(int precision) {
return precisionToLonWidth[precision];
}

/** approximate height of geohash tile for a specific precision in degrees */
public static double latHeightInDegrees(int precision) {
return precisionToLatHeight[precision];
}

/** returns the latitude value from the string based geohash */
public static final double decodeLatitude(final String geohash) {
return decodeLatitude(Geohash.mortonEncode(geohash));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@
import org.apache.lucene.queries.function.FunctionValues;
import org.apache.lucene.queries.function.ValueSource;
import org.apache.lucene.queries.function.docvalues.DoubleDocValues;
import org.elasticsearch.index.fielddata.AtomicGeoPointFieldData;
import org.elasticsearch.index.fielddata.AtomicGeoFieldData;
import org.elasticsearch.index.fielddata.IndexFieldData;
import org.elasticsearch.index.fielddata.MultiGeoPointValues;
import org.elasticsearch.index.fielddata.MultiGeoValues;

/**
* ValueSource to return non-zero if a field is missing.
Expand All @@ -44,8 +44,8 @@ final class GeoEmptyValueSource extends ValueSource {
@Override
@SuppressWarnings("rawtypes") // ValueSource uses a rawtype
public FunctionValues getValues(Map context, LeafReaderContext leaf) throws IOException {
AtomicGeoPointFieldData leafData = (AtomicGeoPointFieldData) fieldData.load(leaf);
final MultiGeoPointValues values = leafData.getGeoPointValues();
AtomicGeoFieldData leafData = (AtomicGeoFieldData) fieldData.load(leaf);
final MultiGeoValues values = leafData.getGeoValues();
return new DoubleDocValues(this) {
@Override
public double doubleVal(int doc) throws IOException {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@
import org.apache.lucene.queries.function.FunctionValues;
import org.apache.lucene.queries.function.ValueSource;
import org.apache.lucene.queries.function.docvalues.DoubleDocValues;
import org.elasticsearch.index.fielddata.AtomicGeoPointFieldData;
import org.elasticsearch.index.fielddata.AtomicGeoFieldData;
import org.elasticsearch.index.fielddata.IndexFieldData;
import org.elasticsearch.index.fielddata.MultiGeoPointValues;
import org.elasticsearch.index.fielddata.MultiGeoValues;

/**
* ValueSource to return latitudes as a double "stream" for geopoint fields
Expand All @@ -44,13 +44,13 @@ final class GeoLatitudeValueSource extends ValueSource {
@Override
@SuppressWarnings("rawtypes") // ValueSource uses a rawtype
public FunctionValues getValues(Map context, LeafReaderContext leaf) throws IOException {
AtomicGeoPointFieldData leafData = (AtomicGeoPointFieldData) fieldData.load(leaf);
final MultiGeoPointValues values = leafData.getGeoPointValues();
AtomicGeoFieldData leafData = (AtomicGeoFieldData) fieldData.load(leaf);
final MultiGeoValues values = leafData.getGeoValues();
return new DoubleDocValues(this) {
@Override
public double doubleVal(int doc) throws IOException {
if (values.advanceExact(doc)) {
return values.nextValue().getLat();
return values.nextValue().lat();
} else {
return 0.0;
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,9 @@
import org.apache.lucene.queries.function.FunctionValues;
import org.apache.lucene.queries.function.ValueSource;
import org.apache.lucene.queries.function.docvalues.DoubleDocValues;
import org.elasticsearch.index.fielddata.AtomicGeoPointFieldData;
import org.elasticsearch.index.fielddata.AtomicGeoFieldData;
import org.elasticsearch.index.fielddata.IndexFieldData;
import org.elasticsearch.index.fielddata.MultiGeoPointValues;
import org.elasticsearch.index.fielddata.MultiGeoValues;

/**
* ValueSource to return longitudes as a double "stream" for geopoint fields
Expand All @@ -44,13 +44,13 @@ final class GeoLongitudeValueSource extends ValueSource {
@Override
@SuppressWarnings("rawtypes") // ValueSource uses a rawtype
public FunctionValues getValues(Map context, LeafReaderContext leaf) throws IOException {
AtomicGeoPointFieldData leafData = (AtomicGeoPointFieldData) fieldData.load(leaf);
final MultiGeoPointValues values = leafData.getGeoPointValues();
AtomicGeoFieldData leafData = (AtomicGeoFieldData) fieldData.load(leaf);
final MultiGeoValues values = leafData.getGeoValues();
return new DoubleDocValues(this) {
@Override
public double doubleVal(int doc) throws IOException {
if (values.advanceExact(doc)) {
return values.nextValue().getLon();
return values.nextValue().lon();
} else {
return 0.0;
}
Expand Down
Loading