Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Url attributes #91

Closed
wants to merge 75 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
75 commits
Select commit Hold shift + click to select a range
f849bdc
feat: wip add failing test for getting attributes from urls
bogovicj Nov 15, 2022
31f360c
feat: initial N5Url API
cmhulbert Nov 18, 2022
fa78a76
test: update test for getting attributes to use N5URL
bogovicj Nov 18, 2022
52e4add
feat: initial N5URL relativize support
cmhulbert Nov 18, 2022
6c4e70d
feat: getAttribute from url
cmhulbert Nov 21, 2022
02cc783
test: N5URL tests
cmhulbert Nov 21, 2022
ead786d
test: test object parsing from urls
bogovicj Nov 22, 2022
9b6e90d
feat: support generic arrays when getting attribute from N5URL
cmhulbert Nov 22, 2022
bc05979
feat: remove N5Url getAttributes methods in favor of handling interna…
cmhulbert Nov 28, 2022
f12345f
feat: initial handling of structured fragments in `getAttributes`; WIP
cmhulbert Nov 28, 2022
cd9eddf
refactor(test): migrate away from remove `getAttributes(N5Url, Class<…
cmhulbert Nov 28, 2022
62fb3f5
feat: add readAttributesJson
cmhulbert Dec 1, 2022
b3b0b0b
feat: static N5URL constructor from container/group/path
cmhulbert Dec 1, 2022
f5571cb
refactor: use getAttributesJson
cmhulbert Dec 1, 2022
a336886
refactor: rename fields/methods
cmhulbert Dec 1, 2022
520dbcb
refactor: more N5URL renaming
cmhulbert Dec 1, 2022
88d58f3
refactor: align method naming with Path conventions
cmhulbert Dec 1, 2022
bb9bd61
feat: add method to check if N5URL is absolute
cmhulbert Dec 1, 2022
5f0df9c
feat: add method to normalize container path
cmhulbert Dec 1, 2022
36cc96f
refactor: rename and use new methods
cmhulbert Dec 1, 2022
0fde153
fix: properly return when the N5Url is absolute
cmhulbert Dec 1, 2022
ffd5260
refactor: Move ARRAY_INDEX
cmhulbert Dec 2, 2022
d8f208a
feat: allow getAttributes root to be JsonElement
cmhulbert Dec 2, 2022
e0f3c46
feat: support set attribute by attribute path
cmhulbert Dec 2, 2022
84ae29f
fix: don't store temp files in user.home (#88)
cmhulbert Dec 2, 2022
4c3ab8c
test: set attribute by path
cmhulbert Dec 2, 2022
f269285
feat: support replacing incompatible structures based on the attribut…
cmhulbert Dec 5, 2022
a4ef866
refactor: move setAttribute logic to GsonAttributesParser
cmhulbert Dec 5, 2022
8258d3c
refactor(test): move setAttribute test to AbstractN5Test
cmhulbert Dec 5, 2022
fcc6c47
feat(test): add setAttribute tests for replacing intermediate members…
cmhulbert Dec 5, 2022
db0c4d6
feat: expose key
cmhulbert Dec 7, 2022
fe9a659
refactor: separate write/insertAttribute, move readAttribute
cmhulbert Dec 7, 2022
f6bc460
refactor: clarify that the input should be normalized
cmhulbert Dec 21, 2022
ab433d8
fix: handle JsonPrimitive values as leaf
cmhulbert Dec 21, 2022
fed78a9
fix: support when attributes.json doesn't contain JsonObject
cmhulbert Dec 21, 2022
fbc1b02
feat(test): add method to createN5Write from specified location
cmhulbert Dec 21, 2022
0759c37
feat(test): add method for createN5Writer with location
cmhulbert Dec 21, 2022
201d595
feat: support writing/reading opaque keys via `getAttributes`/`setAtt…
cmhulbert Dec 21, 2022
ed1dc19
test: test adding attributes via opaque keys
cmhulbert Dec 21, 2022
e21fbad
test: test replacement of existing root/structure of attributes
cmhulbert Dec 21, 2022
0f6a29a
test: add normalize identity check
cmhulbert Dec 21, 2022
1f15bbb
fix: return root for group and attribute if they weren't specified
cmhulbert Dec 21, 2022
254b46d
fix: always absolute if scheme is specified
cmhulbert Dec 21, 2022
b575124
refactor: rename parameters
cmhulbert Dec 21, 2022
cf5c71b
doc: add N5URL documentation
cmhulbert Dec 21, 2022
c69d850
test: add isAbsolute tests
cmhulbert Dec 21, 2022
a81cfc4
fix: N5FSReader should fail if base path does not exist (#89)
cmhulbert Dec 22, 2022
fa0f1cc
feat: expose encodeAsUri
cmhulbert Dec 22, 2022
1cc8d1c
Merge branch 'master' into urlAttributes
cmhulbert Dec 22, 2022
c1abdfc
feat: inline createDirectories now that non-existent path throws IOEx…
cmhulbert Jan 3, 2023
023dfed
fix(test): use new writer
cmhulbert Jan 3, 2023
2d023a5
fix(test): test for RawCompression inheritance (used by zarr)
cmhulbert Jan 3, 2023
17c90b8
fix(test): specify separate n5containers for Version and deeplist tests
cmhulbert Feb 7, 2023
5198fed
feat(test): add whitespace encoding and nested array tests
cmhulbert Feb 8, 2023
f4c3d71
feat!,refactor: update how path tokens behave
cmhulbert Feb 8, 2023
8eea1ec
docs: document LinkedAttributePathToken
cmhulbert Feb 9, 2023
3914a33
test: when constructing a reader should fail
bogovicj Feb 21, 2023
3adb677
test: setting attrs with escaping special chars for keys
bogovicj Feb 21, 2023
a335b07
feat!: replace Map<String, JsonElemen> variants with root JsonElement
cmhulbert Feb 22, 2023
4d36c2a
feat,test: support escape characters in attribute paths
cmhulbert Feb 22, 2023
536dc40
fix: improper handling of escape characters
cmhulbert Feb 22, 2023
13f535a
test: move to abstract, comment some currently non-functional tests
cmhulbert Feb 22, 2023
607314d
feat(test): add whitespace and escaped-character tests
cmhulbert Feb 23, 2023
bfa7dd4
fix(test): evaluate `\` through json properly
cmhulbert Feb 23, 2023
b92ecfb
fix: issue where attribute was not fully decoding encoded characters
cmhulbert Feb 23, 2023
ea20335
feat!: initial `removeAttribute` support, separate get/parse/read att…
cmhulbert Feb 23, 2023
62ed447
feat: properly normalize path, respecting escape sequences
cmhulbert Feb 27, 2023
7925afb
feat!: add explicit `removeAttribute` methods
cmhulbert Feb 27, 2023
233c30c
feat(test): add tests for `removeAttribute(s)`
cmhulbert Feb 27, 2023
69e3c8f
feat: add GsonN5Reader/Writer
cmhulbert Mar 3, 2023
1f4b13d
feat!: migrate away from AbstractGsonParser to GsonN5Reader/Writer
cmhulbert Mar 3, 2023
63e8b5d
feat!: deprecate for future removal
cmhulbert Mar 3, 2023
a6d92f6
feat: finalize and document attribute path regex normalization
cmhulbert Mar 3, 2023
9d01d96
feat!: deprecate for future removal, change `getAttributes` to `getAt…
cmhulbert Mar 3, 2023
7fe6799
fix(test): fix some incorrect tests
cmhulbert Mar 3, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
76 changes: 56 additions & 20 deletions src/main/java/org/janelia/saalfeldlab/n5/AbstractGsonReader.java
Original file line number Diff line number Diff line change
@@ -1,16 +1,16 @@
/**
* Copyright (c) 2017, Stephan Saalfeld
* All rights reserved.
*
* <p>
* Redistribution and use in source and binary forms, with or without
* modification, are permitted provided that the following conditions are met:
*
* <p>
* 1. Redistributions of source code must retain the above copyright notice,
* this list of conditions and the following disclaimer.
* this list of conditions and the following disclaimer.
* 2. Redistributions in binary form must reproduce the above copyright notice,
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
*
* this list of conditions and the following disclaimer in the documentation
* and/or other materials provided with the distribution.
* <p>
* THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
* AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
* IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE
Expand All @@ -25,14 +25,18 @@
*/
package org.janelia.saalfeldlab.n5;

import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.JsonElement;
import com.google.gson.JsonObject;
import com.google.gson.reflect.TypeToken;

import java.io.IOException;
import java.lang.reflect.Type;
import java.net.URISyntaxException;
import java.util.Arrays;
import java.util.HashMap;

import com.google.gson.Gson;
import com.google.gson.GsonBuilder;
import com.google.gson.JsonElement;
import java.util.Map;

/**
* Abstract base class implementing {@link N5Reader} with JSON attributes
Expand All @@ -41,7 +45,9 @@
* @author Stephan Saalfeld
* @author Igor Pisarev
* @author Philipp Hanslovsky
* @author Caleb Hulbert
*/
@Deprecated
public abstract class AbstractGsonReader implements GsonAttributesParser, N5Reader {

protected final Gson gson;
Expand All @@ -52,6 +58,7 @@ public abstract class AbstractGsonReader implements GsonAttributesParser, N5Read
*
* @param gsonBuilder
*/
@Deprecated
public AbstractGsonReader(final GsonBuilder gsonBuilder) {

gsonBuilder.registerTypeAdapter(DataType.class, new DataType.JsonAdapter());
Expand All @@ -64,40 +71,48 @@ public AbstractGsonReader(final GsonBuilder gsonBuilder) {
* Constructs an {@link AbstractGsonReader} with a default
* {@link GsonBuilder}.
*/
@Deprecated
public AbstractGsonReader() {

this(new GsonBuilder());
}

@Override
@Deprecated
public Gson getGson() {

return gson;
}

@Override
@Deprecated
public DatasetAttributes getDatasetAttributes(final String pathName) throws IOException {

final HashMap<String, JsonElement> map = getAttributes(pathName);
final JsonElement root = getAttributes(pathName);
final Gson gson = getGson();

final long[] dimensions = GsonAttributesParser.parseAttribute(map, DatasetAttributes.dimensionsKey, long[].class, gson);
if (root == null || !root.isJsonObject())
return null;

final JsonObject rootObject = root.getAsJsonObject();

final long[] dimensions = GsonN5Reader.readAttribute(rootObject, DatasetAttributes.dimensionsKey, long[].class, gson);
if (dimensions == null)
return null;

final DataType dataType = GsonAttributesParser.parseAttribute(map, DatasetAttributes.dataTypeKey, DataType.class, gson);
final DataType dataType = GsonN5Reader.readAttribute(rootObject, DatasetAttributes.dataTypeKey, DataType.class, gson);
if (dataType == null)
return null;

int[] blockSize = GsonAttributesParser.parseAttribute(map, DatasetAttributes.blockSizeKey, int[].class, gson);
int[] blockSize = GsonN5Reader.readAttribute(rootObject, DatasetAttributes.blockSizeKey, int[].class, gson);
if (blockSize == null)
blockSize = Arrays.stream(dimensions).mapToInt(a -> (int)a).toArray();

Compression compression = GsonAttributesParser.parseAttribute(map, DatasetAttributes.compressionKey, Compression.class, gson);
Compression compression = GsonN5Reader.readAttribute(rootObject, DatasetAttributes.compressionKey, Compression.class, gson);

/* version 0 */
if (compression == null) {
switch (GsonAttributesParser.parseAttribute(map, DatasetAttributes.compressionTypeKey, String.class, gson)) {
switch (GsonN5Reader.readAttribute(rootObject, DatasetAttributes.compressionTypeKey, String.class, gson)) {
case "raw":
compression = new RawCompression();
break;
Expand All @@ -120,22 +135,43 @@ public DatasetAttributes getDatasetAttributes(final String pathName) throws IOEx
}

@Override
@Deprecated
public <T> T getAttribute(
final String pathName,
final String key,
final Class<T> clazz) throws IOException {

final HashMap<String, JsonElement> map = getAttributes(pathName);
return GsonAttributesParser.parseAttribute(map, key, clazz, getGson());
return getAttribute(pathName, key, TypeToken.get(clazz).getType());
}

@Override
@Deprecated
public <T> T getAttribute(
final String pathName,
final String key,
final Type type) throws IOException {

final HashMap<String, JsonElement> map = getAttributes(pathName);
return GsonAttributesParser.parseAttribute(map, key, type, getGson());
final String normalizedGroupPath;
final String normalizedAttributePath;
try {
final N5URL n5url = N5URL.from(null, pathName, key);
normalizedGroupPath = n5url.normalizeGroupPath();
normalizedAttributePath = n5url.normalizeAttributePath();
} catch (URISyntaxException e) {
throw new IOException(e);
}
return GsonN5Reader.readAttribute(getAttributes(normalizedGroupPath), normalizedAttributePath, type, gson);
}
@Deprecated
public HashMap<String, JsonElement> getAttributesMap(String pathName) throws IOException {

return GsonAttributesParser.super.getAttributesMap(getAttributes(pathName));
}

@Override
@Deprecated
public Map<String, Class<?>> listAttributes(String pathName) throws IOException {

return GsonN5Reader.listAttributes(getAttributes(pathName));
}
}
Loading