-
Notifications
You must be signed in to change notification settings - Fork 24
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
N5 support #6466
Merged
Merged
N5 support #6466
Changes from 9 commits
Commits
Show all changes
19 commits
Select commit
Hold shift + click to select a range
3d0f4ae
add N5 layer classes
deb0602
first working version
leowe 3a05223
refactor zarr and n5 with common superclasses, and refactor datareade…
leowe 7d02c62
Merge branch 'master' into n5-support
leowe 4f55598
add PaddedChunkReader
leowe 7e7b5f8
rename zarrMagOpt to n5magopt
leowe 9ab0548
add CompressionOption
leowe e89d2ad
update changelog
leowe f498de6
Merge branch 'master' into n5-support
leowe 69ac5ec
Merge branch 'master' into n5-support
leowe d379b89
Update webknossos-datastore/app/com/scalableminds/webknossos/datastor…
leowe c918707
Merge branch 'master' into n5-support
leowe 7cb1d55
Merge branch 'master' into n5-support
leowe a1dab94
incorporate pr feedback
leowe 8df6aeb
reformat
leowe 94edba2
Merge branch 'master' into n5-support
fm3 a9fce98
Fix JsonImplicits import, rename DatasetLocatorMag to MagLocator
fm3 491515e
Merge branch 'master' into n5-support
fm3 b0fc5ce
Merge branch 'master' into n5-support
fm3 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
55 changes: 55 additions & 0 deletions
55
...atastore/app/com/scalableminds/webknossos/datastore/dataformats/n5/N5BucketProvider.scala
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| package com.scalableminds.webknossos.datastore.dataformats.n5 | ||
|
|
||
| import com.scalableminds.util.geometry.Vec3Int | ||
| import com.scalableminds.util.requestlogging.RateLimitedErrorLogging | ||
| import com.scalableminds.util.tools.Fox | ||
| import com.scalableminds.webknossos.datastore.dataformats.zarr.{N5Layer, N5Mag} | ||
| import com.scalableminds.webknossos.datastore.dataformats.{BucketProvider, DataCubeHandle} | ||
| import com.scalableminds.webknossos.datastore.models.BucketPosition | ||
| import com.scalableminds.webknossos.datastore.models.requests.DataReadInstruction | ||
| import com.scalableminds.webknossos.datastore.datareaders.n5.N5Array | ||
| import com.typesafe.scalalogging.LazyLogging | ||
| import net.liftweb.common.{Box, Empty, Failure, Full} | ||
| import net.liftweb.util.Helpers.tryo | ||
|
|
||
| import java.nio.file.Path | ||
| import scala.concurrent.ExecutionContext | ||
|
|
||
| class N5CubeHandle(n5Array: N5Array) extends DataCubeHandle with LazyLogging with RateLimitedErrorLogging { | ||
|
|
||
| def cutOutBucket(bucket: BucketPosition)(implicit ec: ExecutionContext): Fox[Array[Byte]] = { | ||
| val shape = Vec3Int.full(bucket.bucketLength) | ||
| val offset = Vec3Int(bucket.voxelXInMag, bucket.voxelYInMag, bucket.voxelZInMag) | ||
| n5Array.readBytesXYZ(shape, offset).recover { | ||
| case t: Throwable => logError(t); Failure(t.getMessage, Full(t), Empty) | ||
| } | ||
| } | ||
|
|
||
| override protected def onFinalize(): Unit = () | ||
|
|
||
| } | ||
|
|
||
| class N5BucketProvider(layer: N5Layer) extends BucketProvider with LazyLogging with RateLimitedErrorLogging { | ||
|
|
||
| override def loadFromUnderlying(readInstruction: DataReadInstruction): Box[N5CubeHandle] = { | ||
| val n5MagOpt: Option[N5Mag] = | ||
| layer.mags.find(_.mag == readInstruction.bucket.mag) | ||
|
|
||
| n5MagOpt match { | ||
| case None => Empty | ||
| case Some(n5Mag) => | ||
| val magPathOpt: Option[Path] = { | ||
| n5Mag.remoteSource match { | ||
| case Some(remoteSource) => remotePathFrom(remoteSource) | ||
| case None => localPathFrom(readInstruction, n5Mag.pathWithFallback) | ||
| } | ||
| } | ||
| magPathOpt match { | ||
| case None => Empty | ||
| case Some(magPath) => | ||
| tryo(onError = e => logError(e))(N5Array.open(magPath, n5Mag.axisOrder)).map(new N5CubeHandle(_)) | ||
| } | ||
| } | ||
|
|
||
| } | ||
| } |
79 changes: 79 additions & 0 deletions
79
...os-datastore/app/com/scalableminds/webknossos/datastore/dataformats/n5/N5DataLayers.scala
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,79 @@ | ||
| package com.scalableminds.webknossos.datastore.dataformats.zarr | ||
|
|
||
| import java.net.URI | ||
| import com.scalableminds.util.geometry.{BoundingBox, Vec3Int} | ||
| import com.scalableminds.webknossos.datastore.dataformats.n5.N5BucketProvider | ||
| import com.scalableminds.webknossos.datastore.datareaders.AxisOrder | ||
| import com.scalableminds.webknossos.datastore.models.datasource.LayerViewConfiguration.LayerViewConfiguration | ||
| import com.scalableminds.webknossos.datastore.models.datasource._ | ||
| import com.scalableminds.webknossos.datastore.storage.FileSystemsHolder | ||
| import play.api.libs.json.{Json, OFormat} | ||
|
|
||
| case class N5Mag(mag: Vec3Int, | ||
| path: Option[String], | ||
| credentials: Option[FileSystemCredentials], | ||
| axisOrder: Option[AxisOrder]) { | ||
|
|
||
| lazy val pathWithFallback: String = | ||
| path.getOrElse(if (mag.isIsotropic) s"${mag.x}" else s"${mag.x}-${mag.y}-${mag.z}") | ||
| private lazy val uri: URI = new URI(pathWithFallback) | ||
| private lazy val isRemote: Boolean = FileSystemsHolder.isSupportedRemoteScheme(uri.getScheme) | ||
| lazy val remoteSource: Option[RemoteSourceDescriptor] = | ||
| if (isRemote) | ||
| Some(RemoteSourceDescriptor(uri, credentials.map(_.user), credentials.flatMap(_.password))) | ||
| else | ||
| None | ||
|
|
||
| } | ||
|
|
||
| object N5Mag extends ResolutionFormatHelper { | ||
| implicit val jsonFormat: OFormat[N5Mag] = Json.format[N5Mag] | ||
| } | ||
|
|
||
| trait N5Layer extends DataLayer { | ||
|
|
||
| val dataFormat: DataFormat.Value = DataFormat.n5 | ||
|
|
||
| lazy val bucketProvider = new N5BucketProvider(this) | ||
|
|
||
| def resolutions: List[Vec3Int] = mags.map(_.mag) | ||
|
|
||
| def mags: List[N5Mag] | ||
|
|
||
| def lengthOfUnderlyingCubes(resolution: Vec3Int): Int = Int.MaxValue // Prevents the wkw-shard-specific handle caching | ||
|
|
||
| def numChannels: Option[Int] = Some(if (elementClass == ElementClass.uint24) 3 else 1) | ||
|
|
||
| } | ||
|
|
||
| case class N5DataLayer( | ||
| name: String, | ||
| category: Category.Value, | ||
| boundingBox: BoundingBox, | ||
| elementClass: ElementClass.Value, | ||
| mags: List[N5Mag], | ||
| defaultViewConfiguration: Option[LayerViewConfiguration] = None, | ||
| adminViewConfiguration: Option[LayerViewConfiguration] = None, | ||
| override val numChannels: Option[Int] = Some(1) | ||
| ) extends N5Layer | ||
|
|
||
| object N5DataLayer { | ||
| implicit val jsonFormat: OFormat[N5DataLayer] = Json.format[N5DataLayer] | ||
| } | ||
|
|
||
| case class N5SegmentationLayer( | ||
| name: String, | ||
| boundingBox: BoundingBox, | ||
| elementClass: ElementClass.Value, | ||
| mags: List[N5Mag], | ||
| largestSegmentId: Long, | ||
| mappings: Option[Set[String]] = None, | ||
| defaultViewConfiguration: Option[LayerViewConfiguration] = None, | ||
| adminViewConfiguration: Option[LayerViewConfiguration] = None, | ||
| override val numChannels: Option[Int] = Some(1) | ||
| ) extends SegmentationLayer | ||
| with N5Layer | ||
|
|
||
| object N5SegmentationLayer { | ||
| implicit val jsonFormat: OFormat[N5SegmentationLayer] = Json.format[N5SegmentationLayer] | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
22 changes: 22 additions & 0 deletions
22
...ssos-datastore/app/com/scalableminds/webknossos/datastore/datareaders/ArrayDataType.scala
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,22 @@ | ||
| package com.scalableminds.webknossos.datastore.datareaders | ||
|
|
||
| import com.scalableminds.util.enumeration.ExtendedEnumeration | ||
|
|
||
| object ArrayDataType extends ExtendedEnumeration { | ||
| type ArrayDataType = Value | ||
| val f8, f4, i8, u8, i4, u4, i2, u2, i1, u1 = Value | ||
|
|
||
| def bytesPerElementFor(dataType: ArrayDataType): Int = | ||
| dataType match { | ||
| case ArrayDataType.f8 => 8 | ||
| case ArrayDataType.f4 => 4 | ||
| case ArrayDataType.i8 => 8 | ||
| case ArrayDataType.u8 => 8 | ||
| case ArrayDataType.i4 => 4 | ||
| case ArrayDataType.u4 => 4 | ||
| case ArrayDataType.i2 => 2 | ||
| case ArrayDataType.u2 => 2 | ||
| case ArrayDataType.i1 => 1 | ||
| case ArrayDataType.u1 => 1 | ||
| } | ||
| } |
2 changes: 1 addition & 1 deletion
2
...bknossos/datastore/jzarr/ArrayOrder.scala → ...os/datastore/datareaders/ArrayOrder.scala
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This reads a lot like ZarrMag. Maybe there can be a common trait? Not sure about a good name, though, ExtendedMag? Maybe you have a better Idea :D
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is exactly
ZarrMag. I went ahead and used hte new class in both cases (DatasetLocatorMag)There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If you have better ideas for the name, feel free to share ;)