Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix nml upload with only ds name in nml by using user's orga as fallback #8277

Merged
merged 4 commits into from
Dec 16, 2024
Merged

Fix nml upload with only ds name in nml by using user's orga as fallback #8277

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
648c5e3
Fix nml upload with only ds name set by using user's orga as fallback
Dec 13, 2024
8ea5c3a
add changelog entry
Dec 13, 2024
3b8761e
apply feedback
Dec 16, 2024
eed10e9
fix test compilation
Dec 16, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.unreleased.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,6 +26,7 @@ For upgrade instructions, please check the [migration guide](MIGRATIONS.released
- A "Locked by anonymous user" banner is no longer shown when opening public editable annotations of other organizations. [#8273](https://github.com/scalableminds/webknossos/pull/8273)
- Fixed a bug that uploading a zarr dataset with an already existing `datasource-properties.json` file failed. [#8268](https://github.com/scalableminds/webknossos/pull/8268)
- Fixed the organization switching feature for datasets opened via old links. [#8257](https://github.com/scalableminds/webknossos/pull/8257)
- Fixed that uploading an NML file without an organization id failed. Now the user's organization is used as fallback. [#8277](https://github.com/scalableminds/webknossos/pull/8277)
- Fixed that the frontend did not ensure a minium length for annotation layer names. Moreover, names starting with a `.` are also disallowed now. [#8244](https://github.com/scalableminds/webknossos/pull/8244)
- Fixed a bug where in the add remote dataset view the dataset name setting was not in sync with the datasource setting of the advanced tab making the form not submittable. [#8245](https://github.com/scalableminds/webknossos/pull/8245)
- Fix read and update dataset route for versions 8 and lower. [#8263](https://github.com/scalableminds/webknossos/pull/8263)
Expand Down
6 changes: 4 additions & 2 deletions app/controllers/AnnotationIOController.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -105,10 +105,12 @@ class AnnotationIOController @Inject()(
request.body.dataParts("createGroupForEachFile").headOption.contains("true")
val overwritingDatasetId: Option[String] =
request.body.dataParts.get("datasetId").flatMap(_.headOption)
val userOrganizationId = request.identity._organization
val attachedFiles = request.body.files.map(f => (f.ref.path.toFile, f.filename))
for {
parsedFiles <- annotationUploadService
.extractFromFiles(attachedFiles, SharedParsingParameters(useZipName = true, overwritingDatasetId))
parsedFiles <- annotationUploadService.extractFromFiles(
attachedFiles,
SharedParsingParameters(useZipName = true, overwritingDatasetId, userOrganizationId))
parsedFilesWrapped = annotationUploadService.wrapOrPrefixGroups(parsedFiles.parseResults,
shouldCreateGroupForEachFile)
parseResultsFiltered: List[NmlParseResult] = parsedFilesWrapped.filter(_.succeeded)
Expand Down
6 changes: 4 additions & 2 deletions app/controllers/TaskController.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -82,14 +82,16 @@ class TaskController @Inject()(taskCreationService: TaskCreationService,
_ <- bool2Fox(inputFiles.nonEmpty) ?~> "nml.file.notFound"
jsonString <- body.dataParts.get("formJSON").flatMap(_.headOption) ?~> "format.json.missing"
params <- JsonHelper.parseAndValidateJson[NmlTaskParameters](jsonString) ?~> "task.create.failed"
userOrganizationId = request.identity._organization
_ <- taskCreationService.assertBatchLimit(inputFiles.length, List(params.taskTypeId))
taskTypeIdValidated <- ObjectId.fromString(params.taskTypeId) ?~> "taskType.id.invalid"
taskType <- taskTypeDAO.findOne(taskTypeIdValidated) ?~> "taskType.notFound" ~> NOT_FOUND
project <- projectDAO
.findOneByNameAndOrganization(params.projectName, request.identity._organization) ?~> "project.notFound" ~> NOT_FOUND
_ <- Fox.assertTrue(userService.isTeamManagerOrAdminOf(request.identity, project._team))
extractedFiles <- nmlService.extractFromFiles(inputFiles.map(f => (f.ref.path.toFile, f.filename)),
SharedParsingParameters(useZipName = false, isTaskUpload = true))
extractedFiles <- nmlService.extractFromFiles(
inputFiles.map(f => (f.ref.path.toFile, f.filename)),
SharedParsingParameters(useZipName = false, isTaskUpload = true, userOrganizationId = userOrganizationId))
extractedTracingBoxesRaw: List[TracingBoxContainer] = extractedFiles.toBoxes
extractedTracingBoxes: List[TracingBoxContainer] <- taskCreationService.addVolumeFallbackBoundingBoxes(
extractedTracingBoxesRaw)
Expand Down
4 changes: 2 additions & 2 deletions app/models/annotation/AnnotationUploadService.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,7 @@ case class UploadedVolumeLayer(tracing: VolumeTracing, dataZipLocation: String,

case class SharedParsingParameters(useZipName: Boolean,
overwritingDatasetId: Option[String] = None,
userOrganizationId: String,
isTaskUpload: Boolean = false)

class AnnotationUploadService @Inject()(tempFileService: TempFileService, nmlParser: NmlParser) extends LazyLogging {
Expand All @@ -53,8 +54,7 @@ class AnnotationUploadService @Inject()(tempFileService: TempFileService, nmlPar
nmlParser.parse(
name,
inputStream,
sharedParsingParameters.overwritingDatasetId,
sharedParsingParameters.isTaskUpload,
Comment on lines -56 to -57
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all members of sharedParsingParameters are passed to nmlParser.parse anyway. So just pass the whole sharedParsingParameters object

sharedParsingParameters,
basePath
)
parserOutput.futureBox.map {
Expand Down
29 changes: 15 additions & 14 deletions app/models/annotation/nml/NmlParser.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ import com.scalableminds.webknossos.tracingstore.tracings.ColorGenerator
import com.scalableminds.webknossos.tracingstore.tracings.skeleton.updating.TreeType
import com.scalableminds.webknossos.tracingstore.tracings.skeleton.{MultiComponentTreeSplitter, TreeValidator}
import com.typesafe.scalalogging.LazyLogging
import models.annotation.UploadedVolumeLayer
import models.annotation.{SharedParsingParameters, UploadedVolumeLayer}
import models.dataset.DatasetDAO
import net.liftweb.common.Box._
import net.liftweb.common.{Box, Empty, Failure, Full}
Expand All @@ -48,13 +48,12 @@ class NmlParser @Inject()(datasetDAO: DatasetDAO) extends LazyLogging with Proto

def parse(name: String,
nmlInputStream: InputStream,
overwritingDatasetId: Option[String],
isTaskUpload: Boolean,
sharedParsingParameters: SharedParsingParameters,
basePath: Option[String] = None)(implicit m: MessagesProvider,
ec: ExecutionContext,
ctx: DBAccessContext): Fox[NmlParseSuccessWithoutFile] =
for {
nmlParsedParameters <- getParametersFromNML(nmlInputStream, name, overwritingDatasetId, isTaskUpload).toFox
nmlParsedParameters <- getParametersFromNML(nmlInputStream, name, sharedParsingParameters).toFox
parsedResult <- nmlParametersToResult(nmlParsedParameters, basePath)
} yield parsedResult

Expand Down Expand Up @@ -121,10 +120,10 @@ class NmlParser @Inject()(datasetDAO: DatasetDAO) extends LazyLogging with Proto
} yield
NmlParseSuccessWithoutFile(skeletonTracingOpt, volumeLayers, dataset._id, nmlParams.description, nmlParams.wkUrl)

private def getParametersFromNML(nmlInputStream: InputStream,
name: String,
overwritingDatasetId: Option[String],
isTaskUpload: Boolean)(implicit m: MessagesProvider): Box[NmlParsedParameters] =
private def getParametersFromNML(
nmlInputStream: InputStream,
name: String,
sharedParsingParameters: SharedParsingParameters)(implicit m: MessagesProvider): Box[NmlParsedParameters] =
try {
val nmlData = XML.load(nmlInputStream)
for {
Expand All @@ -143,21 +142,23 @@ class NmlParser @Inject()(datasetDAO: DatasetDAO) extends LazyLogging with Proto
_ <- TreeValidator.validateTrees(treesSplit, treeGroupsAfterSplit, branchPoints, comments)
additionalAxisProtos <- parseAdditionalAxes(parameters \ "additionalAxes")
datasetName = parseDatasetName(parameters \ "experiment")
datasetIdOpt = if (overwritingDatasetId.isDefined) overwritingDatasetId
datasetIdOpt = if (sharedParsingParameters.overwritingDatasetId.isDefined)
sharedParsingParameters.overwritingDatasetId
else parseDatasetId(parameters \ "experiment")
organizationId = parseOrganizationId(parameters \ "experiment")
organizationId = parseOrganizationId(parameters \ "experiment", sharedParsingParameters.userOrganizationId)
description = parseDescription(parameters \ "experiment")
wkUrl = parseWkUrl(parameters \ "experiment")
activeNodeId = parseActiveNode(parameters \ "activeNode")
(editPosition, editPositionAdditionalCoordinates) = parseEditPosition(parameters \ "editPosition")
.getOrElse((SkeletonTracingDefaults.editPosition, Seq()))
editRotation = parseEditRotation(parameters \ "editRotation").getOrElse(SkeletonTracingDefaults.editRotation)
zoomLevel = parseZoomLevel(parameters \ "zoomLevel").getOrElse(SkeletonTracingDefaults.zoomLevel)
taskBoundingBox: Option[BoundingBox] = if (isTaskUpload) parseTaskBoundingBox(parameters \ "taskBoundingBox")
taskBoundingBox: Option[BoundingBox] = if (sharedParsingParameters.isTaskUpload)
parseTaskBoundingBox(parameters \ "taskBoundingBox")
else None
userBoundingBoxes = parseBoundingBoxes(parameters \ "userBoundingBox")
} yield {
if (!isTaskUpload) {
if (!sharedParsingParameters.isTaskUpload) {
parseTaskBoundingBoxAsUserBoundingBox(parameters \ "taskBoundingBox", userBoundingBoxes)
.map(asUserBoundingBox => userBoundingBoxes :+ asUserBoundingBox)
}
Expand Down Expand Up @@ -379,8 +380,8 @@ class NmlParser @Inject()(datasetDAO: DatasetDAO) extends LazyLogging with Proto
private def parseWkUrl(nodes: NodeSeq): Option[String] =
nodes.headOption.map(node => getSingleAttribute(node, "wkUrl"))

private def parseOrganizationId(nodes: NodeSeq): String =
nodes.headOption.map(node => getSingleAttribute(node, "organization")).getOrElse("")
private def parseOrganizationId(nodes: NodeSeq, fallbackOrganization: String): String =
nodes.headOption.flatMap(node => getSingleAttributeOpt(node, "organization")).getOrElse(fallbackOrganization)

private def parseActiveNode(nodes: NodeSeq): Option[Int] =
nodes.headOption.flatMap(node => getSingleAttribute(node, "id").toIntOpt)
Expand Down
7 changes: 5 additions & 2 deletions test/backend/NMLUnitTestSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,7 @@ import com.scalableminds.webknossos.datastore.SkeletonTracing._
import com.scalableminds.webknossos.datastore.geometry.{AdditionalAxisProto, Vec2IntProto}
import com.scalableminds.webknossos.datastore.models.annotation.{AnnotationLayer, FetchedAnnotationLayer}
import com.scalableminds.webknossos.tracingstore.tracings.volume.VolumeDataZipFormat
import models.annotation.SharedParsingParameters
import models.annotation.nml.{NmlParseSuccessWithoutFile, NmlParser, NmlWriter}
import net.liftweb.common.{Box, Full}
import org.apache.commons.io.output.ByteArrayOutputStream
Expand Down Expand Up @@ -54,8 +55,10 @@ class NMLUnitTestSuite @Inject()(nmlParser: NmlParser) extends PlaySpec {
.parse(
"",
new ByteArrayInputStream(array),
overwritingDatasetId = None,
isTaskUpload = true,
SharedParsingParameters(useZipName = false,
overwritingDatasetId = None,
userOrganizationId = "testOrganization",
isTaskUpload = true),
Comment on lines +58 to +61
Copy link
Contributor

@coderabbitai coderabbitai bot Dec 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🛠️ Refactor suggestion

Add test coverage for organization fallback behavior.

While the parameter changes look good, considering the PR's objective to "Fix nml upload with only ds name in nml by using user's orga as fallback", we should add test cases to verify this specific behavior.

Consider adding test cases that:

  1. Verify successful parsing when only dataset name is provided (without datasetId)
  2. Confirm the fallback to user's organization works as expected
  3. Test edge cases like missing or invalid organization values

Example test case structure:

"NML Parser" should {
  "use user's organization as fallback when only dataset name is provided" in {
    // Setup tracing without datasetId
    val tracingWithoutDatasetId = dummyTracing.copy(/* setup without datasetId */)
    
    writeAndParseTracing(tracingWithoutDatasetId) match {
      case Full(NmlParseSuccessWithoutFile(Some(tracing), _, _, _, _)) =>
        // Verify organization fallback worked
        assert(/* verify organization fallback */)
      case _ => fail("Should succeed with organization fallback")
    }
  }
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@fm3 Writing such a test is rather difficult as the nmls for the tests are generated with NmlWriter().toNmlStream which requires an organization id. This would mean to rewrite toNmlStream which is not preferable as one should always supply the toNmlStream with an organization. But writing a copy of toNmlSteam that supports an optional organizationId is also not preferable :/

=> I skipped this for now but it should be useful for the future to avoid this regression bug introduced by #8075

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like the humans are having a chat. I'll hop back into my burrow for now. If you need me again, just tag @coderabbitai in a new comment, and I'll come hopping out!

Copy link
Contributor Author

@MichaelBuessemeyer MichaelBuessemeyer Dec 16, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what are your thoughts on this?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I’d say this is part of the broader problem of testing the libs and webknossos together. That’s a topic we’ll definitely discuss next year.

basePath = None
)(messagesProvider, scala.concurrent.ExecutionContext.global, GlobalAccessContext)
.futureBox,
Expand Down