Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Stub does not list files if prefix is a directory #410

Closed
hbibel opened this issue Mar 23, 2023 · 0 comments
Closed

Stub does not list files if prefix is a directory #410

hbibel opened this issue Mar 23, 2023 · 0 comments

Comments

@hbibel
Copy link
Contributor

hbibel commented Mar 23, 2023

Hi all,

I found an inconsistency between the stub and the live implementations of the s3 service with regards to listing objects with a prefix. This small app highlights the issue:

package zio.s3

import zio._
import zio.s3._
import software.amazon.awssdk.regions.Region
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials
import java.net.URI
import zio.nio.file.Path

object ListingObjectsWithPrefix extends ZIOAppDefault {
  // Run in the root of this repo
  // Run docker-compose up before running this app

  override def run =
    for {
      _ <- listInMinio
      _ <- listWithStub
    } yield ()

  def listInMinio =
    list("minio").provide(
      zio.s3
        .live(
          Region.CA_CENTRAL_1,
          AwsBasicCredentials.create("TESTKEY", "TESTSECRET"),
          Some(URI.create("http://127.0.0.1:9000"))
        )
    )

  def listWithStub =
    list("stub").provide(
      stub(Path("./test-data"))
    )

  def list(label: String) =
    for {
      objs <- listObjects(
                "bucket-1",
                ListObjectOptions(prefix = Some("dir1"), maxKeys = 10, delimiter = None, starAfter = None)
              )
      _    <- Console.printLine(s"Objects using $label: " + objs.objectSummaries.map(_.key).mkString(", "))
    } yield ()

}

This will print:

Objects using minio: dir1/hello.txt, dir1/user.csv
Objects using stub:

From my understanding of the AWS API the minio output is correct. I could also reproduce it using an actual s3 bucket.

I've looked into the code and believe I found the cause of the issue. I'll create a PR shortly.

hbibel added a commit to hbibel/zio-s3 that referenced this issue Mar 23, 2023
regis-leray pushed a commit that referenced this issue Mar 27, 2023
* Add test for listing objects with path-like prefix

* #410 - Fix stub filtering by prefix and delimiter
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants