Load dataset offline

[charlie1986]

Hi guys!
I'm following the books d2l java version to learning deep learning concepts. The following code shows how to load fashion mnist dataset to train the model:

int batchSize = 256;
FashionMnist trainIter = FashionMnist.builder()
.optUsage(Dataset.Usage.TRAIN)
.setSampling(batchSize, true)
.optLimit(Long.getLong("DATASET_LIMIT", Long.MAX_VALUE))
.build();
But this code requires internet connection to download the dataset from fashion mnist repo. Could we manually download the dataset and load it offline locally? How to do that?
P/s: My computer at work doesnt have internet connection.

[frankfliu]

@charlie1986

You can specify Repository in FashionMnist.builder()`:

Repository repo = Repository.newInstance("local", Paths.get(MY_LOCAL_PATH));
FashionMnist trainIter = FashionMnist.builder()
    .opeRepository(repo)
    .optUsage(Dataset.Usage.TRAIN)
    .setSampling(batchSize, true)
    .build()

[frankfliu]

@charlie1986
By the way, once you download the dataset for the first time, it will fetch the file from your cache, see: https://docs.djl.ai/master/docs/development/cache_management.html

[charlie1986]

@FrankLiu
I have changed the path to: C:/Users/charlie/.djl.ai/cache/repo/dataset/cv/ai/djl/basicdataset/fashmnist/, but an exception was thrown:
Cannot invoke "ai.djl.repository.Artifact$Item.getName()" because "item" is null.
How to resolve this?

[charlie1986]

@FrankLiu
I changed the path to: C:/Users/charlie/.djl.ai/cache/repo/
The code runs, but it cannot find the true path. I must copy all files in C:/Users/charlie/.djl.ai/cache/repo/dataset/cv/ai/djl/basicdataset/fashmnist/1.0/ to C:/Users/charlie/.djl.ai/cache/repo/dataset/cv/ai/djl/basicdataset/fashmnist/1.0/1.0/ and change all file types to .gz to make the code runs.
It is quite strange why we need to double /1.0/1.0/ and change file type?

[frankfliu]

When we download the files from remote repo, we uncompress the file, so in local you won't see .gz file.

It looks like a bug to me that we are using 1.0/1.0 in the file path.