Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement file uploading endpoint #29

Merged
merged 31 commits into from
Sep 4, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
31 commits
Select commit Hold shift + click to select a range
129a38d
Clean up docker run example and add :rw
Xkonti Aug 27, 2023
b751d90
Clean up stack entries
Xkonti Aug 27, 2023
5c52476
Update the roadmap with endpoint verbs and paths
Xkonti Aug 27, 2023
c053984
Update roadmap
Xkonti Aug 27, 2023
184f84c
Create file uploading endpoint
Xkonti Aug 27, 2023
97477d3
Allow binary stream file uploads
Xkonti Aug 28, 2023
2cf19ce
Cover more edge cases during file upload
Xkonti Aug 29, 2023
de58c9c
Update pathUtils tests
Xkonti Aug 29, 2023
d26ccf2
Create basic testing file system and use it to test directory checking
Xkonti Aug 29, 2023
05b077d
Update tests to use the test file system
Xkonti Aug 29, 2023
6f3bc2d
Rework tests to use temporary file system
Xkonti Aug 30, 2023
c266a82
Test file writing utility
Xkonti Aug 30, 2023
fde6cb1
Update bun types
Xkonti Sep 4, 2023
a565f8d
Add permutations generator
Xkonti Sep 4, 2023
9ccbbdb
Use permutations in file writing tests
Xkonti Sep 4, 2023
e59e9e2
Allow setting body in requestBuilder
Xkonti Sep 4, 2023
21620c0
Fix empty file contents handling
Xkonti Sep 4, 2023
41f9150
Actually attach body to request
Xkonti Sep 4, 2023
f853dcb
Add initial file upload tests
Xkonti Sep 4, 2023
736e942
Remove `skip` to allow other tests to run
Xkonti Sep 4, 2023
905496c
Remove unused code
Xkonti Sep 4, 2023
5043f5b
Improve content type sanitation
Xkonti Sep 4, 2023
498cc46
Promote error messages to constants
Xkonti Sep 4, 2023
dda765d
Add more uploading tests
Xkonti Sep 4, 2023
57f9164
Expect 400 or 422 when testing empty uploads
Xkonti Sep 4, 2023
f8227cc
Don't parse JSON requests when uploading files
Xkonti Sep 4, 2023
eb53ebf
Implement rest of file uploading tests
Xkonti Sep 4, 2023
01414a5
Lower test coverage threshold to 0.7
Xkonti Sep 4, 2023
4415038
Update readme with file upload info
Xkonti Sep 4, 2023
49ce20f
Bump minor version
Xkonti Sep 4, 2023
8b32a9c
Update Elysia.js
Xkonti Sep 4, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 59 additions & 24 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,11 @@ A simple RESTful API that allows you to manipulate files and directories.
The easiest way to run the File API is to use the Docker image. The image is available on [Docker Hub](https://hub.docker.com/r/xkonti/file-api):

```bash
docker run -d -p 3210:3000 -e API_KEY=YOUR_API_KEY -v /your/directory/to/mount:/data xkonti/file-api:latest
docker run -d \
-p 3210:3000 \
-e API_KEY=YOUR_API_KEY \
-v /your/directory/to/mount:/data:rw \
xkonti/file-api:latest
```

Available configuration options:
Expand All @@ -37,9 +41,9 @@ Each HTTP request will be first checked for API key. If the API key is not provi

### List files and directories in a directory

You can list files and/or directories by using the `/list` endpoint with the following parameters:
You can list files and/or directories by using the `GET /list` endpoint with the following parameters:

- `path` - the path of the directory to list (remember to encode it)
- `path` - the path of the directory to list (remember to url-encode it)
- `dirs` - whether to include directories or not (default: `false`). If directories are listed their contents will be nested.
- `depth` - how deep to list directories. The default `1` will list only files and directories within the specified directory - no contents of subdirectories will be listed.

Expand Down Expand Up @@ -181,9 +185,9 @@ You can list files and/or directories by using the `/list` endpoint with the fol

### Download a file

You can download a specific file by using the `/file` endpoint with the following parameters:
You can download a specific file by using the `GET /file` endpoint with the following parameters:

- `path` - the path of the file to download (remember to encode it)
- `path` - the path of the file to download (remember to url-encode it)

If the file does not exist or a path is pointing to a directory, a `404 Not Found` response will be returned.

Expand All @@ -193,29 +197,61 @@ If the file does not exist or a path is pointing to a directory, a `404 Not Foun
- `curl --request GET --url 'http://localhost:3000/file?path=%2FExpenses%202022.xlsx'` - download the `Expenses 2022.xlsx` file (`%2FExpenses%202022.xlsx` is the encoded `/Expenses 2022.xlsx`)
</details>

### Upload a file

You can upload a specific file by using the `POST /file` endpoint with the following parameters:

- `path` - the destination path of the file to upload (remember to url-encode it)
- `overwrite` - whether to overwrite the file if it already exists (default: `false`)

All the parent directories of the destination path will be created automatically if they do not exist.

The file contents should be sent in the request body in one of the following ways:

- `multipart/form-data` - the file contents should be sent as a `file` field
- `application/octet-stream` - the file contents should be sent as the request body

Possible responses:

- `201` - the file was uploaded successfully
- `400` - the request was malformed in some way:
- the `path` parameter is missing
- the `path` parameter is pointing to a directory
- the file contents are missing
- `415` - unsupported content type
- `422` - unrecognized file contents format

<details>
<summary>Examples [click to expand]</summary>

- `curl --request POST --url 'http://localhost:3000/file?path=%2FExpenses%202022.xlsx' --header 'Content-Type: application/octet-stream' --data-binary @/path/to/file` - upload the `Expenses 2022.xlsx` file (`%2FExpenses%202022.xlsx` is the encoded `/Expenses 2022.xlsx`)
</details>

## Roadmap

### Directories

- [x] List files and directories in a directory
- [ ] Check if a directory exists
- [ ] Create a directory
- [ ] Delete a directory
- [ ] Rename a directory
- [ ] Move a directory
- [ ] Copy a directory
- [x] List files and directories in a directory as a flat list: `GET /list`
- [ ] List files and directories in a directory as a tree: `GET /tree`
- [ ] Check if a directory exists: `GET /dir/exists`
- [ ] Create a directory: `POST /dir`
- [ ] Delete a directory: `DELETE /dir`
- [ ] Rename a directory: : `POST /dir/rename`
- [ ] Move a directory: `POST /dir/move`
- [ ] Copy a directory: `POST /dir/copy`

### Files

- [ ] Check if a file exists
- [x] Download a file
- [ ] Upload a file
- [ ] Delete a file
- [ ] Create an empty file
- [ ] Rename a file
- [ ] Move a file
- [ ] Copy a file
- [ ] Get file metadata
- [ ] Check if a file exists: `GET /file/exists`
- [x] Download a file: `GET /file`
- [x] Upload a file: `POST /file`
- [ ] Delete a file: `DELETE /file`
- [ ] Create an empty file: `POST /file/touch`
- [ ] Rename a file: `POST /file/rename`
- [ ] Move a file: `POST /file/move`
- [ ] Copy a file: `POST /file/copy`
- [ ] Get file metadata: `GET /file/meta`
- [ ] Get file size: `GET /file/size`

### Permissions

Expand All @@ -235,6 +271,5 @@ If the file does not exist or a path is pointing to a directory, a `404 Not Foun

## Stack

- Runtime: Bun
- Bundler: Bun
- API framework: Elysia.js
- Runtime and bundler: [Bun](https://bun.sh/)
- API framework: [Elysia.js](https://elysiajs.com/)
Binary file modified bun.lockb
Binary file not shown.
4 changes: 1 addition & 3 deletions bunfig.toml
Original file line number Diff line number Diff line change
@@ -1,5 +1,3 @@
[test]

# always enable coverage
coverage = true
coverageThreshold = 0.8
coverageThreshold = 0.7
8 changes: 5 additions & 3 deletions package.json
Original file line number Diff line number Diff line change
@@ -1,16 +1,18 @@
{
"name": "file-api",
"version": "0.2.0",
"version": "0.3.0",
"scripts": {
"test": "echo \"Error: no test specified\" && exit 1",
"dev": "bun run --watch src/index.ts",
"prod": "bun run src/index.ts"
},
"dependencies": {
"elysia": "latest"
"elysia": "^0.6.17",
"neverthrow": "^6.0.0",
"rimraf": "^5.0.1"
},
"devDependencies": {
"bun-types": "^0.7.3"
"bun-types": "^0.8.1"
},
"module": "src/index.js"
}
2 changes: 2 additions & 0 deletions src/app.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@ import {addGetListEndpoint} from './endpoints/list/getList';
import {addApiKeyGuard} from './endpoints/guards';
import {addErrorHandling} from './endpoints/errors';
import {addGetFileEndpoint} from './endpoints/file/getFile';
import {addUploadFileEndpoint} from './endpoints/file/uploadFile';

export function buildApp() {
let app = new Elysia();
Expand All @@ -14,5 +15,6 @@ export function buildApp() {
// Endpoints
addGetListEndpoint(app);
addGetFileEndpoint(app);
addUploadFileEndpoint(app);
return app;
}
20 changes: 20 additions & 0 deletions src/constants/commonResponses.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
// Success
export const fileUploadSuccessMsg = 'File uploaded successfully';

// Bad requests

export const dirNotExistsMsg = 'The directory does not exist';
export const fileAlreadyExistsMsg = 'The file already exists';
export const fileEmptyMsg = 'The file is empty';
export const fileMustNotEmptyMsg = 'File contents cannot be empty';
export const provideValidPathMsg = 'You must provide a valid path to a file';
export const unsupportedContentTypeMsg = 'Unsupported content type';
export const unsupportedBinaryDataTypeMsg = 'Unsupported binary data type';
export const invalidFileFormatInFormDataMsg = 'Invalid file format in form data';
export const noFileFieldInFormDataMsg = 'No file field in form data';
export const invalidFormDataStructure = 'Invalid structure for multipart/form-data';

// Internal server errors
export const dirCreateFailMsg = 'Could not create directory';
export const unknownErrorMsg = 'An unknown error occurred';
export const uploadErrorMsg = 'An error occurred while uploading the file';
52 changes: 22 additions & 30 deletions src/endpoints/file/getFile.test.ts
Original file line number Diff line number Diff line change
@@ -1,17 +1,9 @@
import {beforeEach, expect, test} from 'bun:test';
import {afterEach, beforeEach, expect, test} from 'bun:test';
import {buildApp} from '../../app';
import {setConfig} from '../../utils/config';
import {RequestBuilder} from '../../utils/requestBuilder';
import {UrlBuilder} from '../../utils/urlBuilder';
import {
dir1Name,
emptyFileName,
illegalPaths,
loremFileContents,
loremFileName,
pathToNowhere,
testingDirectoryRelativePath,
} from '../../testing/testingUtils.test';
import {fs1Files, fs1TestDirectoryContents, illegalPaths} from '../../testing/constants';
import {createTestFileSystem, destroyTestFileSystem} from '../../testing/testFileSystem';

function buildFileRequest(path: string | null) {
let url = new UrlBuilder('http://localhost/file');
Expand All @@ -25,36 +17,38 @@ function buildFileRequest(path: string | null) {
// App instance
let app: ReturnType<typeof buildApp>;

beforeEach(() => {
// Update config
setConfig({
apiKey: 'test',
dataDir: './',
});
const files = fs1Files;
const filesList = Object.values(files);
const testDirectoryContents = fs1TestDirectoryContents;

// Create app
beforeEach(async () => {
await createTestFileSystem(testDirectoryContents);
app = buildApp();
});

// TODO: Should return a 200 if the file is found
afterEach(async () => {
await destroyTestFileSystem();
});

test('should return file contents if the file is found', async () => {
const filePath = `${testingDirectoryRelativePath}/${loremFileName}`;
const request = buildFileRequest(filePath);
const response = await app.handle(request);
expect(response.status).toBe(200);
const bodyAsText = await response.text();
expect(bodyAsText).toBe(loremFileContents);
for (const fileDef of filesList) {
const request = buildFileRequest(fileDef.relativePath);
const response = await app.handle(request);
expect(response.status).toBe(200);
const bodyAsText = await response.text();
expect(bodyAsText).toBe(fileDef.contents);
}
});

test('should return a 404 if the file is not found', async () => {
const filePath = `${testingDirectoryRelativePath}/${pathToNowhere}`;
const filePath = `/path/to/nowhere`;
const request = buildFileRequest(filePath);
const response = await app.handle(request);
expect(response.status).toBe(404);
});

test('should return a 404 if the path is not a file', async () => {
const filePath = `${testingDirectoryRelativePath}/${dir1Name}`;
const filePath = `/TerryPratchett`;
const request = buildFileRequest(filePath);
const response = await app.handle(request);
expect(response.status).toBe(404);
Expand All @@ -80,10 +74,8 @@ test('should return a 400 when illegal path', async () => {
}
});

// TODO: Should return a 200 if the file is empty
test('should return a 200 if the file is empty', async () => {
const filePath = `${testingDirectoryRelativePath}/${emptyFileName}`;
const request = buildFileRequest(filePath);
const request = buildFileRequest(files.emptiness.relativePath);
const response = await app.handle(request);
expect(response.status).toBe(200);
const bodyAsText = await response.text();
Expand Down
Loading