Multipart request with file array does not work as expected

[mattiaforc]

Hi everyone,
I'm struggling with multipart/form-data when making requests with arrays of file as content.
My backend accepts a multipart request with two keys: files and info. The former is an array of files (with MIME type application/pdf) and the latter is just a JSON object with some metadata (which is negligible for this situation).
The problem is that I can not create a multipart request with multiple files, because it sends just one file.

Environment

• k6 version: k6 v0.27.0 (2020-07-14T12:19:13+0000/v0.27.0-0-g6fa889d0, go1.14.4, linux/amd64)
• OS and version: Ubuntu 18.04.4 LTS, Kernel: Linux 5.4.0-42-generic, Architecture: x86-64

Expected Behavior

Just like in Postman requests, I expect to be able to define multiple files under the same key files, which are sent and then received by the backend as an array of files.

This is also possible in JS (according to MDN web docs, see also this SO question):

The append() method of the FormData interface appends a new value onto an existing key inside a FormData object, or adds the key if it does not already exist. The difference between FormData.set and append() is that if the specified key already exists, FormData.set will overwrite all existing values with the new one, whereas append() will append the new value onto the end of the existing set of values.

Actual Behavior

My guess is that this is the most correct code (note that the file is read during init phase, and used later on in the rest of the snippet)

// init
const file1 = open('loadTestFile1.pdf', 'b')
const file2 = open('loadTestFile2.pdf', 'b')

// actual VU function
let files = [
    http.file(file1, "loadTestFile1.pdf", "application/pdf"), 
    http.file(file2, "loadTestFile2.pdf", "application/pdf")
]

let info = http.file(JSON.stringify({
      "irrelevant": 1234
}), "info", "application/json")

let payloadData = { files, info }

let res = http.post(url, payloadData, { headers: { 'Authorization': 'Basic value' } })

which results in a single larger files object. This is a problem not only because it creates only one object but also due to my backend file size limit (10MB) being exceeded (note that the limit is per-file, so I have to be able to send two 8MB files).

I also tried to do the following, with files resulting in only the last element (so the backend receives only 1 file):

let file1 = http.file(file1, "loadTestFile1.pdf", "application/pdf")
let file2 = http.file(file2, "loadTestFile2.pdf", "application/pdf")
let info = ... // same as above

let payloadData = {files: file1, files: file2, info: info}

I also noticed that the same two files, when sent with k6 and with Postman, have different sizes. The one I send with k6 trigger the backend size limit even if they are just ~7MB, while the Postman request works just fine. What could this possibly be?

Thanks in advance!

[idengaurav]

Any work around for this?? I just want to upload multiple files in the same key as multipart/formdata.

[marcorosi]

I'm interested too

[imiric]

Hi there,

unfortunately this is part of known deficiencies with the current HTTP API that is due for an overhaul to fix this and other issues. See #747, #1382, etc.

One issue in this case is being unable to specify the same field name for multiple files, as that won't work as expected with plain JS objects. We'd need a FormData implementation, and I'm not sure if a polyfill would help in this case.

This is high on our priority list to fix, but in the meantime you can build the multipart/form-data request body manually.

Here's something that works for this example:

import { sleep, group } from 'k6'
import http from 'k6/http'
import encoding from 'k6/encoding';

let files = [
    {filename: '1.pdf', content_type: 'application/pdf', data: encoding.b64encode(open('1.pdf', 'b'))},
    {filename: '2.pdf', content_type: 'application/pdf', data: encoding.b64encode(open('2.pdf', 'b'))},
]

let info = {
  filename: 'info', content_type: 'application/json',
  data: JSON.stringify({ 'irrelevant': 1234 })
};

function makeMultipart(file, key, transferEncoding) {
    return `--boundary\r\nContent-Disposition: form-data; name="${key}"; filename="${file.filename}"\r\n` +
      `Content-Type: ${file.content_type}\r\n${transferEncoding || ''}\r\n` + file.data + `\r\n\r\n`;
}

export default function() {
  let body = '';

  // Add the files under the same key
  for (const file of files) {
    body += makeMultipart(file, 'files', 'Content-Transfer-Encoding: base64\r\n');
  }

  // Add other JSON data
  body += makeMultipart(info, 'info');

  // Close the multipart body
  body += `--boundary--\r\n`;

  http.post('http://localhost:9000/post', body, { headers: { 'Content-Type': 'multipart/form-data; boundary=boundary' } });
}

Note that because of issues with handling binary data (#1020) this is encoding and sending base64, which your server should interpret and decode.

An alternative to this suggested by @na-- if you want to avoid base64 might be constructing the body outside of k6 and submitting the result of open('my-body.bin', 'b'). I haven't tested this, but it should work.

[imiric]

This is possible with the FormData polyfill, which is the recommended way to submit multipart requests, so I'll close this issue.

In the future, we might integrate FormData into k6 as part of the new HTTP API (initial design document), but we haven't decided on the details yet.