Image upload failures - Help me help you stop buffering your POSTs

[WWhitt]

Nutanix Cluster Information

Foundation 5.1.1
This would also apply to any cluster or PC version as it has nothing to do with any of these

Terraform Version

[wwhitt@phx-se-automation-dev-1 test]$ terraform -v

• Terraform v1.1.9 on linux_amd64 (also tested on v1.1.2)
• provider registry.terraform.io/nutanix/nutanix v1.5.0-beta.2

Affected Resource(s)

• nutanix_foundation_image
• foundation_file_management_service
• client/client.go
• v3_service.go

• Terraform Configuration Files

[wwhitt@phx-se-automation-dev-1 test]$ cat main.tf
terraform {
  required_providers {
    nutanix = {
      source = "nutanix/nutanix"
      version = "1.5.0-beta.2"
    }
  }
}

provider "nutanix" {
  foundation_endpoint= "10.55.76.52"
}

// upload aos image
resource "nutanix_foundation_image" "image1" {
  source = "/home/wwhitt/tmp/AOS/6.1/nutanix_installer_package-release-fraser-6.1-stable-d0d01a21610008feb6f566fbfe4aa767bda26e0a-x86_64.tar.gz"
  filename = "nos-tempfile-6.1.tar"
  installer_type = "nos"
}

data "nutanix_foundation_nos_packages" "nos" {
  depends_on = [resource.nutanix_foundation_image.image1]
}

output "nos" {
  value = data.nutanix_foundation_nos_packages.nos
}

Panic Output

fatal error: runtime: out of memory

runtime stack:
runtime.throw({0xd30f11, 0x118c00000})
	runtime/panic.go:1198 +0x71
runtime.sysMap(0xc23a000000, 0x4298e0, 0xc00006be90)
	runtime/mem_linux.go:169 +0x96
runtime.(*mheap).grow(0x13aa460, 0x8c479)
	runtime/mheap.go:1393 +0x225
runtime.(*mheap).allocSpan(0x13aa460, 0x8c479, 0x0, 0x1)
	runtime/mheap.go:1179 +0x165
runtime.(*mheap).alloc.func1()
	runtime/mheap.go:913 +0x69
runtime.systemstack()
	runtime/asm_amd64.s:383 +0x49

goroutine 39 [running]:
runtime.systemstack_switch()
	runtime/asm_amd64.s:350 fp=0xc00054f2c8 sp=0xc00054f2c0 pc=0x461420
runtime.(*mheap).alloc(0x1188f2000, 0x8c479, 0xd3, 0x0)
	runtime/mheap.go:907 +0x73 fp=0xc00054f318 sp=0xc00054f2c8 pc=0x425c13
runtime.(*mcache).allocLarge(0xc00054f3f8, 0x1188f2000, 0x60, 0x1)
	runtime/mcache.go:227 +0x89 fp=0xc00054f378 sp=0xc00054f318 pc=0x416889
runtime.mallocgc(0x1188f2000, 0x0, 0x0)
	runtime/malloc.go:1088 +0x5c5 fp=0xc00054f3f8 sp=0xc00054f378 pc=0x40ce05
runtime.growslice(0xc0002f85d8, {0xc090356000, 0x2ce3c000, 0x2ce3c000}, 0xb38ec001)
	runtime/slice.go:261 +0x4ac fp=0xc00054f460 sp=0xc00054f3f8 pc=0x44a56c
io.ReadAll({0xe33e60, 0xc0002f85d8})
	io/io.go:631 +0xa5 fp=0xc00054f4e0 sp=0xc00054f460 pc=0x48c3a5
io/ioutil.ReadAll(...)
	io/ioutil/ioutil.go:27

Expected Behavior

It uploads an image without chewing through at least (imageSize * 2) worth of memory. Chewing up more than 1G is probably unreasonable regardless of incoming filesize. Imagine uploading a Windows qcow 😄 while it eats windows qcow * 2 memory doing it.

Actual Behavior

SIGSEGV; Fails to upload because we read the entire file into memory, then we clone that into a new buffer.... THEN we pass it to net/http. double buffering. mmmmmm

Steps to Reproduce

1. terraform apply where image size > mem+swap allocated to machine

Important Factors

*os.File implements io.reader which is what net/http is looking for, so if the only reason we're reading the entire file to spawn a new buffer is to satisfy the net/http NewRequest format... that's not necessary
Here's an example where I do the same thing against the same Foundation API endpoint, but I'm simply using base go libraries and passing the file ptr to http.Post:

{
  "md5sum": null, 
  "name": "/home/nutanix/foundation/nos/go-upload-2-aos-6.1.tar.gz", 
  "in_whitelist": true
}
[wwhitt@phx-se-automation-dev-1 test]$ cat test.go 
package main

import (
	"io"
	"fmt"
	"os"
	"net/http"
)

func main() {
	file, err := os.Open("/home/wwhitt/tmp/AOS/6.1/nutanix_installer_package-release-fraser-6.1-stable-d0d01a21610008feb6f566fbfe4aa767bda26e0a-x86_64.tar.gz")
	if err != nil { fmt.Print(err) }
	defer file.Close()

	resp, err := http.Post("http://10.55.76.52:8000/foundation/upload?installer_type=nos&filename=go-upload-2-aos-6.1.tar.gz", "application/octet-stream", file)
	defer resp.Body.Close()
	ret, err := io.ReadAll(resp.Body)
	fmt.Print(string(ret))
} 

^ the above code is stupid simple, but I've profiled both and actually streaming as shown in the example above never went above like .4% memory usage, compared to balloon -> balloon -> balloon -> crash. and as you can see, the file uploaded successfully.