Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix(otelgin): remove multipartform temporary file #6609

Open
wants to merge 7 commits into
base: main
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ This project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.htm

- Fix error logged by Jaeger remote sampler on empty or unset `OTEL_TRACES_SAMPLER_ARG` environment variable (#6511)
- Relax minimum Go version to 1.22.0 in various modules. (#6595)
- Fix issue in `otelgin` middleware where temporary files for multipart/form-data were not being removed. (#6609)

<!-- Released section -->
<!-- Don't change this section unless doing release -->
Expand Down
14 changes: 14 additions & 0 deletions instrumentation/github.com/gin-gonic/gin/otelgin/gintrace.go
Original file line number Diff line number Diff line change
Expand Up @@ -85,6 +85,20 @@

// pass the span through the request context
c.Request = c.Request.WithContext(ctx)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't we be fixing this rather more easily with the following:

Suggested change
c.Request = c.Request.WithContext(ctx)
c.Request = c.Request.Clone(ctx)

WithContext performs a shallow clone, and therefore doesn't clone the multipart form.
However, Clone performs a deep copy, and makes an explicit clone of MultipartForm.

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem here is, go http request will call MultipartForm.RemoveAll() automatically when request finishes. However, the otelgin middleware creates a new request, and temporary files will be stored in this new request's MultipartForm field. Unfortunately, no one is calling MultipartForm.RemoveAll() on this new request.

A deep clone wouldn’t resolve the problem because the MultipartForm is also copied. A shallow clone might seem like it should work since the MultipartForm field is shared, but it doesn’t. This is because the MultipartForm field is lazily initialized—it’s only assigned a value when you parse the form. As a result, in the middleware, the field is nil.

Related issues: golang/go#58809, labstack/echo#2413

I think other OpenTelemetry middlewares, such as otelmux and otelehco, might also have the same issue.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What if we parse the form before making the new request?
Wouldn't we share the object then?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this approach could work, but it feels a bit unusual to enforce form parsing in the middleware.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Having to force remove multipart files is a bit unusual too.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm probably more in favor of doing the initialization on the original request as well, explicitly handling clean up logic does seem a bit strange

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@dmathieu maybe we should discuss this in the SIG since it might have implications in the other instrumentation modules

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I won't be able to attend this week's meeting (it's the week where it's late for EU). But yes, feel free to bring it there.

defer func() {
// as we have created new http.Request object we need to make sure that temporary files created to hold MultipartForm
// files are cleaned up. This is done by http.Server at the end of request lifecycle but Server does not
// have reference to our new Request instance therefore it is our responsibility to fix the mess we caused.
//
// This means that when we are on returning path from handler middlewares up in chain from this middleware
// can not access these temporary files anymore because we deleted them here.
if c.Request.MultipartForm != nil {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm wondering, if there's another middleware layer outside this one, would it be removed because of this?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's hard to say, but in all cases, regardless of how many middlewares are chained, temporary files should be removed. I didn't check the return error of RemoveAll here, which aligns with the behavior in both Go's http package and echo-contrib. I believe each middleware should independently attempt to remove temporary files.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

err := c.Request.MultipartForm.RemoveAll()
if err != nil {
otel.Handle(err)
}

Check warning on line 99 in instrumentation/github.com/gin-gonic/gin/otelgin/gintrace.go

View check run for this annotation

Codecov / codecov/patch

instrumentation/github.com/gin-gonic/gin/otelgin/gintrace.go#L98-L99

Added lines #L98 - L99 were not covered by tests
}
}()

// serve the request to the next middleware
c.Next()
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -6,10 +6,14 @@
package test

import (
"bytes"
"errors"
"html/template"
"io/fs"
"mime/multipart"
"net/http"
"net/http/httptest"
"runtime"
"strconv"
"testing"

Expand Down Expand Up @@ -430,3 +434,45 @@ func TestWithGinFilter(t *testing.T) {
assert.Len(t, sr.Ended(), 1)
})
}

func TestTemporaryFormFileRemove(t *testing.T) {
if runtime.GOOS == "windows" {
// Windows sometimes refuses to remove a file that was just closed.
t.Skip("https://go.dev/issue/25965")
}
sr := tracetest.NewSpanRecorder()
provider := sdktrace.NewTracerProvider(sdktrace.WithSpanProcessor(sr))

router := gin.New()
router.MaxMultipartMemory = 1
router.Use(otelgin.Middleware("foobar", otelgin.WithTracerProvider(provider)))
var fileHeader *multipart.FileHeader
router.POST("/upload", func(c *gin.Context) {
_, err := c.FormFile("file")
require.NoError(t, err)
fileHeader = c.Request.MultipartForm.File["file"][0]
_, err = fileHeader.Open()
require.NoError(t, err)
c.JSON(http.StatusOK, nil)
})

var body bytes.Buffer

mw := multipart.NewWriter(&body)
fw, err := mw.CreateFormFile("file", "file")
require.NoError(t, err)

_, err = fw.Write([]byte("hello world"))
require.NoError(t, err)
err = mw.Close()
require.NoError(t, err)
r := httptest.NewRequest("POST", "/upload", &body)
r.Header.Add("Content-Type", mw.FormDataContentType())
w := httptest.NewRecorder()

router.ServeHTTP(w, r)
assert.Len(t, sr.Ended(), 1)
require.Equal(t, http.StatusOK, w.Code)
_, err = fileHeader.Open()
require.ErrorIs(t, err, fs.ErrNotExist)
}
Loading