Optimization #28

777Denoiser · 2024-10-28T10:26:26Z

No description provided.

jsquyres

It is unexpected that you closed #26 and then opened this PR (from the same source branch).

I think nearly all of my comments on #26 still apply to this PR.

777Denoiser · 2024-11-03T07:55:08Z

I answered and resolved the comments on #26 and will get to creating the 2 requested unit tests by professor and create an extra third unit test, then I will then start working on the EDIS poster if all checks pass.

777Denoiser · 2024-11-03T09:34:17Z

                 Added a 4 Multiple Unit Tests for the Optimization Portion which were all verified to work by the CI Test. In optimization_test.go, there are 4 distinct unit tests added under the main TestDownloadOptimization function:

"Valid File Not Redownloaded"
"Invalid File Redownloaded"
"Multiple Resources"
"Resource Management"

                 Each test focuses on a specific aspect of the optimization functionality, and they run as subtests using t.Run() within the main test function. The test outputs clearly show all 4 tests executing and passing successfully.

                 The optimization unit tests in optimization_test.go serve as critical validation for the download caching and verification features in the grabit repository. These tests ensure that the optimization logic works correctly across different scenarios that users might encounter.
                 
                 The "Valid File Not Redownloaded" test verifies a key optimization feature - when a file exists and its integrity matches the expected hash, grabit skips unnecessary downloads. This test confirms that the system properly validates existing files and avoids redundant network operations, improving performance and reducing bandwidth usage.
                 
                 The "Invalid File Redownloaded" test validates the security aspect of the optimization. When a file's integrity check fails (simulated by corrupting the file), the system correctly identifies the mismatch and triggers a fresh download. This ensures that compromised or corrupted files are automatically replaced with valid versions, maintaining security while still leveraging optimization when possible.
                 
                 The "Multiple Resources" test demonstrates how the optimization handles batch operations. It verifies that the system correctly manages multiple files, applying the optimization logic independently to each resource. This test ensures that optimization works at scale and handles complex scenarios where multiple files need different treatment based on their integrity status.
                 
                 The "Resource Management" test validates the underlying resource tracking functionality that supports the optimization feature. It confirms that resources can be properly added, verified, and removed from the lock file, which is essential for maintaining accurate records of which files should be optimized.
                 
                 Together, these tests form a comprehensive validation suite for the optimization features, ensuring reliable performance, security, and correct behavior across various real-world usage patterns. The tests use a controlled HTTP test server to provide consistent, reproducible results while validating both the optimization logic and its integration with grabit's existing resource management capabilities.

test/utils.go

jsquyres

Overall, I think the sentiment is the same here: some of these are interesting changes, but we really want to see a PR that is only the "don't re-download if the file already exists in the filesystem and validates properly".

Please move the other improvements / fixes to other PRs where they can be discussed separately.

cmd/dowload_test.go

internal/resource.go

test/utils.go

internal/lock.go

internal/resource.go

internal/lock.go

internal/optimization_test.go

internal/resource.go

777Denoiser · 2024-11-13T07:11:35Z

hey thank you for listening to, and reading my comments :)

internal/resource.go

cmd/dowload_test.go

internal/resource.go

rabadin · 2024-11-13T16:07:09Z

internal/resource.go

+	// Remove existing file if present
+	os.Remove(fileName)
+
+	// Create new file with immediate close


Why the need to do this?

You've marked this resolved but you have not answered my question.

You've marked this resolved but you have not answered my question.

this is not included and has been deleted in the final 1 commit apologies for not communicating that with you raphael, last week in the last few commits i did it was fixed

| this is not included and has been deleted in the final 1 commit apologies for not communicating that with you raphael, last week in the last few commits i did it was fixed

Do you mean that this code was removed? I see it's still very much here.

oh! i was unable to find it in the resource.go when i did the search finder in the go file editor, i could have sworn i removed os.Remove(fileName), i have double checked to not find it in the resource.go file as of today,

let me know which line do you see it on.

wait it moved since i updated it in the previous commits

ive deleted it as its supposed to return the file name it should not remove the file name that would break the code haha your right

Code is still here...

internal/resource.go

777Denoiser · 2024-11-17T14:49:27Z

thank you for being patient with this PR, i fixed most of the CI test errors :)

777Denoiser · 2024-11-18T02:29:12Z

may i get the CI test to run please, thank you so much and apologies for bugging you i just want to make sure i get my part of the code completed before EDIS
@rabadin

internal/lock_test.go

777Denoiser · 2024-11-19T08:22:58Z

Hey! after our meeting today i decided to remove and revert some things that we discussed in our meeting today @jsquyres

After reverting my code that i did to fix the CI test on lock_test.go and delete_test.go leaving just resource.go and dowload_test.go.

I now have the same error that i had before i had made the changes to my code to fix the CI test so that it would pass,

now i have a "go test ./cmd -v" error in:

FAIL: TestRunDownloadMultipleErrors
FAIL: TestRunDownloadWithTags
FAIL: TestRunDelete

These were the tests that i had fixed by adding lock_test.go and delete_test.go.

I am 100% positive that the CI test will not pass because of this change by deleting the other go files.

I am not sure how to proceed further to closing this PR i would like some guidance. @rabadin @jsquyres

Thank You!!

rabadin · 2024-11-19T08:29:33Z

I am not sure how to proceed further to closing this PR i would like some guidance. @rabadin @jsquyres

The error is in red in the CI run output: Error: cmd/dowload_test.go:12:2: "runtime" imported and not used.
If you remove this the CI passes.

777Denoiser · 2024-11-20T08:07:41Z

I am not sure how to proceed further to closing this PR i would like some guidance. @rabadin @jsquyres

The error is in red in the CI run output: Error: cmd/dowload_test.go:12:2: "runtime" imported and not used. If you remove this the CI passes.

thank you so much for letting me know, i accidentally figured it out while checking the goland ID, i removed it and hopefully the CI test passes now :)

best regards

internal/resource.go

cmd/dowload_test.go

777Denoiser · 2024-11-27T13:32:10Z

if this CI test passes i will squash my 30 commits down to one commit hopefully without error

777Denoiser · 2024-11-27T16:20:44Z

@rabadin @jsquyres

Hello Raphael,

Me and Dr.Squyres during our meeting today we figured out the problem with why the commits were not squashing after a bit of troubleshooting,

Thanks to Dr.Squyres and his help I managed to get the 35 commits squashed down to 22 commits then 1 commit, all the while preserving the changes in the process.

We both have requested a review for this PR on your end to approve this PR for the merge.

Best Wishes and Happy Thanksgiving!

rabadin · 2024-11-29T08:20:44Z

internal/resource.go

+	// Remove existing file if present
+	os.Remove(fileName)
+
+	// Create new file with immediate close


You've marked this resolved but you have not answered my question.

internal/resource.go

rabadin · 2024-12-02T08:39:10Z

internal/resource.go

+	// Remove existing file if present
+	os.Remove(fileName)
+
+	// Create new file with immediate close


| this is not included and has been deleted in the final 1 commit apologies for not communicating that with you raphael, last week in the last few commits i did it was fixed

Do you mean that this code was removed? I see it's still very much here.

internal/resource.go

rabadin · 2024-12-03T08:13:27Z

internal/resource.go

+	// Remove existing file if present
+	os.Remove(fileName)
+
+	// Create new file with immediate close


Code is still here...

777Denoiser · 2024-12-04T00:41:37Z

this is what i see on my end for the lines 74 to 78, maybe you are referencing an older resource.go,

i made sure to remove any instances of os.Remove(fileName).

jsquyres · 2024-12-04T02:19:45Z

this is what i see on my end for the lines 74 to 78, maybe you are referencing an older resource.go,

i made sure to remove any instances of os.Remove(fileName).

Right, and what you see is what @rabadin is asking about: you deleted this line:

return getUrl(u, fileName, ctx)

and replaced it with a bunch of new code. There's no reason to do that on this PR. You can go ahead and remove all that new code from GetUrlToDir() and just put that one line back.

777Denoiser · 2024-12-04T06:22:36Z

@jsquyres not a problem at all, thank you for pointing that out, i was confused so thanks! i managed to delete the code i added and returned the line back to where it was before.

jsquyres · 2024-12-04T13:03:07Z

@jsquyres not a problem at all, thank you for pointing that out, i was confused so thanks! i managed to delete the code i added and returned the line back to where it was before.

Minor problem -- it doesn't compile. Can you fix?

Error: internal/resource.go:71:2: declared and not used: fileName
Error: internal/resource.go:72:20: undefined: filename

jsquyres · 2024-12-04T13:12:15Z

Still got some compile errors:

Error: cmd/dowload_test.go:150:41: not enough arguments in call to lock.Download
	have (string, nil, nil, string)
	want (string, []string, []string, string, bool)
Error: cmd/dowload_test.go:1[7](https://github.com/cisco-open/grabit/actions/runs/12160436133/job/33912772022?pr=28#step:4:8)6:41: not enough arguments in call to lock.Download
	have (string, nil, nil, string)
	want (string, []string, []string, string, bool)

jsquyres · 2024-12-04T13:32:10Z

I think the confusing point here is:

This PR compiles and runs the tests ok.
BUT CI is failing.

Why?

The reason why is because github is effectively making a pretend merge commit from this branch and main, and then doing the compile and running the tests there. That's where the compile fail / test fails are occurring.

My suggestion would be to rebase this PR on top of main. Then build and run the tests, and you should see the same failures that CI is showing.

777Denoiser · 2024-12-06T02:27:30Z

i am not sure but i hope i fixed the problem that was with the PR, even though the CI test passes i know there will be conflicts which is why i am not sure.

rabadin · 2024-12-06T10:11:26Z

i am not sure but i hope i fixed the problem that was with the PR, even though the CI test passes i know there will be conflicts which is why i am not sure.

Conflicts happen when something lands on the target branch that modifies the exact same lines that your change touches. Unless something lands before your PR does, there won't be conflicts.

rabadin · 2024-12-06T10:12:23Z

@777Denoiser Can you please squash your commits into just one commit?

777Denoiser · 2024-12-07T05:02:11Z

I've been trying on my own to squash the 6 commits and I'm having trouble doing it without creating useless commits, I got successful rebase on my end several times and to no luck.. wants to tell you this afters hours of trying...

if a file is already i the file system and it validates, we do not need to download it again. Update resource.go added comment back to the code Update resource.go removed os.Remove(fileName) which was previously on line 82 Update resource.go deleted my code under the geturltodir function and added the missing line that was requested dont download if file already present if a file is already i the file system and it validates, we do not need to download it again. fixed the too less calls in lock.Download go included a missing boolean value for the lock.Download call. dont download if file already present if a file is already i the file system and it validates, we do not need to download it again. Update resource.go added comment back to the code Update resource.go removed os.Remove(fileName) which was previously on line 82 Update resource.go deleted my code under the geturltodir function and added the missing line that was requested dont download if file already present if a file is already i the file system and it validates, we do not need to download it again. fixed the too less calls in lock.Download go included a missing boolean value for the lock.Download call.

777Denoiser · 2024-12-07T09:03:28Z

i believe i did it with some trial and error

- Refactor resource.go code to limit code duplication - Refactor test code to use the same format as the existing tests - Move test code to resource_test.go (since it's testing resource.go) - Improve test coverage Drive-by: fix typo in test file name.

777Denoiser force-pushed the main branch from 5bfa898 to eea6aa6 Compare October 28, 2024 10:34

jsquyres requested changes Oct 28, 2024

View reviewed changes

rabadin reviewed Nov 4, 2024

View reviewed changes

test/utils.go Outdated Show resolved Hide resolved

jsquyres requested changes Nov 4, 2024

View reviewed changes

rabadin requested changes Nov 5, 2024

View reviewed changes

rabadin requested changes Nov 13, 2024

View reviewed changes

internal/resource.go Outdated Show resolved Hide resolved

internal/resource.go Outdated Show resolved Hide resolved

cmd/dowload_test.go Outdated Show resolved Hide resolved

rabadin requested changes Nov 13, 2024

View reviewed changes

cmd/dowload_test.go Outdated Show resolved Hide resolved

rabadin requested changes Nov 13, 2024

View reviewed changes

rabadin reviewed Nov 18, 2024

View reviewed changes

internal/lock_test.go Outdated Show resolved Hide resolved

rabadin reviewed Nov 18, 2024

View reviewed changes

internal/lock_test.go Outdated Show resolved Hide resolved

rabadin reviewed Nov 22, 2024

View reviewed changes

internal/resource.go Outdated Show resolved Hide resolved

jsquyres reviewed Nov 26, 2024

View reviewed changes

cmd/dowload_test.go Show resolved Hide resolved

777Denoiser force-pushed the main branch 2 times, most recently from 30dccb6 to d2e3ce3 Compare November 27, 2024 16:09

rabadin requested changes Nov 29, 2024

View reviewed changes

rabadin requested changes Dec 2, 2024

View reviewed changes

rabadin requested changes Dec 3, 2024

View reviewed changes

777Denoiser mentioned this pull request Dec 4, 2024

Code is still here... #31

Closed

777Denoiser requested review from jsquyres and rabadin December 6, 2024 09:57

777Denoiser force-pushed the main branch from 375f243 to 446d9ec Compare December 7, 2024 09:02

Adjust code and test

48f8a9b

- Refactor resource.go code to limit code duplication - Refactor test code to use the same format as the existing tests - Move test code to resource_test.go (since it's testing resource.go) - Improve test coverage Drive-by: fix typo in test file name.

rabadin approved these changes Dec 11, 2024

View reviewed changes

jsquyres approved these changes Dec 11, 2024

View reviewed changes

rabadin merged commit 3f21120 into cisco-open:main Dec 11, 2024
1 check passed

rabadin mentioned this pull request Dec 13, 2024

Skip download if file already present #6

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimization #28

Optimization #28

777Denoiser commented Oct 28, 2024 •

edited by rabadin

Loading

jsquyres left a comment

777Denoiser commented Nov 3, 2024

777Denoiser commented Nov 3, 2024

jsquyres left a comment

777Denoiser commented Nov 13, 2024

rabadin Nov 13, 2024

rabadin Nov 29, 2024

777Denoiser Dec 1, 2024

rabadin Dec 2, 2024

777Denoiser Dec 3, 2024

777Denoiser Dec 3, 2024

777Denoiser Dec 3, 2024

rabadin Dec 3, 2024

777Denoiser commented Nov 17, 2024

777Denoiser commented Nov 18, 2024 •

edited

Loading

777Denoiser commented Nov 19, 2024

rabadin commented Nov 19, 2024

777Denoiser commented Nov 20, 2024

777Denoiser commented Nov 27, 2024

777Denoiser commented Nov 27, 2024

rabadin Nov 29, 2024

rabadin Dec 2, 2024

rabadin Dec 3, 2024

777Denoiser commented Dec 4, 2024

jsquyres commented Dec 4, 2024

777Denoiser commented Dec 4, 2024

jsquyres commented Dec 4, 2024

jsquyres commented Dec 4, 2024

jsquyres commented Dec 4, 2024

777Denoiser commented Dec 6, 2024

rabadin commented Dec 6, 2024

rabadin commented Dec 6, 2024

777Denoiser commented Dec 7, 2024

777Denoiser commented Dec 7, 2024

Optimization #28

Optimization #28

Conversation

777Denoiser commented Oct 28, 2024 • edited by rabadin Loading

jsquyres left a comment

Choose a reason for hiding this comment

777Denoiser commented Nov 3, 2024

777Denoiser commented Nov 3, 2024

jsquyres left a comment

Choose a reason for hiding this comment

777Denoiser commented Nov 13, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

777Denoiser commented Nov 17, 2024

777Denoiser commented Nov 18, 2024 • edited Loading

777Denoiser commented Nov 19, 2024

rabadin commented Nov 19, 2024

777Denoiser commented Nov 20, 2024

777Denoiser commented Nov 27, 2024

777Denoiser commented Nov 27, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

777Denoiser commented Dec 4, 2024

jsquyres commented Dec 4, 2024

777Denoiser commented Dec 4, 2024

jsquyres commented Dec 4, 2024

jsquyres commented Dec 4, 2024

jsquyres commented Dec 4, 2024

777Denoiser commented Dec 6, 2024

rabadin commented Dec 6, 2024

rabadin commented Dec 6, 2024

777Denoiser commented Dec 7, 2024

777Denoiser commented Dec 7, 2024

777Denoiser commented Oct 28, 2024 •

edited by rabadin

Loading

777Denoiser commented Nov 18, 2024 •

edited

Loading