Fix metadata parsing #1953
Merged
Fix metadata parsing #1953
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
gapra-msft
requested review from
siminsavani-msft,
gapra-msft,
tasherif-msft and
nakulkar-msft
gapra-msft
reviewed
Nov 16, 2022
common/fe-ste-models.go
Outdated
| @@ -110,7 +111,7 @@ type PartNumber uint32 | |||
| type Version uint32 | |||
| type Status uint32 | |||
|
|
|||
| //////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// | |||
| // ////////////////////////////////////////////////////////////////////////////////////////////////////////////////////// | |||
There was a problem hiding this comment.
*the comment changes specifically
There was a problem hiding this comment.
this formatting setting can be changed in GoLand.
File -> Settings -> Editor -> Code Style -> Go -> Other -> Add a leading space to comments should be unchecked.
gapra-msft
reviewed
Nov 16, 2022
siminsavani-msft
approved these changes
Jan 6, 2023
tasherif-msft
suggested changes
Jan 9, 2023
vibhansa-msft
approved these changes
Jan 12, 2023
tasherif-msft
approved these changes
Jan 18, 2023
adreed-msft
added a commit
that referenced
this pull request
Jan 23, 2023
* Add mitigation for weird NtQuerySecurityObject behavior on NAS sources (#1872) * Add check for 0 length, attempt to validate the returned object. * Change to grabbing real SD length * Add comment describing issue * Prevent infinite loop upon listing failure * Fix GCP error checking * Fix GCP disable * Fix bad URL delete (#1892) * Manipulate URLs safely * Fix folder deletion test * Prevent infinite loop upon listing failure * Fix GCP error checking * Fix GCP disable * Fail when errors listing/clearing bucket * Update MacOS testing pipeline (#1896) * fixing small typo (,) in help of jobs clean (#1899) * Microsoft mandatory file * fixing small typo (,) in help of jobs clean Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> * Implement MD OAuth testing (#1859) * Implement MD OAuth testing * Handle async on RevokeAccess, handle job cancel/failure better * Prevent parallel testing of managed disks * lint check * Prevent infinite loop upon listing failure * Fix GCP error checking * Fix GCP disable * Fail when errors listing/clearing bucket * Add env vars * Avoid revoking MD access, as it can be shared. * Fix intermittent failures * Disable MD OAuth testing temporarily. * Add "all" to documentation (#1902) * 10.16.1 patch notes (#1913) * Add bugfixes to change log. * Correct wording & punctuation * Correct version * Export Successfully Updated bytes (#1884) * Add info in error message for mkdir on Log/Plan (#1883) * Microsoft mandatory file * Add info in error message for mkdir on Log/Plan Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> * Fix fixupTokenJson (#1890) * Microsoft mandatory file * Fix fixupTokenJson Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> Co-authored-by: Adam Orosz <[email protected]> * Do not log request/response for container creation error (#1893) * Expose AZCOPY_DOWNLOAD_TO_TEMP_PATH environment variable. (#1895) * Slice against the correct string (#1927) * UX improvement: avoid crash when copying S2S with user delegation SAS (#1932) * Fix bad build + Prevent bad builds in the future (#1917) * Fix bad build + Prevent bad builds in the future * Add Windows build * Make sync use last write time for Azure Files (#1930) * Make sync use last write time for Azure Files * Implement test * 10.16.2 Changelog (#1948) * Update azcopy version * Fixed a bug where preserve permissions would not work with OAuth * Added CODEOWNERS file * Fixed issue where CPK would not be injected on retries * remove OAuth from test * Updated version check string to indicate current AzCopy version (#1969) * added codeowner * Enhance job summary with details about file/folders (#1952) * Add flag to disable version check (#1950) * darwin arm64 * Update golang version to 10.19.2 (#1925) * enable cgo * added tests * Minor fixes: More in description (#1968) * Echo auto-login failure if any * Update help for sync command to use trailing slash on directories * azcopy fail to copy 12TB file to Storage containers in Dev. The logic is used to calculate proper blockSize if it’s not provided, and due to the uint32 cast, it can’t give proper blockSize if filesize is between 50000 * (8 * 1024 * 1024) * X + 1, to 50000 * (8 * 1024 * 1024) * X + 49999. It should return 16MB instead of 8MB blockSize. Accommodated the changes suggested by Narasimha Kulkarni * Added extra logging when switching endpoints * Enable support for preserving SMB info on Linux. (#1723) * Microsoft mandatory file * Enable support for preserving SMB info on Linux. Implemented the GetSDDL/PutSDDL GetSMBProperties/PutSMBProperties methods for Linux using extended attributes. Following are the xattrs we use for fetching/setting various required info. // Extended Attribute (xattr) keys for fetching various information from Linux cifs client. const ( CIFS_XATTR_CREATETIME = "user.cifs.creationtime" // File creation time. CIFS_XATTR_ATTRIB = "user.cifs.dosattrib" // FileAttributes. CIFS_XATTR_CIFS_ACL = "system.cifs_acl" // DACL only. CIFS_XATTR_CIFS_NTSD = "system.cifs_ntsd" // Owner, Group, DACL. CIFS_XATTR_CIFS_NTSD_FULL = "system.cifs_ntsd_full" // Owner, Group, DACL, SACL. ) Majority of the changes are in sddl/sddlHelper_linux.go which implement the following Win32 APIs for dealing with SIDs. ConvertSecurityDescriptorToStringSecurityDescriptorW ConvertStringSecurityDescriptorToSecurityDescriptorW ConvertSidToStringSidW ConvertStringSidToSidW Note: I have skipped Object ACE support in sddl/sddlHelper_linux.go as those should not be used for filesystem properties, only AD object properties. Can someone confirm this? TBD: Conditional SID * Audited, fixed, tested support for "No ACL"/NO_ACCESS_CONTROL and ACL w/o any ACE Tested the following cases: c:\Users\natomar\Downloads>cd testacl // This has "No ACLs" and everyone should be allowed access. c:\Users\natomar\Downloads\testacl>touch NO_ACCESS_CONTROL.txt c:\Users\natomar\Downloads\testacl>cacls NO_ACCESS_CONTROL.txt /S:D:NO_ACCESS_CONTROL Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\NO_ACCESS_CONTROL.txt // This has "No ACLs" and everyone should be allowed access. // It additionally has the "P" (protected) flag set, but that won't have // any effect as that just prevents ACE inheritance but this ACL will // not have any ACLs due to the NO_ACCESS_CONTROL flag. c:\Users\natomar\Downloads\testacl>touch PNO_ACCESS_CONTROL.txt c:\Users\natomar\Downloads\testacl>cacls PNO_ACCESS_CONTROL.txt /S:D:PNO_ACCESS_CONTROL Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\PNO_ACCESS_CONTROL.txt // This should set DACL but with no ACEs, but since "P" is not set it // inherits ACEs from the parent dir. c:\Users\natomar\Downloads\testacl>touch empty_d.txt c:\Users\natomar\Downloads\testacl>cacls empty_d.txt /S:D: Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\empty_d.txt // This should set DACL but with no ACEs, but since "P" is set it // doesn't inherit ACEs from the parent dir and hence this will block // all users. c:\Users\natomar\Downloads\testacl>touch empty_d_with_p.txt c:\Users\natomar\Downloads\testacl>cacls empty_d_with_p.txt /S:D:P Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\empty_d_with_p.txt * Don't fail outright for ACL revision 4. Though our supported ACL types must carry ACL revision 2 as per the doc https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-dtyp/20233ed8-a6c6-4097-aafa-dd545ed24428 but I've seen some dirs have ACL revision 4 but ACL types are still supported ones. So instead of failing upfront, let it fail with unsupported ACE type. Also hexadecimal aceRights are more commonly seen than I expected, so removing a log. * Minor fix after running azcopy on a large dir. This was something which I have doubt on. Now that we got a real world issue due to this, it's all clear :-) * Some minor updates after the rebase to latest Azcopy. * Set default value of flag preserve-smb-info to true on Windows and false on other OS (cherry picked from commit ac5bedb) Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Nagendra Tomar <[email protected]> * Added log indicating a sub-directory is being enqueued (#1999) * Log sync deletions to scanning logger (#2000) * ieproxy fix * remove cgo * fix * fix * fix * more testing * more testing * more testing * more testing * mod tidy * mod tidy * more testing * Added codespell (#2008) * Added codespell * Fixed initial codespell errors * Fix format in codespell.yml * Added s3 url parts * Added CodeQL (#2009) * Added linting file * Upgrade codeql to v2 * Fix incorrect conversion between integer types * Fix GCP URL parts * Fix for rare infinite loop on mutex acquisition (#2012) * small fix * removed test * Added trivy file (#2015) * Added trivy file * renamed trivy * Improve debug-ability of e2e tests by uploading logs of failed jobs (#1898) * Upload testing logs to storage account on failed test * Handle as pipeline artifact instead * mkdirall * copy plan files too * Fix failing tests * Change overwrite to affect any "locked in"/completed state * Fail copy job if single blob does not exist (#1981) * Job fail if single file does not exist * fixed change * fail only on a single file not existing * fail on file not found * fail on file not found * fail on file not found * cleanup * added tests * cleanup * removed test * Correct odd behavior around folder overwrites (#1961) * Fix files sync by determining which LMT to use via smb properties flag (#1958) * Fix files sync by determining which LMT to use via smb properties flag * Implement testing for LMT switch * Fix testing * Limit SMB testing to SMB-compatible environment * Enforce SMB LMT for Linux/MacOS test of SMB LMT preference * Fix metadata parsing (#1953) * Fix metadata parsing * rework metadata parsing to be more robust; add test * Fix comment lines * Codespell :| * Fix ADLSG2 intermittent failure (#1901) * Fix ADLSG2 intermittent failure * Add test * Reduce code dupe * Fix build errors * Fix infinite loop maybe? * Store source token and pass to other threads (#1996) * Store source token * testing * failing pipe * cleanup * test logger * fix test failure * fix 2 * fix * sync fix * cleanup check * Hash based sync (#2020) * Implement hash based sync for MD5 * Implement testing * Ensure folders are handled properly in HBS & Test S2S * Add skip/process logging * Include generic xattr syncmeta application * Fix 0-size blobs * Fix core testing * Revert "Include generic xattr syncmeta application" This reverts commit fba55e4. * Warn on no hash @ source, remove MHP * Comments * Comments * Copy properties from Source (#1964) * Copy properties from Source * Remove unnecessary ws changes * Preserve UNIX properties * Move entity type to Overwrite option * Add python suite * Review comments * Fix test * Release notes and version update (#2028) Co-authored-by: adreed-msft <[email protected]> Co-authored-by: mstenz <[email protected]> Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> Co-authored-by: Adele Reed <[email protected]> Co-authored-by: Karla Saur <[email protected]> Co-authored-by: adam-orosz <[email protected]> Co-authored-by: Adam Orosz <[email protected]> Co-authored-by: Ze Qian Zhang <[email protected]> Co-authored-by: Gauri Prasad <[email protected]> Co-authored-by: Gauri Prasad <[email protected]> Co-authored-by: Tamer Sherif <[email protected]> Co-authored-by: Tamer Sherif <[email protected]> Co-authored-by: reshmav18 <[email protected]> Co-authored-by: linuxsmiths <[email protected]> Co-authored-by: Nagendra Tomar <[email protected]>
nakulkar-msft
added a commit
that referenced
this pull request
Mar 28, 2023
* Release 10.17.0 (#2029) * Add mitigation for weird NtQuerySecurityObject behavior on NAS sources (#1872) * Add check for 0 length, attempt to validate the returned object. * Change to grabbing real SD length * Add comment describing issue * Prevent infinite loop upon listing failure * Fix GCP error checking * Fix GCP disable * Fix bad URL delete (#1892) * Manipulate URLs safely * Fix folder deletion test * Prevent infinite loop upon listing failure * Fix GCP error checking * Fix GCP disable * Fail when errors listing/clearing bucket * Update MacOS testing pipeline (#1896) * fixing small typo (,) in help of jobs clean (#1899) * Microsoft mandatory file * fixing small typo (,) in help of jobs clean Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> * Implement MD OAuth testing (#1859) * Implement MD OAuth testing * Handle async on RevokeAccess, handle job cancel/failure better * Prevent parallel testing of managed disks * lint check * Prevent infinite loop upon listing failure * Fix GCP error checking * Fix GCP disable * Fail when errors listing/clearing bucket * Add env vars * Avoid revoking MD access, as it can be shared. * Fix intermittent failures * Disable MD OAuth testing temporarily. * Add "all" to documentation (#1902) * 10.16.1 patch notes (#1913) * Add bugfixes to change log. * Correct wording & punctuation * Correct version * Export Successfully Updated bytes (#1884) * Add info in error message for mkdir on Log/Plan (#1883) * Microsoft mandatory file * Add info in error message for mkdir on Log/Plan Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> * Fix fixupTokenJson (#1890) * Microsoft mandatory file * Fix fixupTokenJson Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> Co-authored-by: Adam Orosz <[email protected]> * Do not log request/response for container creation error (#1893) * Expose AZCOPY_DOWNLOAD_TO_TEMP_PATH environment variable. (#1895) * Slice against the correct string (#1927) * UX improvement: avoid crash when copying S2S with user delegation SAS (#1932) * Fix bad build + Prevent bad builds in the future (#1917) * Fix bad build + Prevent bad builds in the future * Add Windows build * Make sync use last write time for Azure Files (#1930) * Make sync use last write time for Azure Files * Implement test * 10.16.2 Changelog (#1948) * Update azcopy version * Fixed a bug where preserve permissions would not work with OAuth * Added CODEOWNERS file * Fixed issue where CPK would not be injected on retries * remove OAuth from test * Updated version check string to indicate current AzCopy version (#1969) * added codeowner * Enhance job summary with details about file/folders (#1952) * Add flag to disable version check (#1950) * darwin arm64 * Update golang version to 10.19.2 (#1925) * enable cgo * added tests * Minor fixes: More in description (#1968) * Echo auto-login failure if any * Update help for sync command to use trailing slash on directories * azcopy fail to copy 12TB file to Storage containers in Dev. The logic is used to calculate proper blockSize if it’s not provided, and due to the uint32 cast, it can’t give proper blockSize if filesize is between 50000 * (8 * 1024 * 1024) * X + 1, to 50000 * (8 * 1024 * 1024) * X + 49999. It should return 16MB instead of 8MB blockSize. Accommodated the changes suggested by Narasimha Kulkarni * Added extra logging when switching endpoints * Enable support for preserving SMB info on Linux. (#1723) * Microsoft mandatory file * Enable support for preserving SMB info on Linux. Implemented the GetSDDL/PutSDDL GetSMBProperties/PutSMBProperties methods for Linux using extended attributes. Following are the xattrs we use for fetching/setting various required info. // Extended Attribute (xattr) keys for fetching various information from Linux cifs client. const ( CIFS_XATTR_CREATETIME = "user.cifs.creationtime" // File creation time. CIFS_XATTR_ATTRIB = "user.cifs.dosattrib" // FileAttributes. CIFS_XATTR_CIFS_ACL = "system.cifs_acl" // DACL only. CIFS_XATTR_CIFS_NTSD = "system.cifs_ntsd" // Owner, Group, DACL. CIFS_XATTR_CIFS_NTSD_FULL = "system.cifs_ntsd_full" // Owner, Group, DACL, SACL. ) Majority of the changes are in sddl/sddlHelper_linux.go which implement the following Win32 APIs for dealing with SIDs. ConvertSecurityDescriptorToStringSecurityDescriptorW ConvertStringSecurityDescriptorToSecurityDescriptorW ConvertSidToStringSidW ConvertStringSidToSidW Note: I have skipped Object ACE support in sddl/sddlHelper_linux.go as those should not be used for filesystem properties, only AD object properties. Can someone confirm this? TBD: Conditional SID * Audited, fixed, tested support for "No ACL"/NO_ACCESS_CONTROL and ACL w/o any ACE Tested the following cases: c:\Users\natomar\Downloads>cd testacl // This has "No ACLs" and everyone should be allowed access. c:\Users\natomar\Downloads\testacl>touch NO_ACCESS_CONTROL.txt c:\Users\natomar\Downloads\testacl>cacls NO_ACCESS_CONTROL.txt /S:D:NO_ACCESS_CONTROL Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\NO_ACCESS_CONTROL.txt // This has "No ACLs" and everyone should be allowed access. // It additionally has the "P" (protected) flag set, but that won't have // any effect as that just prevents ACE inheritance but this ACL will // not have any ACLs due to the NO_ACCESS_CONTROL flag. c:\Users\natomar\Downloads\testacl>touch PNO_ACCESS_CONTROL.txt c:\Users\natomar\Downloads\testacl>cacls PNO_ACCESS_CONTROL.txt /S:D:PNO_ACCESS_CONTROL Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\PNO_ACCESS_CONTROL.txt // This should set DACL but with no ACEs, but since "P" is not set it // inherits ACEs from the parent dir. c:\Users\natomar\Downloads\testacl>touch empty_d.txt c:\Users\natomar\Downloads\testacl>cacls empty_d.txt /S:D: Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\empty_d.txt // This should set DACL but with no ACEs, but since "P" is set it // doesn't inherit ACEs from the parent dir and hence this will block // all users. c:\Users\natomar\Downloads\testacl>touch empty_d_with_p.txt c:\Users\natomar\Downloads\testacl>cacls empty_d_with_p.txt /S:D:P Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\empty_d_with_p.txt * Don't fail outright for ACL revision 4. Though our supported ACL types must carry ACL revision 2 as per the doc https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-dtyp/20233ed8-a6c6-4097-aafa-dd545ed24428 but I've seen some dirs have ACL revision 4 but ACL types are still supported ones. So instead of failing upfront, let it fail with unsupported ACE type. Also hexadecimal aceRights are more commonly seen than I expected, so removing a log. * Minor fix after running azcopy on a large dir. This was something which I have doubt on. Now that we got a real world issue due to this, it's all clear :-) * Some minor updates after the rebase to latest Azcopy. * Set default value of flag preserve-smb-info to true on Windows and false on other OS (cherry picked from commit ac5bedb) Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Nagendra Tomar <[email protected]> * Added log indicating a sub-directory is being enqueued (#1999) * Log sync deletions to scanning logger (#2000) * ieproxy fix * remove cgo * fix * fix * fix * more testing * more testing * more testing * more testing * mod tidy * mod tidy * more testing * Added codespell (#2008) * Added codespell * Fixed initial codespell errors * Fix format in codespell.yml * Added s3 url parts * Added CodeQL (#2009) * Added linting file * Upgrade codeql to v2 * Fix incorrect conversion between integer types * Fix GCP URL parts * Fix for rare infinite loop on mutex acquisition (#2012) * small fix * removed test * Added trivy file (#2015) * Added trivy file * renamed trivy * Improve debug-ability of e2e tests by uploading logs of failed jobs (#1898) * Upload testing logs to storage account on failed test * Handle as pipeline artifact instead * mkdirall * copy plan files too * Fix failing tests * Change overwrite to affect any "locked in"/completed state * Fail copy job if single blob does not exist (#1981) * Job fail if single file does not exist * fixed change * fail only on a single file not existing * fail on file not found * fail on file not found * fail on file not found * cleanup * added tests * cleanup * removed test * Correct odd behavior around folder overwrites (#1961) * Fix files sync by determining which LMT to use via smb properties flag (#1958) * Fix files sync by determining which LMT to use via smb properties flag * Implement testing for LMT switch * Fix testing * Limit SMB testing to SMB-compatible environment * Enforce SMB LMT for Linux/MacOS test of SMB LMT preference * Fix metadata parsing (#1953) * Fix metadata parsing * rework metadata parsing to be more robust; add test * Fix comment lines * Codespell :| * Fix ADLSG2 intermittent failure (#1901) * Fix ADLSG2 intermittent failure * Add test * Reduce code dupe * Fix build errors * Fix infinite loop maybe? * Store source token and pass to other threads (#1996) * Store source token * testing * failing pipe * cleanup * test logger * fix test failure * fix 2 * fix * sync fix * cleanup check * Hash based sync (#2020) * Implement hash based sync for MD5 * Implement testing * Ensure folders are handled properly in HBS & Test S2S * Add skip/process logging * Include generic xattr syncmeta application * Fix 0-size blobs * Fix core testing * Revert "Include generic xattr syncmeta application" This reverts commit fba55e4. * Warn on no hash @ source, remove MHP * Comments * Comments * Copy properties from Source (#1964) * Copy properties from Source * Remove unnecessary ws changes * Preserve UNIX properties * Move entity type to Overwrite option * Add python suite * Review comments * Fix test * Release notes and version update (#2028) Co-authored-by: adreed-msft <[email protected]> Co-authored-by: mstenz <[email protected]> Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> Co-authored-by: Adele Reed <[email protected]> Co-authored-by: Karla Saur <[email protected]> Co-authored-by: adam-orosz <[email protected]> Co-authored-by: Adam Orosz <[email protected]> Co-authored-by: Ze Qian Zhang <[email protected]> Co-authored-by: Gauri Prasad <[email protected]> Co-authored-by: Gauri Prasad <[email protected]> Co-authored-by: Tamer Sherif <[email protected]> Co-authored-by: Tamer Sherif <[email protected]> Co-authored-by: reshmav18 <[email protected]> Co-authored-by: linuxsmiths <[email protected]> Co-authored-by: Nagendra Tomar <[email protected]> * Add in content md5 * Change variable name * Change to base64 encoding * Update list.go * Fix the object for md5. * Fix name to mathc help and formating. * Switch to decode to string * Fix syntax error. * Left of b * Undoing conversion * Fix syntax error. * Fixed camelcase. * Fix syntax error. * Fix syntax error. --------- Co-authored-by: Narasimha Kulkarni <[email protected]> Co-authored-by: adreed-msft <[email protected]> Co-authored-by: mstenz <[email protected]> Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> Co-authored-by: Adele Reed <[email protected]> Co-authored-by: Karla Saur <[email protected]> Co-authored-by: adam-orosz <[email protected]> Co-authored-by: Adam Orosz <[email protected]> Co-authored-by: Ze Qian Zhang <[email protected]> Co-authored-by: Gauri Prasad <[email protected]> Co-authored-by: Gauri Prasad <[email protected]> Co-authored-by: Tamer Sherif <[email protected]> Co-authored-by: Tamer Sherif <[email protected]> Co-authored-by: reshmav18 <[email protected]> Co-authored-by: linuxsmiths <[email protected]> Co-authored-by: Nagendra Tomar <[email protected]>
nakulkar-msft
added a commit
that referenced
this pull request
Mar 30, 2023
* Release 10.17.0 (#2029) * Add mitigation for weird NtQuerySecurityObject behavior on NAS sources (#1872) * Add check for 0 length, attempt to validate the returned object. * Change to grabbing real SD length * Add comment describing issue * Prevent infinite loop upon listing failure * Fix GCP error checking * Fix GCP disable * Fix bad URL delete (#1892) * Manipulate URLs safely * Fix folder deletion test * Prevent infinite loop upon listing failure * Fix GCP error checking * Fix GCP disable * Fail when errors listing/clearing bucket * Update MacOS testing pipeline (#1896) * fixing small typo (,) in help of jobs clean (#1899) * Microsoft mandatory file * fixing small typo (,) in help of jobs clean Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> * Implement MD OAuth testing (#1859) * Implement MD OAuth testing * Handle async on RevokeAccess, handle job cancel/failure better * Prevent parallel testing of managed disks * lint check * Prevent infinite loop upon listing failure * Fix GCP error checking * Fix GCP disable * Fail when errors listing/clearing bucket * Add env vars * Avoid revoking MD access, as it can be shared. * Fix intermittent failures * Disable MD OAuth testing temporarily. * Add "all" to documentation (#1902) * 10.16.1 patch notes (#1913) * Add bugfixes to change log. * Correct wording & punctuation * Correct version * Export Successfully Updated bytes (#1884) * Add info in error message for mkdir on Log/Plan (#1883) * Microsoft mandatory file * Add info in error message for mkdir on Log/Plan Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> * Fix fixupTokenJson (#1890) * Microsoft mandatory file * Fix fixupTokenJson Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> Co-authored-by: Adam Orosz <[email protected]> * Do not log request/response for container creation error (#1893) * Expose AZCOPY_DOWNLOAD_TO_TEMP_PATH environment variable. (#1895) * Slice against the correct string (#1927) * UX improvement: avoid crash when copying S2S with user delegation SAS (#1932) * Fix bad build + Prevent bad builds in the future (#1917) * Fix bad build + Prevent bad builds in the future * Add Windows build * Make sync use last write time for Azure Files (#1930) * Make sync use last write time for Azure Files * Implement test * 10.16.2 Changelog (#1948) * Update azcopy version * Fixed a bug where preserve permissions would not work with OAuth * Added CODEOWNERS file * Fixed issue where CPK would not be injected on retries * remove OAuth from test * Updated version check string to indicate current AzCopy version (#1969) * added codeowner * Enhance job summary with details about file/folders (#1952) * Add flag to disable version check (#1950) * darwin arm64 * Update golang version to 10.19.2 (#1925) * enable cgo * added tests * Minor fixes: More in description (#1968) * Echo auto-login failure if any * Update help for sync command to use trailing slash on directories * azcopy fail to copy 12TB file to Storage containers in Dev. The logic is used to calculate proper blockSize if it’s not provided, and due to the uint32 cast, it can’t give proper blockSize if filesize is between 50000 * (8 * 1024 * 1024) * X + 1, to 50000 * (8 * 1024 * 1024) * X + 49999. It should return 16MB instead of 8MB blockSize. Accommodated the changes suggested by Narasimha Kulkarni * Added extra logging when switching endpoints * Enable support for preserving SMB info on Linux. (#1723) * Microsoft mandatory file * Enable support for preserving SMB info on Linux. Implemented the GetSDDL/PutSDDL GetSMBProperties/PutSMBProperties methods for Linux using extended attributes. Following are the xattrs we use for fetching/setting various required info. // Extended Attribute (xattr) keys for fetching various information from Linux cifs client. const ( CIFS_XATTR_CREATETIME = "user.cifs.creationtime" // File creation time. CIFS_XATTR_ATTRIB = "user.cifs.dosattrib" // FileAttributes. CIFS_XATTR_CIFS_ACL = "system.cifs_acl" // DACL only. CIFS_XATTR_CIFS_NTSD = "system.cifs_ntsd" // Owner, Group, DACL. CIFS_XATTR_CIFS_NTSD_FULL = "system.cifs_ntsd_full" // Owner, Group, DACL, SACL. ) Majority of the changes are in sddl/sddlHelper_linux.go which implement the following Win32 APIs for dealing with SIDs. ConvertSecurityDescriptorToStringSecurityDescriptorW ConvertStringSecurityDescriptorToSecurityDescriptorW ConvertSidToStringSidW ConvertStringSidToSidW Note: I have skipped Object ACE support in sddl/sddlHelper_linux.go as those should not be used for filesystem properties, only AD object properties. Can someone confirm this? TBD: Conditional SID * Audited, fixed, tested support for "No ACL"/NO_ACCESS_CONTROL and ACL w/o any ACE Tested the following cases: c:\Users\natomar\Downloads>cd testacl // This has "No ACLs" and everyone should be allowed access. c:\Users\natomar\Downloads\testacl>touch NO_ACCESS_CONTROL.txt c:\Users\natomar\Downloads\testacl>cacls NO_ACCESS_CONTROL.txt /S:D:NO_ACCESS_CONTROL Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\NO_ACCESS_CONTROL.txt // This has "No ACLs" and everyone should be allowed access. // It additionally has the "P" (protected) flag set, but that won't have // any effect as that just prevents ACE inheritance but this ACL will // not have any ACLs due to the NO_ACCESS_CONTROL flag. c:\Users\natomar\Downloads\testacl>touch PNO_ACCESS_CONTROL.txt c:\Users\natomar\Downloads\testacl>cacls PNO_ACCESS_CONTROL.txt /S:D:PNO_ACCESS_CONTROL Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\PNO_ACCESS_CONTROL.txt // This should set DACL but with no ACEs, but since "P" is not set it // inherits ACEs from the parent dir. c:\Users\natomar\Downloads\testacl>touch empty_d.txt c:\Users\natomar\Downloads\testacl>cacls empty_d.txt /S:D: Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\empty_d.txt // This should set DACL but with no ACEs, but since "P" is set it // doesn't inherit ACEs from the parent dir and hence this will block // all users. c:\Users\natomar\Downloads\testacl>touch empty_d_with_p.txt c:\Users\natomar\Downloads\testacl>cacls empty_d_with_p.txt /S:D:P Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\empty_d_with_p.txt * Don't fail outright for ACL revision 4. Though our supported ACL types must carry ACL revision 2 as per the doc https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-dtyp/20233ed8-a6c6-4097-aafa-dd545ed24428 but I've seen some dirs have ACL revision 4 but ACL types are still supported ones. So instead of failing upfront, let it fail with unsupported ACE type. Also hexadecimal aceRights are more commonly seen than I expected, so removing a log. * Minor fix after running azcopy on a large dir. This was something which I have doubt on. Now that we got a real world issue due to this, it's all clear :-) * Some minor updates after the rebase to latest Azcopy. * Set default value of flag preserve-smb-info to true on Windows and false on other OS (cherry picked from commit ac5bedb) Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Nagendra Tomar <[email protected]> * Added log indicating a sub-directory is being enqueued (#1999) * Log sync deletions to scanning logger (#2000) * ieproxy fix * remove cgo * fix * fix * fix * more testing * more testing * more testing * more testing * mod tidy * mod tidy * more testing * Added codespell (#2008) * Added codespell * Fixed initial codespell errors * Fix format in codespell.yml * Added s3 url parts * Added CodeQL (#2009) * Added linting file * Upgrade codeql to v2 * Fix incorrect conversion between integer types * Fix GCP URL parts * Fix for rare infinite loop on mutex acquisition (#2012) * small fix * removed test * Added trivy file (#2015) * Added trivy file * renamed trivy * Improve debug-ability of e2e tests by uploading logs of failed jobs (#1898) * Upload testing logs to storage account on failed test * Handle as pipeline artifact instead * mkdirall * copy plan files too * Fix failing tests * Change overwrite to affect any "locked in"/completed state * Fail copy job if single blob does not exist (#1981) * Job fail if single file does not exist * fixed change * fail only on a single file not existing * fail on file not found * fail on file not found * fail on file not found * cleanup * added tests * cleanup * removed test * Correct odd behavior around folder overwrites (#1961) * Fix files sync by determining which LMT to use via smb properties flag (#1958) * Fix files sync by determining which LMT to use via smb properties flag * Implement testing for LMT switch * Fix testing * Limit SMB testing to SMB-compatible environment * Enforce SMB LMT for Linux/MacOS test of SMB LMT preference * Fix metadata parsing (#1953) * Fix metadata parsing * rework metadata parsing to be more robust; add test * Fix comment lines * Codespell :| * Fix ADLSG2 intermittent failure (#1901) * Fix ADLSG2 intermittent failure * Add test * Reduce code dupe * Fix build errors * Fix infinite loop maybe? * Store source token and pass to other threads (#1996) * Store source token * testing * failing pipe * cleanup * test logger * fix test failure * fix 2 * fix * sync fix * cleanup check * Hash based sync (#2020) * Implement hash based sync for MD5 * Implement testing * Ensure folders are handled properly in HBS & Test S2S * Add skip/process logging * Include generic xattr syncmeta application * Fix 0-size blobs * Fix core testing * Revert "Include generic xattr syncmeta application" This reverts commit fba55e4. * Warn on no hash @ source, remove MHP * Comments * Comments * Copy properties from Source (#1964) * Copy properties from Source * Remove unnecessary ws changes * Preserve UNIX properties * Move entity type to Overwrite option * Add python suite * Review comments * Fix test * Release notes and version update (#2028) Co-authored-by: adreed-msft <[email protected]> Co-authored-by: mstenz <[email protected]> Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> Co-authored-by: Adele Reed <[email protected]> Co-authored-by: Karla Saur <[email protected]> Co-authored-by: adam-orosz <[email protected]> Co-authored-by: Adam Orosz <[email protected]> Co-authored-by: Ze Qian Zhang <[email protected]> Co-authored-by: Gauri Prasad <[email protected]> Co-authored-by: Gauri Prasad <[email protected]> Co-authored-by: Tamer Sherif <[email protected]> Co-authored-by: Tamer Sherif <[email protected]> Co-authored-by: reshmav18 <[email protected]> Co-authored-by: linuxsmiths <[email protected]> Co-authored-by: Nagendra Tomar <[email protected]> * Add in content md5 * Change variable name * Change to base64 encoding * Update list.go * Fix the object for md5. * Fix name to mathc help and formating. * Switch to decode to string * Fix syntax error. * Left of b * Undoing conversion * Fix syntax error. * Fixed camelcase. * Fix syntax error. * Fix syntax error. --------- Co-authored-by: Narasimha Kulkarni <[email protected]> Co-authored-by: adreed-msft <[email protected]> Co-authored-by: mstenz <[email protected]> Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> Co-authored-by: Adele Reed <[email protected]> Co-authored-by: Karla Saur <[email protected]> Co-authored-by: adam-orosz <[email protected]> Co-authored-by: Adam Orosz <[email protected]> Co-authored-by: Ze Qian Zhang <[email protected]> Co-authored-by: Gauri Prasad <[email protected]> Co-authored-by: Gauri Prasad <[email protected]> Co-authored-by: Tamer Sherif <[email protected]> Co-authored-by: Tamer Sherif <[email protected]> Co-authored-by: reshmav18 <[email protected]> Co-authored-by: linuxsmiths <[email protected]> Co-authored-by: Nagendra Tomar <[email protected]>
adreed-msft
added a commit
that referenced
this pull request
Mar 31, 2023
* Add mitigation for weird NtQuerySecurityObject behavior on NAS sources (#1872) * Add check for 0 length, attempt to validate the returned object. * Change to grabbing real SD length * Add comment describing issue * Prevent infinite loop upon listing failure * Fix GCP error checking * Fix GCP disable * Fix bad URL delete (#1892) * Manipulate URLs safely * Fix folder deletion test * Prevent infinite loop upon listing failure * Fix GCP error checking * Fix GCP disable * Fail when errors listing/clearing bucket * Update MacOS testing pipeline (#1896) * fixing small typo (,) in help of jobs clean (#1899) * Microsoft mandatory file * fixing small typo (,) in help of jobs clean Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> * Implement MD OAuth testing (#1859) * Implement MD OAuth testing * Handle async on RevokeAccess, handle job cancel/failure better * Prevent parallel testing of managed disks * lint check * Prevent infinite loop upon listing failure * Fix GCP error checking * Fix GCP disable * Fail when errors listing/clearing bucket * Add env vars * Avoid revoking MD access, as it can be shared. * Fix intermittent failures * Disable MD OAuth testing temporarily. * Add "all" to documentation (#1902) * 10.16.1 patch notes (#1913) * Add bugfixes to change log. * Correct wording & punctuation * Correct version * Export Successfully Updated bytes (#1884) * Add info in error message for mkdir on Log/Plan (#1883) * Microsoft mandatory file * Add info in error message for mkdir on Log/Plan Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> * Fix fixupTokenJson (#1890) * Microsoft mandatory file * Fix fixupTokenJson Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> Co-authored-by: Adam Orosz <[email protected]> * Do not log request/response for container creation error (#1893) * Expose AZCOPY_DOWNLOAD_TO_TEMP_PATH environment variable. (#1895) * Slice against the correct string (#1927) * UX improvement: avoid crash when copying S2S with user delegation SAS (#1932) * Fix bad build + Prevent bad builds in the future (#1917) * Fix bad build + Prevent bad builds in the future * Add Windows build * Make sync use last write time for Azure Files (#1930) * Make sync use last write time for Azure Files * Implement test * 10.16.2 Changelog (#1948) * Update azcopy version * Fixed a bug where preserve permissions would not work with OAuth * Added CODEOWNERS file * Fixed issue where CPK would not be injected on retries * remove OAuth from test * Updated version check string to indicate current AzCopy version (#1969) * added codeowner * Enhance job summary with details about file/folders (#1952) * Add flag to disable version check (#1950) * darwin arm64 * Update golang version to 10.19.2 (#1925) * enable cgo * added tests * Minor fixes: More in description (#1968) * Echo auto-login failure if any * Update help for sync command to use trailing slash on directories * azcopy fail to copy 12TB file to Storage containers in Dev. The logic is used to calculate proper blockSize if it’s not provided, and due to the uint32 cast, it can’t give proper blockSize if filesize is between 50000 * (8 * 1024 * 1024) * X + 1, to 50000 * (8 * 1024 * 1024) * X + 49999. It should return 16MB instead of 8MB blockSize. Accommodated the changes suggested by Narasimha Kulkarni * Added extra logging when switching endpoints * Enable support for preserving SMB info on Linux. (#1723) * Microsoft mandatory file * Enable support for preserving SMB info on Linux. Implemented the GetSDDL/PutSDDL GetSMBProperties/PutSMBProperties methods for Linux using extended attributes. Following are the xattrs we use for fetching/setting various required info. // Extended Attribute (xattr) keys for fetching various information from Linux cifs client. const ( CIFS_XATTR_CREATETIME = "user.cifs.creationtime" // File creation time. CIFS_XATTR_ATTRIB = "user.cifs.dosattrib" // FileAttributes. CIFS_XATTR_CIFS_ACL = "system.cifs_acl" // DACL only. CIFS_XATTR_CIFS_NTSD = "system.cifs_ntsd" // Owner, Group, DACL. CIFS_XATTR_CIFS_NTSD_FULL = "system.cifs_ntsd_full" // Owner, Group, DACL, SACL. ) Majority of the changes are in sddl/sddlHelper_linux.go which implement the following Win32 APIs for dealing with SIDs. ConvertSecurityDescriptorToStringSecurityDescriptorW ConvertStringSecurityDescriptorToSecurityDescriptorW ConvertSidToStringSidW ConvertStringSidToSidW Note: I have skipped Object ACE support in sddl/sddlHelper_linux.go as those should not be used for filesystem properties, only AD object properties. Can someone confirm this? TBD: Conditional SID * Audited, fixed, tested support for "No ACL"/NO_ACCESS_CONTROL and ACL w/o any ACE Tested the following cases: c:\Users\natomar\Downloads>cd testacl // This has "No ACLs" and everyone should be allowed access. c:\Users\natomar\Downloads\testacl>touch NO_ACCESS_CONTROL.txt c:\Users\natomar\Downloads\testacl>cacls NO_ACCESS_CONTROL.txt /S:D:NO_ACCESS_CONTROL Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\NO_ACCESS_CONTROL.txt // This has "No ACLs" and everyone should be allowed access. // It additionally has the "P" (protected) flag set, but that won't have // any effect as that just prevents ACE inheritance but this ACL will // not have any ACLs due to the NO_ACCESS_CONTROL flag. c:\Users\natomar\Downloads\testacl>touch PNO_ACCESS_CONTROL.txt c:\Users\natomar\Downloads\testacl>cacls PNO_ACCESS_CONTROL.txt /S:D:PNO_ACCESS_CONTROL Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\PNO_ACCESS_CONTROL.txt // This should set DACL but with no ACEs, but since "P" is not set it // inherits ACEs from the parent dir. c:\Users\natomar\Downloads\testacl>touch empty_d.txt c:\Users\natomar\Downloads\testacl>cacls empty_d.txt /S:D: Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\empty_d.txt // This should set DACL but with no ACEs, but since "P" is set it // doesn't inherit ACEs from the parent dir and hence this will block // all users. c:\Users\natomar\Downloads\testacl>touch empty_d_with_p.txt c:\Users\natomar\Downloads\testacl>cacls empty_d_with_p.txt /S:D:P Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\empty_d_with_p.txt * Don't fail outright for ACL revision 4. Though our supported ACL types must carry ACL revision 2 as per the doc https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-dtyp/20233ed8-a6c6-4097-aafa-dd545ed24428 but I've seen some dirs have ACL revision 4 but ACL types are still supported ones. So instead of failing upfront, let it fail with unsupported ACE type. Also hexadecimal aceRights are more commonly seen than I expected, so removing a log. * Minor fix after running azcopy on a large dir. This was something which I have doubt on. Now that we got a real world issue due to this, it's all clear :-) * Some minor updates after the rebase to latest Azcopy. * Set default value of flag preserve-smb-info to true on Windows and false on other OS (cherry picked from commit ac5bedb) Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Nagendra Tomar <[email protected]> * Added log indicating a sub-directory is being enqueued (#1999) * Log sync deletions to scanning logger (#2000) * ieproxy fix * remove cgo * fix * fix * fix * more testing * more testing * more testing * more testing * mod tidy * mod tidy * more testing * Added codespell (#2008) * Added codespell * Fixed initial codespell errors * Fix format in codespell.yml * Added s3 url parts * Added CodeQL (#2009) * Added linting file * Upgrade codeql to v2 * Fix incorrect conversion between integer types * Fix GCP URL parts * Fix for rare infinite loop on mutex acquisition (#2012) * small fix * removed test * Added trivy file (#2015) * Added trivy file * renamed trivy * Improve debug-ability of e2e tests by uploading logs of failed jobs (#1898) * Upload testing logs to storage account on failed test * Handle as pipeline artifact instead * mkdirall * copy plan files too * Fix failing tests * Change overwrite to affect any "locked in"/completed state * Fail copy job if single blob does not exist (#1981) * Job fail if single file does not exist * fixed change * fail only on a single file not existing * fail on file not found * fail on file not found * fail on file not found * cleanup * added tests * cleanup * removed test * Correct odd behavior around folder overwrites (#1961) * Fix files sync by determining which LMT to use via smb properties flag (#1958) * Fix files sync by determining which LMT to use via smb properties flag * Implement testing for LMT switch * Fix testing * Limit SMB testing to SMB-compatible environment * Enforce SMB LMT for Linux/MacOS test of SMB LMT preference * Fix metadata parsing (#1953) * Fix metadata parsing * rework metadata parsing to be more robust; add test * Fix comment lines * Codespell :| * Fix ADLSG2 intermittent failure (#1901) * Fix ADLSG2 intermittent failure * Add test * Reduce code dupe * Fix build errors * Fix infinite loop maybe? * Store source token and pass to other threads (#1996) * Store source token * testing * failing pipe * cleanup * test logger * fix test failure * fix 2 * fix * sync fix * cleanup check * Hash based sync (#2020) * Implement hash based sync for MD5 * Implement testing * Ensure folders are handled properly in HBS & Test S2S * Add skip/process logging * Include generic xattr syncmeta application * Fix 0-size blobs * Fix core testing * Revert "Include generic xattr syncmeta application" This reverts commit fba55e4. * Warn on no hash @ source, remove MHP * Comments * Comments * Copy properties from Source (#1964) * Copy properties from Source * Remove unnecessary ws changes * Preserve UNIX properties * Move entity type to Overwrite option * Add python suite * Review comments * Fix test * Release notes and version update (#2028) * Fix SIGSERV on MacOS causes by mattn/go-ieproxy#35. Fixes #1790. (#1993) * Revert "Fix SIGSERV on MacOS causes by mattn/go-ieproxy#35. Fixes #1790. (#1993)" (#2044) This reverts commit dc7666a. * Fixed HTML Blob Type (#2048) * fixed typos * typo * Implement upload/download of symlinks (#1829) * Implement symlink upload/download * Fix tests * Handle comments * Testing * Fix test arguments * Fix files MD5 issues * Limit symlink persistence to relevant locations * Update block ID generation logic (#2050) * Change BlockID generation Logic * Add testcase * Add comment to explain naming of block IDS * Reformt blockID name convention * Upgrade GCS storage library (#2064) * Update GCS module * Increase lint timeout * Set GC to default (#2063) * Fix incorrectly displayed environment variable name for plan file path * Updating helpMessages.go to further clarify the --compare-hash flag * Add golangci-lint (#2010) * Upgrade linting version (#2105) * Implement container-level ACL copies (#2049) * Implement container-level ACL copies * Prevent AssertNoErr from crashing test suite * Fix delete functionality & testing * Grab subdir in a different way * Fix Build * Lint * Reimplement POSIX properties download (#1835) * Reimplement POSIX properties download * Fix upload/download of special file types (pipe, fifo, dev, cdev) * Posix Download testing + Fix for symlink download * Prevent folders from generating while using CPK * Remove removed bit * Fix tests, add ADLSG2 tests * Limit scope of POSIX test metadata * Golint * Fix wildcard handling when a file of the same name as the wildcard input exists (#2062) * Fix stgexp bug * Fix test compilation * Upgrade azcopy dependencies (#2115) * Option to transfer properties-only in jobPart (#2071) * Added a scenario to the help messages (#2121) * Allow specifying "Cold" tier (#2096) * Allow specifying "Cold" tier * Switch service version when setting tier * Do not create parent directory if it is root (#2120) * Do not create parent directory if it is root * Add testcase * Add testcase2 * Resume transfer of incomplete file (#2119) * Resume file * iIntroduce transfer status restarted * Spelling mistake * Add resume to uploads * Add environment variable and make log msg more clear * Change log level to Debug for a few statements (#2123) * Change log level to Debug for a few statements * Fix repeated messages on empty input * Perf testing for AzCopy (#2006) * Base commit for perf test * Add tests * Add local tests * Re-enable S2S and use /dev/null for local tests * Add auth mode to Clean destination Containers * Modify bench command to have runtime around 30min * Change image name * Add auth mode to cli command * Change large files count to 50 * Redact CPK headers in log output. (#2127) Currently, azcopy will emit the following headers to log files, when using Customer Provided Keys: x-ms-encryption-key x-ms-encryption-key-sha256 I believe this is unexpected for some users. This patch redacts those headers from log files, such that encryption keys do not accidentally leak. Co-authored-by: Mikkel Krautz <[email protected]> * [AzCopyV10] Add in content-md5 to available list properties (#2033) * Release 10.17.0 (#2029) * Add mitigation for weird NtQuerySecurityObject behavior on NAS sources (#1872) * Add check for 0 length, attempt to validate the returned object. * Change to grabbing real SD length * Add comment describing issue * Prevent infinite loop upon listing failure * Fix GCP error checking * Fix GCP disable * Fix bad URL delete (#1892) * Manipulate URLs safely * Fix folder deletion test * Prevent infinite loop upon listing failure * Fix GCP error checking * Fix GCP disable * Fail when errors listing/clearing bucket * Update MacOS testing pipeline (#1896) * fixing small typo (,) in help of jobs clean (#1899) * Microsoft mandatory file * fixing small typo (,) in help of jobs clean Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> * Implement MD OAuth testing (#1859) * Implement MD OAuth testing * Handle async on RevokeAccess, handle job cancel/failure better * Prevent parallel testing of managed disks * lint check * Prevent infinite loop upon listing failure * Fix GCP error checking * Fix GCP disable * Fail when errors listing/clearing bucket * Add env vars * Avoid revoking MD access, as it can be shared. * Fix intermittent failures * Disable MD OAuth testing temporarily. * Add "all" to documentation (#1902) * 10.16.1 patch notes (#1913) * Add bugfixes to change log. * Correct wording & punctuation * Correct version * Export Successfully Updated bytes (#1884) * Add info in error message for mkdir on Log/Plan (#1883) * Microsoft mandatory file * Add info in error message for mkdir on Log/Plan Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> * Fix fixupTokenJson (#1890) * Microsoft mandatory file * Fix fixupTokenJson Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> Co-authored-by: Adam Orosz <[email protected]> * Do not log request/response for container creation error (#1893) * Expose AZCOPY_DOWNLOAD_TO_TEMP_PATH environment variable. (#1895) * Slice against the correct string (#1927) * UX improvement: avoid crash when copying S2S with user delegation SAS (#1932) * Fix bad build + Prevent bad builds in the future (#1917) * Fix bad build + Prevent bad builds in the future * Add Windows build * Make sync use last write time for Azure Files (#1930) * Make sync use last write time for Azure Files * Implement test * 10.16.2 Changelog (#1948) * Update azcopy version * Fixed a bug where preserve permissions would not work with OAuth * Added CODEOWNERS file * Fixed issue where CPK would not be injected on retries * remove OAuth from test * Updated version check string to indicate current AzCopy version (#1969) * added codeowner * Enhance job summary with details about file/folders (#1952) * Add flag to disable version check (#1950) * darwin arm64 * Update golang version to 10.19.2 (#1925) * enable cgo * added tests * Minor fixes: More in description (#1968) * Echo auto-login failure if any * Update help for sync command to use trailing slash on directories * azcopy fail to copy 12TB file to Storage containers in Dev. The logic is used to calculate proper blockSize if it’s not provided, and due to the uint32 cast, it can’t give proper blockSize if filesize is between 50000 * (8 * 1024 * 1024) * X + 1, to 50000 * (8 * 1024 * 1024) * X + 49999. It should return 16MB instead of 8MB blockSize. Accommodated the changes suggested by Narasimha Kulkarni * Added extra logging when switching endpoints * Enable support for preserving SMB info on Linux. (#1723) * Microsoft mandatory file * Enable support for preserving SMB info on Linux. Implemented the GetSDDL/PutSDDL GetSMBProperties/PutSMBProperties methods for Linux using extended attributes. Following are the xattrs we use for fetching/setting various required info. // Extended Attribute (xattr) keys for fetching various information from Linux cifs client. const ( CIFS_XATTR_CREATETIME = "user.cifs.creationtime" // File creation time. CIFS_XATTR_ATTRIB = "user.cifs.dosattrib" // FileAttributes. CIFS_XATTR_CIFS_ACL = "system.cifs_acl" // DACL only. CIFS_XATTR_CIFS_NTSD = "system.cifs_ntsd" // Owner, Group, DACL. CIFS_XATTR_CIFS_NTSD_FULL = "system.cifs_ntsd_full" // Owner, Group, DACL, SACL. ) Majority of the changes are in sddl/sddlHelper_linux.go which implement the following Win32 APIs for dealing with SIDs. ConvertSecurityDescriptorToStringSecurityDescriptorW ConvertStringSecurityDescriptorToSecurityDescriptorW ConvertSidToStringSidW ConvertStringSidToSidW Note: I have skipped Object ACE support in sddl/sddlHelper_linux.go as those should not be used for filesystem properties, only AD object properties. Can someone confirm this? TBD: Conditional SID * Audited, fixed, tested support for "No ACL"/NO_ACCESS_CONTROL and ACL w/o any ACE Tested the following cases: c:\Users\natomar\Downloads>cd testacl // This has "No ACLs" and everyone should be allowed access. c:\Users\natomar\Downloads\testacl>touch NO_ACCESS_CONTROL.txt c:\Users\natomar\Downloads\testacl>cacls NO_ACCESS_CONTROL.txt /S:D:NO_ACCESS_CONTROL Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\NO_ACCESS_CONTROL.txt // This has "No ACLs" and everyone should be allowed access. // It additionally has the "P" (protected) flag set, but that won't have // any effect as that just prevents ACE inheritance but this ACL will // not have any ACLs due to the NO_ACCESS_CONTROL flag. c:\Users\natomar\Downloads\testacl>touch PNO_ACCESS_CONTROL.txt c:\Users\natomar\Downloads\testacl>cacls PNO_ACCESS_CONTROL.txt /S:D:PNO_ACCESS_CONTROL Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\PNO_ACCESS_CONTROL.txt // This should set DACL but with no ACEs, but since "P" is not set it // inherits ACEs from the parent dir. c:\Users\natomar\Downloads\testacl>touch empty_d.txt c:\Users\natomar\Downloads\testacl>cacls empty_d.txt /S:D: Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\empty_d.txt // This should set DACL but with no ACEs, but since "P" is set it // doesn't inherit ACEs from the parent dir and hence this will block // all users. c:\Users\natomar\Downloads\testacl>touch empty_d_with_p.txt c:\Users\natomar\Downloads\testacl>cacls empty_d_with_p.txt /S:D:P Are you sure (Y/N)?y processed file: c:\Users\natomar\Downloads\testacl\empty_d_with_p.txt * Don't fail outright for ACL revision 4. Though our supported ACL types must carry ACL revision 2 as per the doc https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-dtyp/20233ed8-a6c6-4097-aafa-dd545ed24428 but I've seen some dirs have ACL revision 4 but ACL types are still supported ones. So instead of failing upfront, let it fail with unsupported ACE type. Also hexadecimal aceRights are more commonly seen than I expected, so removing a log. * Minor fix after running azcopy on a large dir. This was something which I have doubt on. Now that we got a real world issue due to this, it's all clear :-) * Some minor updates after the rebase to latest Azcopy. * Set default value of flag preserve-smb-info to true on Windows and false on other OS (cherry picked from commit ac5bedb) Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Nagendra Tomar <[email protected]> * Added log indicating a sub-directory is being enqueued (#1999) * Log sync deletions to scanning logger (#2000) * ieproxy fix * remove cgo * fix * fix * fix * more testing * more testing * more testing * more testing * mod tidy * mod tidy * more testing * Added codespell (#2008) * Added codespell * Fixed initial codespell errors * Fix format in codespell.yml * Added s3 url parts * Added CodeQL (#2009) * Added linting file * Upgrade codeql to v2 * Fix incorrect conversion between integer types * Fix GCP URL parts * Fix for rare infinite loop on mutex acquisition (#2012) * small fix * removed test * Added trivy file (#2015) * Added trivy file * renamed trivy * Improve debug-ability of e2e tests by uploading logs of failed jobs (#1898) * Upload testing logs to storage account on failed test * Handle as pipeline artifact instead * mkdirall * copy plan files too * Fix failing tests * Change overwrite to affect any "locked in"/completed state * Fail copy job if single blob does not exist (#1981) * Job fail if single file does not exist * fixed change * fail only on a single file not existing * fail on file not found * fail on file not found * fail on file not found * cleanup * added tests * cleanup * removed test * Correct odd behavior around folder overwrites (#1961) * Fix files sync by determining which LMT to use via smb properties flag (#1958) * Fix files sync by determining which LMT to use via smb properties flag * Implement testing for LMT switch * Fix testing * Limit SMB testing to SMB-compatible environment * Enforce SMB LMT for Linux/MacOS test of SMB LMT preference * Fix metadata parsing (#1953) * Fix metadata parsing * rework metadata parsing to be more robust; add test * Fix comment lines * Codespell :| * Fix ADLSG2 intermittent failure (#1901) * Fix ADLSG2 intermittent failure * Add test * Reduce code dupe * Fix build errors * Fix infinite loop maybe? * Store source token and pass to other threads (#1996) * Store source token * testing * failing pipe * cleanup * test logger * fix test failure * fix 2 * fix * sync fix * cleanup check * Hash based sync (#2020) * Implement hash based sync for MD5 * Implement testing * Ensure folders are handled properly in HBS & Test S2S * Add skip/process logging * Include generic xattr syncmeta application * Fix 0-size blobs * Fix core testing * Revert "Include generic xattr syncmeta application" This reverts commit fba55e4. * Warn on no hash @ source, remove MHP * Comments * Comments * Copy properties from Source (#1964) * Copy properties from Source * Remove unnecessary ws changes * Preserve UNIX properties * Move entity type to Overwrite option * Add python suite * Review comments * Fix test * Release notes and version update (#2028) Co-authored-by: adreed-msft <[email protected]> Co-authored-by: mstenz <[email protected]> Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> Co-authored-by: Adele Reed <[email protected]> Co-authored-by: Karla Saur <[email protected]> Co-authored-by: adam-orosz <[email protected]> Co-authored-by: Adam Orosz <[email protected]> Co-authored-by: Ze Qian Zhang <[email protected]> Co-authored-by: Gauri Prasad <[email protected]> Co-authored-by: Gauri Prasad <[email protected]> Co-authored-by: Tamer Sherif <[email protected]> Co-authored-by: Tamer Sherif <[email protected]> Co-authored-by: reshmav18 <[email protected]> Co-authored-by: linuxsmiths <[email protected]> Co-authored-by: Nagendra Tomar <[email protected]> * Add in content md5 * Change variable name * Change to base64 encoding * Update list.go * Fix the object for md5. * Fix name to mathc help and formating. * Switch to decode to string * Fix syntax error. * Left of b * Undoing conversion * Fix syntax error. * Fixed camelcase. * Fix syntax error. * Fix syntax error. --------- Co-authored-by: Narasimha Kulkarni <[email protected]> Co-authored-by: adreed-msft <[email protected]> Co-authored-by: mstenz <[email protected]> Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> Co-authored-by: Adele Reed <[email protected]> Co-authored-by: Karla Saur <[email protected]> Co-authored-by: adam-orosz <[email protected]> Co-authored-by: Adam Orosz <[email protected]> Co-authored-by: Ze Qian Zhang <[email protected]> Co-authored-by: Gauri Prasad <[email protected]> Co-authored-by: Gauri Prasad <[email protected]> Co-authored-by: Tamer Sherif <[email protected]> Co-authored-by: Tamer Sherif <[email protected]> Co-authored-by: reshmav18 <[email protected]> Co-authored-by: linuxsmiths <[email protected]> Co-authored-by: Nagendra Tomar <[email protected]> * Fail early if we detect invalid URL on one of targets (#2128) * Fail early if we detect invalid URL on one of targets * Add testcase * [AzCopyV10][Bug] Uploading from top directory on linux fails when sub folder has the same name (#2125) * Adding TODOs and bug fix seen with linux subfolders * Add test for relative path and commented out old test * Removed TODOs and cleaned up comments * 10.18 changelog (#2135) * Make golang version configurable (#2137) * Make golang version configurable * Make golang version configurable * Update azure-pipelines.yml for Azure Pipelines * Update azure-pipelines.yml for Azure Pipelines * Increment version # --------- Co-authored-by: mstenz <[email protected]> Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com> Co-authored-by: Mohit Sharma <[email protected]> Co-authored-by: Narasimha Kulkarni <[email protected]> Co-authored-by: Karla Saur <[email protected]> Co-authored-by: adam-orosz <[email protected]> Co-authored-by: Adam Orosz <[email protected]> Co-authored-by: Ze Qian Zhang <[email protected]> Co-authored-by: Gauri Prasad <[email protected]> Co-authored-by: Gauri Prasad <[email protected]> Co-authored-by: Tamer Sherif <[email protected]> Co-authored-by: Tamer Sherif <[email protected]> Co-authored-by: reshmav18 <[email protected]> Co-authored-by: linuxsmiths <[email protected]> Co-authored-by: Nagendra Tomar <[email protected]> Co-authored-by: Richard Kettelerij <[email protected]> Co-authored-by: Norm Estabrook <[email protected]> Co-authored-by: Mikkel Krautz <[email protected]> Co-authored-by: Venkat Malladi <[email protected]> Co-authored-by: siminsavani-msft <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Corrects bug in #1946