[AzCopyV10] Add in content-md5 to available list properties (#2033)

* Release 10.17.0 (#2029)

* Add mitigation for weird NtQuerySecurityObject behavior on NAS sources (#1872)

* Add check for 0 length, attempt to validate the returned object.

* Change to grabbing real SD length

* Add comment describing issue

* Prevent infinite loop upon listing failure

* Fix GCP error checking

* Fix GCP disable

* Fix bad URL delete (#1892)

* Manipulate URLs safely

* Fix folder deletion test

* Prevent infinite loop upon listing failure

* Fix GCP error checking

* Fix GCP disable

* Fail when errors listing/clearing bucket

* Update MacOS testing pipeline (#1896)

* fixing small typo (,) in help of jobs clean (#1899)

* Microsoft mandatory file

* fixing small typo (,) in help of jobs clean

Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com>
Co-authored-by: Mohit Sharma <[email protected]>

* Implement MD OAuth testing (#1859)

* Implement MD OAuth testing

* Handle async on RevokeAccess, handle job cancel/failure better

* Prevent parallel testing of managed disks

* lint check

* Prevent infinite loop upon listing failure

* Fix GCP error checking

* Fix GCP disable

* Fail when errors listing/clearing bucket

* Add env vars

* Avoid revoking MD access, as it can be shared.

* Fix intermittent failures

* Disable MD OAuth testing temporarily.

* Add "all" to documentation (#1902)

* 10.16.1 patch notes (#1913)

* Add bugfixes to change log.

* Correct wording & punctuation

* Correct version

* Export Successfully Updated bytes (#1884)

* Add info in error message for mkdir on Log/Plan (#1883)

* Microsoft mandatory file

* Add info in error message for mkdir on Log/Plan

Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com>
Co-authored-by: Mohit Sharma <[email protected]>

* Fix fixupTokenJson (#1890)

* Microsoft mandatory file

* Fix fixupTokenJson

Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com>
Co-authored-by: Mohit Sharma <[email protected]>
Co-authored-by: Adam Orosz <[email protected]>

* Do not log request/response for container creation error (#1893)

* Expose AZCOPY_DOWNLOAD_TO_TEMP_PATH environment variable. (#1895)

* Slice against the correct string (#1927)

* UX improvement: avoid crash when copying S2S with user delegation SAS (#1932)

* Fix bad build + Prevent bad builds in the future (#1917)

* Fix bad build + Prevent bad builds in the future

* Add Windows build

* Make sync use last write time for Azure Files (#1930)

* Make sync use last write time for Azure Files

* Implement test

* 10.16.2 Changelog (#1948)

* Update azcopy version

* Fixed a bug where preserve permissions would not work with OAuth

* Added CODEOWNERS file

* Fixed issue where CPK would not be injected on retries

* remove OAuth from test

* Updated version check string to indicate current AzCopy version (#1969)

* added codeowner

* Enhance job summary with details about file/folders (#1952)

* Add flag to disable version check (#1950)

* darwin arm64

* Update golang version to 10.19.2 (#1925)

* enable cgo

* added tests

* Minor fixes: More in description (#1968)

* Echo auto-login failure if any

* Update help for sync command to use trailing slash on directories

* azcopy fail to copy 12TB file to Storage containers in Dev.

The logic is used to calculate proper blockSize if it’s not provided, and due to the uint32 cast, it can’t give proper blockSize if filesize is between 50000 * (8 * 1024 * 1024) * X + 1, to 50000 * (8 * 1024 * 1024) * X + 49999. It should return 16MB instead of 8MB blockSize.

Accommodated the changes suggested by Narasimha Kulkarni

* Added extra logging when switching endpoints

* Enable support for preserving SMB info on Linux. (#1723)

* Microsoft mandatory file

* Enable support for preserving SMB info on Linux.

Implemented the GetSDDL/PutSDDL GetSMBProperties/PutSMBProperties
methods for Linux using extended attributes.
Following are the xattrs we use for fetching/setting various required
info.

// Extended Attribute (xattr) keys for fetching various information from Linux cifs client.
const (
        CIFS_XATTR_CREATETIME     = "user.cifs.creationtime" // File creation time.
        CIFS_XATTR_ATTRIB         = "user.cifs.dosattrib"    // FileAttributes.
        CIFS_XATTR_CIFS_ACL       = "system.cifs_acl"        // DACL only.
        CIFS_XATTR_CIFS_NTSD      = "system.cifs_ntsd"       // Owner, Group, DACL.
        CIFS_XATTR_CIFS_NTSD_FULL = "system.cifs_ntsd_full"  // Owner, Group, DACL, SACL.
)

Majority of the changes are in sddl/sddlHelper_linux.go which implement
the following Win32 APIs for dealing with SIDs.

	ConvertSecurityDescriptorToStringSecurityDescriptorW
	ConvertStringSecurityDescriptorToSecurityDescriptorW
	ConvertSidToStringSidW
	ConvertStringSidToSidW

Note: I have skipped Object ACE support in sddl/sddlHelper_linux.go as
      those should not be used for filesystem properties, only AD object
      properties.
      Can someone confirm this?

TBD:
Conditional SID

* Audited, fixed, tested support for "No ACL"/NO_ACCESS_CONTROL and ACL w/o any ACE

Tested the following cases:

c:\Users\natomar\Downloads>cd testacl

// This has "No ACLs" and everyone should be allowed access.
c:\Users\natomar\Downloads\testacl>touch NO_ACCESS_CONTROL.txt
c:\Users\natomar\Downloads\testacl>cacls NO_ACCESS_CONTROL.txt /S:D:NO_ACCESS_CONTROL
Are you sure (Y/N)?y
processed file: c:\Users\natomar\Downloads\testacl\NO_ACCESS_CONTROL.txt

// This has "No ACLs" and everyone should be allowed access.
// It additionally has the "P" (protected) flag set, but that won't have
// any effect as that just prevents ACE inheritance but this ACL will
// not have any ACLs due to the NO_ACCESS_CONTROL flag.
c:\Users\natomar\Downloads\testacl>touch PNO_ACCESS_CONTROL.txt
c:\Users\natomar\Downloads\testacl>cacls PNO_ACCESS_CONTROL.txt /S:D:PNO_ACCESS_CONTROL
Are you sure (Y/N)?y
processed file: c:\Users\natomar\Downloads\testacl\PNO_ACCESS_CONTROL.txt

// This should set DACL but with no ACEs, but since "P" is not set it
// inherits ACEs from the parent dir.
c:\Users\natomar\Downloads\testacl>touch empty_d.txt
c:\Users\natomar\Downloads\testacl>cacls empty_d.txt /S:D:
Are you sure (Y/N)?y
processed file: c:\Users\natomar\Downloads\testacl\empty_d.txt

// This should set DACL but with no ACEs, but since "P" is set it
//  doesn't inherit ACEs from the parent dir and hence this will block
// all users.
c:\Users\natomar\Downloads\testacl>touch empty_d_with_p.txt
c:\Users\natomar\Downloads\testacl>cacls empty_d_with_p.txt /S:D:P
Are you sure (Y/N)?y
processed file: c:\Users\natomar\Downloads\testacl\empty_d_with_p.txt

* Don't fail outright for ACL revision 4.

Though our supported ACL types must carry ACL revision 2 as per the doc

https://docs.microsoft.com/en-us/openspecs/windows_protocols/ms-dtyp/20233ed8-a6c6-4097-aafa-dd545ed24428

but I've seen some dirs have ACL revision 4 but ACL types are still
supported ones. So instead of failing upfront, let it fail with
unsupported ACE type.

Also hexadecimal aceRights are more commonly seen than I expected, so
removing a log.

* Minor fix after running azcopy on a large dir.

This was something which I have doubt on. Now that we got a real world
issue due to this, it's all clear :-)

* Some minor updates after the rebase to latest Azcopy.

* Set default value of flag preserve-smb-info to true on Windows and false on other OS

(cherry picked from commit ac5bedb)

Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com>
Co-authored-by: Nagendra Tomar <[email protected]>

* Added log indicating a sub-directory is being enqueued (#1999)

* Log sync deletions to scanning logger (#2000)

* ieproxy fix

* remove cgo

* fix

* fix

* fix

* more testing

* more testing

* more testing

* more testing

* mod tidy

* mod tidy

* more testing

* Added codespell (#2008)

* Added codespell

* Fixed initial codespell errors

* Fix format in codespell.yml

* Added s3 url parts

* Added CodeQL (#2009)

* Added linting file

* Upgrade codeql to v2

* Fix incorrect conversion between integer types

* Fix GCP URL parts

* Fix for rare infinite loop on mutex acquisition (#2012)

* small fix

* removed test

* Added trivy file (#2015)

* Added trivy file

* renamed trivy

* Improve debug-ability of e2e tests by uploading logs of failed jobs (#1898)

* Upload testing logs to storage account on failed test

* Handle as pipeline artifact instead

* mkdirall

* copy plan files too

* Fix failing tests

* Change overwrite to affect any "locked in"/completed state

* Fail copy job if single blob does not exist (#1981)

* Job fail if single file does not exist

* fixed change

* fail only on a single file not existing

* fail on file not found

* fail on file not found

* fail on file not found

* cleanup

* added tests

* cleanup

* removed test

* Correct odd behavior around folder overwrites (#1961)

* Fix files sync by determining which LMT to use via smb properties flag (#1958)

* Fix files sync by determining which LMT to use via smb properties flag

* Implement testing for LMT switch

* Fix testing

* Limit SMB testing to SMB-compatible environment

* Enforce SMB LMT for Linux/MacOS test of SMB LMT preference

* Fix metadata parsing (#1953)

* Fix metadata parsing

* rework metadata parsing to be more robust; add test

* Fix comment lines

* Codespell :|

* Fix ADLSG2 intermittent failure (#1901)

* Fix ADLSG2 intermittent failure

* Add test

* Reduce code dupe

* Fix build errors

* Fix infinite loop maybe?

* Store source token and pass to other threads (#1996)

* Store source token

* testing

* failing pipe

* cleanup

* test logger

* fix test failure

* fix 2

* fix

* sync fix

* cleanup check

* Hash based sync (#2020)

* Implement hash based sync for MD5

* Implement testing

* Ensure folders are handled properly in HBS & Test S2S

* Add skip/process logging

* Include generic xattr syncmeta application

* Fix 0-size blobs

* Fix core testing

* Revert "Include generic xattr syncmeta application"

This reverts commit fba55e4.

* Warn on no hash @ source, remove MHP

* Comments

* Comments

* Copy properties from Source (#1964)

* Copy properties from Source

* Remove unnecessary ws changes

* Preserve UNIX properties

* Move entity type to Overwrite option

* Add python suite

* Review comments

* Fix test

* Release notes and version update (#2028)

Co-authored-by: adreed-msft <[email protected]>
Co-authored-by: mstenz <[email protected]>
Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com>
Co-authored-by: Mohit Sharma <[email protected]>
Co-authored-by: Adele Reed <[email protected]>
Co-authored-by: Karla Saur <[email protected]>
Co-authored-by: adam-orosz <[email protected]>
Co-authored-by: Adam Orosz <[email protected]>
Co-authored-by: Ze Qian Zhang <[email protected]>
Co-authored-by: Gauri Prasad <[email protected]>
Co-authored-by: Gauri Prasad <[email protected]>
Co-authored-by: Tamer Sherif <[email protected]>
Co-authored-by: Tamer Sherif <[email protected]>
Co-authored-by: reshmav18 <[email protected]>
Co-authored-by: linuxsmiths <[email protected]>
Co-authored-by: Nagendra Tomar <[email protected]>

* Add in content md5

* Change variable name

* Change to base64 encoding

* Update list.go

* Fix the object for md5.

* Fix name to mathc help and formating.

* Switch to decode to string

* Fix syntax error.

* Left of b

* Undoing conversion

* Fix syntax error.

* Fixed camelcase.

* Fix syntax error.

* Fix syntax error.

---------

Co-authored-by: Narasimha Kulkarni <[email protected]>
Co-authored-by: adreed-msft <[email protected]>
Co-authored-by: mstenz <[email protected]>
Co-authored-by: microsoft-github-policy-service[bot] <77245923+microsoft-github-policy-service[bot]@users.noreply.github.com>
Co-authored-by: Mohit Sharma <[email protected]>
Co-authored-by: Adele Reed <[email protected]>
Co-authored-by: Karla Saur <[email protected]>
Co-authored-by: adam-orosz <[email protected]>
Co-authored-by: Adam Orosz <[email protected]>
Co-authored-by: Ze Qian Zhang <[email protected]>
Co-authored-by: Gauri Prasad <[email protected]>
Co-authored-by: Gauri Prasad <[email protected]>
Co-authored-by: Tamer Sherif <[email protected]>
Co-authored-by: Tamer Sherif <[email protected]>
Co-authored-by: reshmav18 <[email protected]>
Co-authored-by: linuxsmiths <[email protected]>
Co-authored-by: Nagendra Tomar <[email protected]>

cmd/helpMessages.go

@@ -258,7 +258,7 @@ const listCmdShortDescription = "List the entities in a given resource"
 const listCmdLongDescription = `List the entities in a given resource. Blob, Files, and ADLS Gen 2 containers, folders, and accounts are supported.`
 
 const listCmdExample = "azcopy list [containerURL] --properties [semicolon(;) separated list of attributes " +
-	"(LastModifiedTime, VersionId, BlobType, BlobAccessTier, ContentType, ContentEncoding, LeaseState, LeaseDuration, LeaseStatus) " +
+	"(LastModifiedTime, VersionId, BlobType, BlobAccessTier, ContentType, ContentEncoding, ContentMD5, LeaseState, LeaseDuration, LeaseStatus) " +
 	"enclosed in double quotes (\")]"
 
 // ===================================== LOGIN COMMAND ===================================== //

cmd/list.go

@@ -26,6 +26,7 @@ import (
 	"fmt"
 	"strconv"
 	"strings"
+	"encoding/base64"
 
 	"github.com/Azure/azure-pipeline-go/pipeline"
 
@@ -54,6 +55,7 @@ const (
 	blobAccessTier   validProperty = "BlobAccessTier"
 	contentType      validProperty = "ContentType"
 	contentEncoding  validProperty = "ContentEncoding"
+	contentMD5       validProperty = "ContentMD5"
 	leaseState       validProperty = "LeaseState"
 	leaseDuration    validProperty = "LeaseDuration"
 	leaseStatus      validProperty = "LeaseStatus"
@@ -63,7 +65,7 @@ const (
 // validProperties returns an array of possible values for the validProperty const type.
 func validProperties() []validProperty {
 	return []validProperty{lastModifiedTime, versionId, blobType, blobAccessTier,
-		contentType, contentEncoding, leaseState, leaseDuration, leaseStatus, archiveStatus}
+		contentType, contentEncoding, contentMD5, leaseState, leaseDuration, leaseStatus, archiveStatus}
 }
 
 func (raw *rawListCmdArgs) parseProperties(rawProperties string) []validProperty {
@@ -177,6 +179,8 @@ func (cooked cookedListCmdArgs) processProperties(object StoredObject) string {
 			builder.WriteString(propertyStr + ": " + object.contentType + "; ")
 		case contentEncoding:
 			builder.WriteString(propertyStr + ": " + object.contentEncoding + "; ")
+		case contentMD5:
+			builder.WriteString(propertyStr + ": " + base64.StdEncoding.EncodeToString(object.md5) + "; ")
 		case leaseState:
 			builder.WriteString(propertyStr + ": " + string(object.leaseState) + "; ")
 		case leaseStatus: