Existing sync folders give "Root directoy is not empty" after API credentials change

[rohfle]

I ran into this issue during #426 and pull request #428

WHAT
After the ClientID / ClientSecret is changed, gdrive sync upload results in the following error
Root directoy is not empty, the initial sync requires an empty directory

WHY DOES THIS HAPPEN

• gdrive uses app-specific appProperties to store metadata in sync folders, which is used during the syncing process
• Changing API credentials means that Google APIs see gdrive as a different "app", so appProperties are no longer present

IMPLEMENTATION SPECIFICS
Usage of sync for reference - gdrive sync upload /path/to/localdir **FOLDER_ID**

---

1. **FOLDER_ID**, the sync root folder, has the following appProperties set:

{
    "sync": "true",
    "syncRoot": "true"
}

gdrive/drive/sync_upload.go

Line 126 in c3cbcce

// Update directory with syncRoot property

---

2. Every file and subfolder in /path/to/localdir has the following appProperties set when they are uploaded:

{
    "sync": "true",
    "syncRootId": **FOLDER_ID**,
}

gdrive/drive/sync_upload.go

Line 260 in c3cbcce

dstFile := &drive.File{

and

gdrive/drive/sync_upload.go

Line 299 in c3cbcce

dstFile := &drive.File{

---

3. When gdrive syncs again, it uses this query "appProperties has {key='syncRootId' and value='%s'}" to get a complete listing of the synced remote files

gdrive/drive/sync.go

Line 155 in c3cbcce

func (self *Drive) prepareRemoteFiles(rootDir *drive.File, sortOrder string) ([]*RemoteFile, error) {

SUGGESTIONS FOR FUTURE
properties could be used instead of appProperties with namespaced keys (ie starting with gdrive-*) so that when the client credentials change, the sync properties are still visible

[rohfle]

My workaround was to use a python script to reconstruct the appProperties. It takes a looong time to do this... but it seems to work. Tread carefully as this is not well tested.

PREREQUISITES

• Google API client credentials (see steps 3-14 here) - move to client_id.json in the same folder as the script

CODE
Add your sync root folder ids to folder_rootids

from __future__ import print_function
import pickle
import os.path
from pprint import pprint
from googleapiclient.discovery import build
from google_auth_oauthlib.flow import InstalledAppFlow
from google.auth.transport.requests import Request

# If modifying these scopes, delete the file token.pickle.
SCOPES = ['https://www.googleapis.com/auth/drive']

def main():
    folder_rootids = [
        # Add your sync root folders here
    ]

    creds = None
    # The file token.pickle stores the user's access and refresh tokens, and is
    # created automatically when the authorization flow completes for the first
    # time.
    if os.path.exists('token.pickle'):
        with open('token.pickle', 'rb') as token:
            creds = pickle.load(token)
    # If there are no (valid) credentials available, let the user log in.
    if not creds or not creds.valid:
        if creds and creds.expired and creds.refresh_token:
            creds.refresh(Request())
        else:
            flow = InstalledAppFlow.from_client_secrets_file(
                'client_id.json', SCOPES)
            creds = flow.run_local_server()
        # Save the credentials for the next run
        with open('token.pickle', 'wb') as token:
            pickle.dump(creds, token)

    service = build('drive', 'v3', credentials=creds)

    for rootid in folder_rootids:
        parents = [rootid]
        while len(parents) > 0:
            parentid = parents.pop(0)
            print('  Processing children in subfolder id "{}"'.format(parentid))
            request = service.files().list(pageSize=10,
                                          q="'{}' in parents and trashed=false".format(parentid),
                                          fields='nextPageToken, files(id, name, mimeType, appProperties)')
            while request is not None:
                result = request.execute()
                for file in result.get('files', []):
                    if file['mimeType'] == 'application/vnd.google-apps.folder':
                        parents += [file['id']]
                    print('    Fixing file name "{}" ({})'.format(file['name'], file['id']))
                    service.files().update(
                        fileId=file['id'],
                        body={
                            "appProperties": {
                                "sync": "true",
                                "syncRootId": rootid,
                            }
                        }
                    ).execute()
                request = service.files().list_next(request, result)
        print('Processing root folder id "{}"'.format(rootid))
        result = service.files().get(fileId=rootid, fields='id, name, appProperties').execute()
        service.files().update(
            fileId=rootid,
            body={
                "appProperties": {
                    "sync": "true",
                    "syncRoot": "true"
                }
            }
        ).execute()
        print('COMPLETE')

if __name__ == '__main__':
    main()

[mbenlioglu]

I believe sync_download.go should implement "prepareSyncRoot" and related mechanisms similar to the sync_upload.go

gdrive/drive/sync_upload.go

Line 97 in c3cbcce

func (self *Drive) prepareSyncRoot(args UploadSyncArgs) (*drive.File, error) {

So, existing drive folders can be converted to a sync folder if wanted, or in this case missing appProperties can be recreated.

That way I'd expect following command to fix issues like this.

$ gdrive sync download --keep-local <DIR_ID> /path/to/local

[rohfle]

Additionally to stop this from happening in future there is a further extension. Instead of using app-specific properties metadata, one could use the global properties, but namespace them by appending gdrive- to the key names. Then, if the client id changes in future, the syncrootid would still be accessible.

[dev-slash-null]

Also ran into the issue of the sync directory no longer being recognized after changing oauth client credentials. @rohfle : Is your python script from Jan 31 still the best recourse for using an existing sync directory with new oauth credentials? I agree that as a long-term solution, namespaced global properties would be preferable to app-specific properties that depend on the exact credentials.

[rohfle]

I haven't seen any other solutions to the issue. It should still work but I haven't tried it in a while.

Good luck!

[dev-slash-null]

Thanks for letting me know. I'll give it a try.