Cloud Saves

[hrydgard]

I found this idea in the reviews on Google Play:

We could add integrated Dropbox or Google Drive support for easy save synchronization between your phone and your PC. That way you can start playing a PSP game on your computer, and then just keep playing where you were on your Android phone on the subway to work, for example, without having to manually copy saves around.

This would primarily apply to real proper game saves as they are small, but bandwidth is so good these days even on mobile that it's possible to imagine that save states could be synced as well. Need to be very careful with versioning for states though.

[Daniel-Griffiths]

Great idea!

[unknownbrackets]

I like this idea, but how do we handle conflicts?

Conflicts can happen with both savedata and states. The most likely example would be that I start out with a synced savegame, and then play on my PC. Later, I play on the train (without internet) on my mobile - without the save - but the game autosaves and so there is a newer savegame created.

The simplest answer is that the most recent wins. I think this choice would, even at worst, not confuse any users.

A more complex answer could involve archiving past X versions or keeping the latest version per device.

For savestates, it's not hard to imagine they might have a different version of PPSSPP on their mobile vs. on desktop, although again the simplest answer there is: just sync it anyway.

-[Unknown]

[hrydgard]

How does dropbox itself handle conflicts? If we use dropbox and it handles it by choosing recent, we don't even need to think about it, right?

[Daniel-Griffiths]

Im unsure of how it works on android but on the PC When dropbox detects that a file has changed it will upload it to the cloud, If I log into another computer with dropbox that has the old version of that file it will overwrite it with the new version that was sent to the cloud.

Dropbox also has the ability to restore old versions of files which is handy if you didnt want to save the update on all your devices.

[dlee13]

@BubblegumBalloon you're thinking of normal syncing (when there aren't any conflicts).

But when two devices want to sync an updated file and there is a conflict, dropbox will upload both and append the name of the newer file with "([name]'s conflicted copy [date])."
And then you can just decide which one you don't want to keep yourself. And dropbox does indeed have the feature for restoring previous versions.

BTW, I would love cloud integration. I've been updating my computer and ipad saves manually through dropbox up until now. :P

dropbox's sync API: https://www.dropbox.com/developers/sync 👍

[unknownbrackets]

Would we, when enabled, just sync the SAVEDATA directory whenever the emulator is started, and shortly after any write access to ms0:/PSP/SAVEDATA/? That would possibly do the trick.

The other option is to add hooks for ms0:/PSP/SAVEDATA/ accesses, which would solve the problem of having the emulator open on both desktop and mobile simultaneously, but it's more complicated and would probably be slower. Instead maybe resync every time a game is started would be fine.

"x's conflicted copy" wouldn't be very visible, but at least it provides for recovery options.

I think an interesting thing would be to keep a "history" of savedata, which some cloud services might even support. Even without that, one could copy writes to datestamped directories in ms0:PSP/SAVEDATA/BACKUP/ or something.

Also, since savestates are now (mostly?) compatible between platforms, it's pretty reasonable to sync them too.

-[Unknown]

[hrydgard]

I need to look into the APIs of Dropbox and Google Drive and see what they let us do as an app, but simply syncing all of SAVEDATA and PPSSPP_STATE at chosen times seems like a fine approach.

Or we could write our own backend I suppose but that's a bigger project...

[bollu]

what's the difference between savedata and savestate? (I've never heard that terminology before)

[hrydgard]

It's derived from the directory, "SAVEDATA", that the PSP saves in. So savedata = ingame proper save that you can transfer to your PSP. Savestate = just an emulator snapshot of everything, at any time.

[Amanelle]

I just want to note that there are at least two other emulators which have this option. I'm talking about nds4droid and nds4ios, and it works great. I'm not sure how it was done but I can keeps my saves synched between my pc and iPad thanks to that, and in fact I only created my dropbox account for this specific use.

[Xemiru]

Install data from games is something to note, too.
Games like [Mana Khemia: Student Alliance] require such data to have things like combat voices, and the data usually has more than 50mbs worth of stuff.

[Ofunniku]

While this sounds like a really nice idea, can you have a look at https://mega.co.nz/#dev ?
This is basically a cloud storage service like dropbox and google drive but you start off with 50GB and files on the cloud are encrypted with a key that is only used on the client-side. (Locally generated key is encrypted with your password before sending the encrypted key to the server, on login client grabs encrypted key from server, decrypts key locally using your password. Filenames and data-streams are encrypted when received and are decrypted locally...
...I think).

The massive storage capacity is a boon for certain games.

I currently use it for saves(tates) for all my emulators by adding all the folders to the desktop sync client, and it works really well keeping my desktops and notebooks all properly updated.

[ThePhoenixRises]

I just had an idea of having at least on the PC side of things a system, in which hovering your cursor over a created savestate on the pause menu of the emulator, could display which version of the emulator that particular savestate was created on, thus taking out the guess work of which one was the latest save, and for people who have lost all there save data on a game because its not able to be loaded into the latest version, could check to see which version it belongs too and they could download that version of PPSSPP again to load that savestate, then you would be able to save it hopefully as a proper in game save, then load that into the latest version for it to work okay, so as to continue there game.
Then use Dropbox to backup all game saves including savestates as you mentioned.
Sounds like a brilliant idea using a Cloud Storage.
I am in favour of using this system if introduced, as already have a Dropbox account synced with my Tablet & PC.

[dlee13]

Oh boy, this was started over a year ago. I have since then updated my iPad to unjailbroken iOS 7, so I no longer have PPSSPP on it. :P

[oliverschloebe]

+1 Would love this feature!

[Daniel-ltw]

+1 on this as well

[konistehrad]

Good news bad news time: I've actually started work on this on the Dropbox side, and have made some pretty serious progress. That said, when it comes to approach, there's some interesting engineering decisions that we would need to make:

• Platform-specific cloud API's vs. a C-based API that goes everywhere.
  At the moment, I've preferred using platform-specific libraries. The reasoning is two-fold:
  1. The most recently updated C library for Dropbox no longer functions, as it targeted the v1 Dropbox API, which has been shut down.
  2. Using platform-native libraries on mobile allows Dropbox in particular to go to an installed and authenticated native application for its OAuth token, rather than opening a browser window to perform the OAuth step.

While this leaves desktops in the dark a little, desktop platforms have excellent first-party Dropbox clients which we can lean on, as opposed to finding bindings for every single platform.

Regarding conflicts, I'd love to set a requirement right now that we don't support the case where you're playing the same game at the same time on both Desktop and Mobile. The use case is somewhat pathologic, and while we'll have to deal with some "I was offline but I'm back online now!" logic, I don't think it's worth it spinning our wheels on "Same game same time different computers."

As for implementation, in an ideal case, I've been treating it like this: When a game is being played, SaveState::SaveSlot raises System_SendMessage, sending STATE_SAVED or the like as the command with the path of the save file and thumbnail as the parameter. Same for saved games. This causes an immediate push to the server, or a backing data store marks the file as "unsync'd", and at next STATE_SAVED it'll check to see if connection has been restored, and try to push all these files if possible.

Let me know if any of this sounds crazy, or if I'm way off the mark.

[unknownbrackets]

Hm, are you doing saves also are save states only? We don't want to encourage people to use save states as their main way of saving. People already do that, but it's bad.

I agree that the game being played on multiple devices simultaneously is not terribly interesting. But I think the case of (for example) playing a game on an airplane, subway, or train is a much more common one.

The worst case would be this IMHO:

• I play a game on my laptop or PC or something, and it syncs. Yay.
• Later, I get on a plane and start up the game. The game, like many PSP games, autosaves. As soon as I start it, I realize I've lost progress because I don't have my PC game data synced, and quit.
• When I return from my trip, I start up my mobile again to sync. Now it has wifi.
• The old save data from my mobile is synced as the "newest" data, and overwrites my data on my PC.

This is just an example - there are similar use cases when users get new phones, for example.

-[Unknown]

[hrydgard]

Yeah, that case is a tricky one .. not sure how solvable that even is, with plain file syncing and no additional smartness.

[konistehrad]

Oh, yes, I should clarify, this absolutely is done for both. I'd be pretty irritated if it only did save states and not game saves haha.

Yeah, your example is the exact case I was worried about with "I was offline but I'm back online now"; indeed I've lost data because of inconsistent syncing behavior post-subway. I've spent some time today looking for at least a guidepost on how best to resolve these issues (since of course Dropbox has killed their Sync API 😕), and I've found scattered implementations elsewhere. rclone looks to best suit our needs (FS sync with multiple backends), but of course it's implemented in go, making it less useful for us.

I'll keep digging around, see what's good. In the meantime, if you've got a better lead I'll absolutely take it!

[unknownbrackets]

Personally I think my example can't really be resolved without involving the user. I think the options are:

1. After sync has been offline (either initial setup or after subway), verify conflicts and let the user decide if any are found.
2. When offline, but previously synced, ask the user if they want to "keep trying" or "discard".
3. Keep a version history, at least X versions back, so the user can revert if a Bad Thing happens.
4. Show a message in the UI when there are "pending sync" saves and let the user discard them.

The problem is game autosave - it's hard to tell from PPSSPP's perspective if the user actively wanted to save, or not. For save states, I think it'd be mostly fine to take the latest one, since that's always a positive action.

-[Unknown]

[unknownbrackets]

Just wanted to note: I brought up this example because I think it's important to keep in mind. But I would definitely not be opposed to a PR that implemented something with bad sync behavior - wasn't trying to dissuade.

Also - I realized that we do know when the user selected a save from the "default UI" - not all games use this, but we could also consider taking that as a positive action. Maybe that's not best though, since it'd be a bit of an inconsistent experience between games and might be surprising.

-[Unknown]

[carlos-bernal-gby]

+1 On this feature!

[hrydgard]

Someone recommended me to look at myboy!'s implementation (it's a gameboy emulator).

[AkiraJkr]

There's also DraStic that has such a feature, for Google Drive, so you can check that out too.

[oldmario]

is the googledrive upload only automatic or can users manually do it?

if it's possible to set up the ability for users to manually do it then an interface or method similar to the manual save uploading on the PS4 could be used, basically it shows the time and dates of both the save files on the system and in the cloud but also i think it allows you to see the save file information so you've got the ability to view the name, playtime, level ect...

[ShadyGuyJose]

Someone recommended me to look at myboy!'s implementation (it's a gameboy emulator).

The John line of emulators, while universally hated in the scene (for violating open source licenses or something along those lines), does this brilliantly with Dropbox (although they do require a separate app, John DataSync, to be installed along with their emulators). This allows users to point their desktop emulators' save directories to the ones created by John DataSync in Dropbox, and they work seamlessly unless you're careless enough to play on one platform before the other syncs properly.

[jcchikikomori]

I am curious about how iOS can handle this kind of thing?

[ghost]

I wish this happen in the future-prio cloud saving is a great feature ;)

[hrydgard]

I really want to do something in this area next year, but struggling with finding viable options with minimal pain.

For now disregarding the problems with conflicts and resolution, a much bigger issue is actually finding a place to store the save data.

The ideal would be a service where users already have their own accounts with some free storage, like Google Drive. Many PC users and most Android users have this already, as they'll have Google accounts - but not all! Many Chinese phone brands and several from other countries too don't require or even encourage setting up Google accounts.

Massive drawback of Google Drive is that there's no official way to access it from C++. Really! And barely any open source libraries either, and those that exist are either massively outdated or just weird.

Dropbox could be an alternative, and I'm sure there are more? Do they have a reasonable C++ API?

We could also set up our own savedata storage online on top of some blobstore service (S3 or something - a relational database would be overkill) and allow all PPSSPP users free access, though it would come with the pain of doing our own account management and so on, and scale. Could allow "Login with google" for quick onboarding for google users, but will definitely need real account management too. If we'd restrict this to real saves (with careful size and rate limiting to avoid abuse) and don't include save states, this could be done reasonably cheaply I think. Huge drawback is all the added sysadmin pain and responsibility.

[oldmario]

not really sure about them having a C++ API but on top of Gdrive, Dropbox mentioned already, Onedrive, Box, Mega, and maybe iCloud could be considered to help widen the range

[jcchikikomori]

I really want to do something in this area next year, but struggling with finding viable options with minimal pain.

For now disregarding the problems with conflicts and resolution, a much bigger issue is actually finding a place to store the save data.

The ideal would be a service where users already have their own accounts with some free storage, like Google Drive. Many PC users and most Android users have this already, as they'll have Google accounts - but not all! Many Chinese phone brands and several from other countries too don't require or even encourage setting up Google accounts.

Massive drawback of Google Drive is that there's no official way to access it from C++. Really! And barely any open source libraries either, and those that exist are either massively outdated or just weird.

Dropbox could be an alternative, and I'm sure there are more? Do they have a reasonable C++ API?

We could also set up our own savedata storage online on top of some blobstore service (S3 or something - a relational database would be overkill) and allow all PPSSPP users free access, though it would come with the pain of doing our own account management and so on, and scale. Could allow "Login with google" for quick onboarding for google users, but will definitely need real account management too. If we'd restrict this to real saves (with careful size and rate limiting to avoid abuse) and don't include save states, this could be done reasonably cheaply I think. Huge drawback is all the added sysadmin pain and responsibility.

I see what you mean. But i've been thinking some options though...

1. Custom PPSSPP Cloud Saving Server Code/API(s), for building custom cloud data management servers which is it's ALL on user's responsibility
2. Firebase Login (Google, web-based). Some chinese applications that are published outside China (esp.: Southeast Asia) have Google login using Firebase

[IrfanH495]

this is the first time i felt this feature would be useful, i always ignored it in espxe and drastic.
please add after version 16/17.

[saulojoab]

iCloud seems not to work whatsoever on my non-jailbroken iPhone. Whenever I try setting the memory stick folder to an iCloud folder, PPSSPP just thinks there's no available space for some reason... Also tried Dropbox (the basic plan where you can't access files offline) with no success...

Has anyone been able to sync save files on a Mac and an iPhone (non-jailbroken)?

[joze2016]

+1 On this feature!
Delta has this feature to synchronize game saves via Google Drive or Dropbox, and if there is a conflict, it allows you to manually select whether the device or the cloud will take precedence. See the delta implementation please.

[hoho97]

Would be happy with this feature. I now use siri shortcut to upload savedata to icloud from my iPhone and then draw them to my iPad/PC.
Not a heavy job but pain in ass if accidently overwrite the savedata.

[archanox]

I'd just like to add my vote for this feature request. As ppsspp on iOS is GA there's no real way to automatically sync save games, that I'm aware of. I use ppsspp on Windows, Android, Linux and macOS, which I've been able to use OneDrive to sync my saves up until this point. Personally I don't particularly care where the saves are stored as long as it is free. But I'd really like to emphasise that I'd like it to function on all platforms.

For bonus points, I'd also like to see cheat syncing and possibly on-demand ISO downloads from cloud storage.

[saulojoab]

as an alternative for those who are interested, you can use the PPSSPP core on Retroarch, then setup a Cloud Save using Syncthing (Android only I think).

i'm doing it right now and it works great :) All of my states, saves and roms are saved between my phone and my PC automatically.

[joze2016]

as an alternative for those who are interested, you can use the PPSSPP core on Retroarch, then setup a Cloud Save using Syncthing (Android only I think).

i'm doing it right now and it works great :) All of my states, saves and roms are saved between my phone and my PC automatically.

However, RetroArch's support for some games is not as stable as PPSSPP's, such as God of War, and it is more complicated to configure and operate.

[saulojoab]

@joze2016 Yup, I noticed that after a while.

However my Retroarch saves also work for PPSSPP, so syncing with Syncthing worked with PPSSPP as well :)

[archanox]

Is this penciled in for an upcoming version @hrydgard, or still on the backburner? I've got too many devices that I'd like to play PPSSPP on and continue my progress on, but am currently unable to do so or is too laborious on newly provisioned devices.

[AbdelrahmanMahmoudMD]

After 11 years of requesting this feature, I just want to thank everyone who is working on this emulator.

I'm not a developer however, regarding sync feature, a small suggestion -don't know if it's helpful or not- there's a program called Anki (https://apps.ankiweb.net/), it's a flashcard app which sync between all types of systems without google account/dropbox or anything, maybe it can be helpful. Thanks you all.

@hrydgard

edit : here's the source code of it (https://github.com/ankitects/anki)

[ShadyGuyJose]

After 11 years of requesting this feature, I just want to thank everyone who is working on this emulator.

I'm not a developer however, regarding sync feature, a small suggestion -don't know if it's helpful or not- there's a program called Anki (https://apps.ankiweb.net/), it's a flashcard app which sync between all types of systems without google account/dropbox or anything, maybe it can be helpful. Thanks you all.

@hrydgard

edit : here's the source code of it (https://github.com/ankitects/anki)

Anki uses its own cloud service (Ankiweb), which requires servers. @hrydgard probably doesn't want to spend money running servers for users to sync their savedata (especially since these files are heavier than simple flashcards), so that's why using a third-party solution that users would have to provide their own accounts for would be ideal.