Only spanner image deployed

[kristof-mattei]

As per https://github.com/mozilla-services/syncstorage-rs/blame/68d326701505a7c3ae04d59953eb099cf8add4d2/.circleci/config.yml#L322

The repo only publishes the spanner image.

To stay in line with Docker's usage of latest I feel like it would be better to have 2 separate packages produced, syncstorage-rs-mysql & syncstorage-rs-spanner. That way we can publish a latest tag.

If we push it to a single repo the latest tag is ambiguous.

I'll whip out a PR that assumes this split.

Also, is there any want to convert the CI to GitHub Actions? I would (also) love to take a stab at that.

┆Issue is synchronized with this Jira Task

[jrconlin]

One other point of clarity I think I should make.

The docker images that are here are not really meant for "general consumption".
They were created for testing, so there are a fair number of shortcuts we took to get those running in our CI environment.
(Things like building a python3 environment, pulling the spanner emulator, using crap secrets and passwords, etc.) Likewise, the images don't do the general prep work for the data stores, because the test python scripts do that.

Again, we really don't have the bandwidth (or the team) to build this out for general use, so sadly, we have to rely on outside folk to do those improvements.

(One other note: for a lot of reasons, we're moving to a Continuous Deployment model for a few projects, including sync.) The good news is that bugs would get resolved a bit faster, but the down side is that we will no longer have set versions. We're looking at ways that we can try to lock things down where and when possible, but aside from commit versions, I'm not really sure there's much else we can guarantee.)

[kristof-mattei]

@jrconlin thanks for the response.

I'm offering my bandwidth.

I propose to have 3 images, spanner, mysql and the utils one.

That way we can actually use a bleeding-edge label per image which would build off of master, a latest which is the latest tagged release, and of-course, individual tagged releases.

I'll create a PR soon.

[nook24]

Would be great if Mozilla could put a bit more effort into this project. I love the fact to have an open source self hosted sync server.
Unfortunately everything related to this project is just frustrating.
Back in the day when the python sync server was still a thing, I reported several bugs in Firefox on iOS and today there is a shiny new rust implementation almost without any documentation.

And now also the mysql client is gone because the docker images where "not meant for "general consumption"." Why are they public in the first place?

[jrconlin]

It's always good to know a bit of history when trying to understand something.

Sync started off as a firefox add-on. When it was released to production, distributed databases were fairly new, so we kind of built our own out of a bunch of MySQL databases. It turned out you could take one of those servers and stand it up by itself to be your own sync storage system, which a bunch of folk did.

Of course, that system was not great, particularly if you had millions of people storing petabytes of data. Servers failed, a lot, which meant that folks data would go away, which was frustrating to them, since they believed that Sync was a backup system when it was never designed to be that. (It was designed to make sure that two or more instances of Firefox had the same data eventually, your "backup" was another copy of Firefox you were syncing.)

So we updated things, including the application, which was written in Python 2.7, which was no longer supported as of 2020. We picked Rust because it reduced our costs dramatically. We picked Spanner because it also reduced our costs dramatically, and we could become the backup system that folks presumed we were.

All that aside, it's hard to make a solid business case to keep providing MySQL support for the same reason that cars don't come with horse bridals. There's no solid business reason for us to provide it aside from being nice to have. I mean, the Sync Protocol is published, and there's precious little preventing anyone from either creating a new sync server that's compatible with Sync or even taking the Old sync server and updating it to python 3.12.

I can pass along a few future pointers for whoever does something like that:

1. We really, really want to kill off BrowserID. If you want to drop support for any or all of that, more power to you.
2. We also want to move more to OAuth2 scope keys to allow for things like Thunderbird being able to store its data distinctly from your Firefox Data (which also could extend to lots of other things.)
3. There are plans afoot to make Sync both smarter and dumber. "Smarter" means better tracking of active devices so that you don't have a lot of bloat due to old devices that you forgot to get rid of. "Dumber" in that we try not to have as many weird collections with weird storage rules.

As for why these things are public, they're public because all the code we write is public. We clearly need to do a far better job documenting things so that folk don't make bad assumptions, but it can be hard to do that when folk come in with their own sets of assumptions.

[schroedermatthias]

One other point of clarity I think I should make.

The docker images that are here are not really meant for "general consumption". They were created for testing, so there are a fair number of shortcuts we took to get those running in our CI environment. (Things like building a python3 environment, pulling the spanner emulator, using crap secrets and passwords, etc.) Likewise, the images don't do the general prep work for the data stores, because the test python scripts do that.

Fair enough, but then this should also be described in the documentation. As a user of this software, I instinctively assume that a Dockerfile in the repository and an image on Docker Hub exhibit a certain degree of stability (I understand the software is at version 0.15) and are not solely meant for testing. If it's only a test image, it should be clearly labeled as such.