Decouple default archives list from online source

[machawk1]

MemGator currently looks to http://git.io/archives on startup by default. If git.io goes down, MemGator has no default list of archives. Coupling a local service's functionality to a remote online resource is bad. MemGator ought to work with smart defaults without relying on this resource.

INFO: 2016/09/08 22:02:01.250427 Initializing MemGator:1.0-rc5...
INFO: 2016/09/08 22:02:01.250524 Loading archives from http://git.io/archives
FATAL: main.go:831: Error reading list of archives (http://git.io/archives): Get http://git.io/archives: dial tcp: lookup git.io: no such host

[ibnesayeed]

I thought about it earlier when MemGator was born and concluded that the current approach is the most practical, perhaps not ideal though. The other two approaches I thought about are following:

• A default archive list in the code would remove the reliance on an external service or a config file, but would require updates to the binary each time something changes in the curated list of default archives.
• A fallback archive list in the code that is used only when reading the default registry fails, would show a magical behavior as the user might not even notice what archives are actually being aggregated if not paying careful attention on the startup message. Additionally, it would suffer the same problem as the previous approach.
• A local default archive list file would add additional steps in installing and running the tool. This would make the tool less portable and the out-of-the-box experience would suffer. Additionally, the curated changes to the list of archives would not make it to users' machines as not everyone is always aware of the changes in web archiving sphere.

For advanced users, it is almost always better to use their custom or local archives file and not rely on an external curated list of archives that might go down. Luckily, this service is hit only once on the startup of the tool then it caches the list of archives in the memory for the entire session. Additionally, a failure to read the curated list file results in the fatal error with precise message to explain what went wrong.

That said, do you have any other mechanism that might work better in this case, please feel free to propose.

[machawk1]

Not the best solution, but to mitigate the effect that git.io has on MemGator instances, would it be possible to consult a second or even tertiary source redundant of the information at git.io?

[ibnesayeed]

That is doable, but it would cause a sync overhead in which we will have to find a few distinct hosts where we can keep the copies of the curated list of archives and be able to update the content without changing the URI (e.g., Gist wont work here). The current source is part of the repository, hence it's easy for anyone from our team to update that, another hosting service might not be that easy for all of us to have write access to. The other thing that needs to be considered is to not try other sources if the --archives flag is explicitly set by the user, even if the custom value is the same as default.

[machawk1]

https://github.com/jteeuwen/go-bindata might help with this, see http://rachbelaid.com/embedding-assets-in-go-project/ . I think having the JSON data built into the binary at compile time is a good, safe, default instead of having many people's binaries relying on an online file that you can manipulate.

[ibnesayeed]

I have considered embedding the default list inside the code (we don't even need binary data embedding for that), but shipping default data has it's own implications which I described above.