[WIP] gc: change option to use single remove flags #3207

skshetry · 2020-01-21T14:47:37Z

Related to #2325.

❗ Have you followed the guidelines in the Contributing to DVC list?
📖 Check this box if this PR does not require documentation updates, or if it does and you have created a separate PR in dvc.org with such updates (or at least opened an issue about it in that repo). Please link below to your PR (or issue) in the dvc.org repo.
❌ Have you checked DeepSource, CodeClimate, and other sanity checks below? We consider their findings recommendatory and don't expect everything to be addressed. Please review them carefully and fix those that actually improve code or fix bugs.

Thank you for the contribution - we'll try to review it as soon as possible. 🙏

dvc/command/gc.py

dvc/repo/gc.py

dvc/command/gc.py

efiop · 2020-01-22T10:40:33Z

dvc/command/gc.py

-            all_commits=self.args.all_commits,
+            all_branches="branches" in self.args.remove,
+            all_tags="tags" in self.args.remove,
+            all_commits="commits" in self.args.remove,


Let's pass remove to the repo.gc and let it deal with it. We need to keep API and CLI symmetric here, no reason to change one and keep the other unchanged.

Agree about the symmetricity. But, I find it better to use args rather than a collection of filters, kinda like buttons which you can turn on or off to be more aggressive/effective.

I don't have strong opinions though.

@skshetry well, but why not? :) Think about them as being used in future to remove specific references. E.g. remove=mytag, which you will have to pass as remove=["mytag"] to API anyways, so why bother with additional flags for repo.gc, if you can repo.gc(remove=["commits", "tags", "branches"]) right away.

Yes, it makes sense from future, as it allows a lot of flexibility. I was just saying, all_commits=True, all_branches=True are better interface than remove={"commits", "branches"}, unless there are too many arguments to pass (not really flexible).

I was looking from user interface-side. I'll fix.

I see three downsides to this, on top of @Efiops point:

it's very confusing that all_branches in Repo.gc() is somewhat but not entirely reverse to all_branches in Repo.used_cache()/brancher()

three params instead of one combining them, I was looking into reducing the number of params in our functions and combining them is one of the ways

we try to keep Repo methods API close to cli

@efiop I don't like changing brancher. The logic in both brancher and .used_cache() should stay the same. There is no reason to translate cli/api to internal code.

@efiop additionally all that new logic you propose is more complex than existing one and does not solve anything that is required by this PR/issue.

@Suor Let's be constructive here. What do you suggest?

@efiop wrote that below, but maybe to make it more clear:

leave .brancher(), .used_cache() and any non-cli non Repo api-methods code unchanged.

accept new options in dvc gc, raise some DeprecatedError on the old ones

pass that to Repo.gc() as is

translate remove={...} to all_branches and friends that .brancher() and .used_cache() understand in Repo.gc() implementation.

This way only CmdGC and Repo.gc() would be changed.

accept new options in dvc gc, raise some DeprecatedError on the old ones

No need, we are breaking the default behavior anyway, preserving old options would be hard and confusing. We need to drop them right away, without any DeprecationError's.

I think you are missing something here. There is no direct translation of current options into the new ones, so your proposition is impossible to implement without changing some of that. For example, dvc gc --remove=branches needs to collect all used cache for all references except for all branches, so it would translate to something like git rev-list --all --skip $ALL_BRANCHES, which requires adjustments for used_cache and brancher.

dvc/repo/gc.py

dvc/command/gc.py

dvc/repo/gc.py

efiop · 2020-01-24T10:49:55Z

For the record: @skshetry left for a holiday, so I'm taking over here 🙂

efiop · 2020-01-26T19:47:03Z

dvc/scm/git/__init__.py

@@ -244,8 +244,18 @@ def list_branches(self):
    def list_tags(self):
        return [t.name for t in self.repo.tags]

-    def list_all_commits(self):
-        return [c.hexsha for c in self.repo.iter_commits("--all")]


This was a misuse of iter_commits, where passing --all to it worked accidentally because it was passing it as rev to rev-list. New implementation uses rev-list directly, which should be significantly faster (no benches yet).

Also, previous implementation was silently keeping refs/remotes and refs/stash as well, which might or might not be what we want, considering that we don't currently have any means of deleting cache for those.

So it was previously simply "keep anything, which is referenced from git". Now it's more complicated )

efiop · 2020-01-26T19:49:40Z

dvc/command/gc.py

-            all_branches=self.args.all_branches,
-            all_tags=self.args.all_tags,
-            all_commits=self.args.all_commits,
+            remove_all_tags=self.args.remove_all_tags,


Opted for explicit flags to avoid confusion such as previous --remove=tags,commits(where it is not clear what commits are and why you are not able to pass specific tag or branch to it) and also to simplify completion.

efiop · 2020-01-26T19:50:30Z

tests/func/test_gc.py

@@ -92,16 +92,26 @@ def test(self):

        self._check_cache(4)

-        self.dvc.gc(all_tags=True, all_branches=True)
+        self.dvc.gc()


Will look into converting to pytest and possibly adding new tests soon.

Suor · 2020-01-28T06:12:29Z

dvc/command/gc.py

+            else "and all git branches",
+            history=""
+            if self.args.remove_all_history
+            else "and their history",


This makes the phrase weird - many ands. Maybe just list things instead:

This will remove all cache not referenced from: - the working tree - all branch heads ...

? Will make it more glanceable, the changing items will be always in the same place, any user already familiar with gc, will be simply skipping the entering phrase without loose of any meaning.

Suor · 2020-01-28T06:19:18Z

dvc/command/gc.py

+        "--remove-all-history",
+        action="store_true",
+        default=False,
+        help="Remove cache for all history of all branches and tags.",


As far as I recall, your intention was to remove history by default. Why did you change your mind?

@Suor To keep it as safe as possible for now 🙂 And, well, I implemented a proper approach by excluding refs when not deleting full history.

Suor · 2020-01-28T06:27:24Z

dvc/repo/brancher.py

+        revs = scm.list_all_commits(
+            exclude_all_tags=not all_tags,
+            exclude_all_branches=not all_branches,
+        )


This makes .brancher and .used_cache() interfaces lying ones, all_commits no longer means all commits.

This also makes dvc gc --remove-all-branches --remove-all-tags approaching noop for many scenarios, but not quite noop, a confusing situation. Suppose you make things iteratively and last commits only about some polish like README adjustments.

Yeah, the naming brancher is at fault here. It should really have been something like iter_commits() instead.

But --remove-all-branches --remove-all-tags is not a noop, it will exclude tags and branches from the refs for which it iterates through. Hence why I've mentioned an issue with refs/remotes and refs/stash etc. We should either exclude those by default or add an option to get rid of those. For now this is a "safe" approach, that can be revisited iteratively.

So, it's even more confusing. You need a whole paragraph of text to explain how it works, but still it's usually just a noop.

@Suor Well, that is because of refs/remotes and refs/stash. Will need to handle them somehow. To me seems like a good idea to just drop them by default and only keep refs/heads and refs/tags. What do you think?

Suor · 2020-01-28T16:08:25Z

Looking at this I see it becoming more and more weird. I hate to crush the party, but we should consider closing it with the original ticket.

So far I see only one advantage to whatever we have now - it's safer for no upstream or dvc gc -r/-c case. I see numerous disadvantages though:

it's hard to explain/understand what does it really do
it's hard to use it - long options, that you need to use in most cases, options doesn't reflect whatever people intend to do in many cases
asymmetry with push/pull

What we should care instead probably:

not loosing data sources, everything else might be recalculated
not loosing things not pushed to upstream
maybe separating local/cloud gc better

EDIT. Those things, while making more sense to me, are also not that important probably. It feels that we are inventing the problem or just elevating its priority for no good reason. So the right approach might be saving the effort here.

shcheklein · 2020-01-29T01:39:36Z

Some background, the way I remember/see this. The whole things (interestingly enough) originates from the two discussions:

Being able to show metrics across different commits support push/pull/metrics/gc, etc across different commits #1691:

Can you consider supporting showing different metric values across different commits in the same branch?

(The discussion went into questions around symmetry, and I still believe symmetry is important if we can preserve it)

And the second one (and few other users reported this) - support push/pull/metrics/gc, etc across different commits #1691 (comment)

To be precise, they were concerned about the default behavior. But as I discussed with them providing an option to keep some N last commits should be enough. Though some folks were complaining about the default behavior besides ODS guys.

I agree that changing the default behavior creates a mess, but it also looks like it not exactly necessary to solve the problems pointed by the users. For example, we can do something like:

do not allow to run dvc gc at all, only with -f or any flag that explicitly expands the scope of files to keep (tags, commits, branches)
along with --all-commits introduce -n to keep some reasonable number of last commits. (and later add symmetrical option to all other commands which accept -T/-a - metrics, push, pull) - to finally get out of this rabbit hole and solve one of the actual problems.
minor but nice: add a shortcut for the -all-commits - -C or something?

This way it can be safe enough, no need to change names/behavior, we still reuse the all-commits logic. It's symmetrical with push/pull, etc.

Suor · 2020-01-29T03:40:09Z

@shcheklein my impression now is that people express their concerns with e.g. non-safe defaults but they don't really care about it. I mean they don't do any followups or even +1s on them, which means this is not bugging them in the long run. We are the only ones still pushing it.

I agree that those changes might be beneficial just not that necessary, thus I was talking about saving the effort.

efiop · 2020-01-29T12:45:05Z

Ok, so we are lacking a complete solution here, which was expected, as this change was only meant to make the first iteration on our path to good gc. Something safe that we could jump to and then chip away from it with new options. But now the discussion is again about the broader topic, which will again result in it just hanging out there for a few more months more.

Current gc suffers from the same issues solved in this new version, but has old cryptic options that won't allow us to do something like --all-commits --all-tags with the intention of only getting rid of branches. And in the new one, you could do that with --remove-all-branches, that will delete branches and their dangling history automatically(e.g. say you have mybranch that has mytag midway, then all commits from mytag up to mybranch HEAD will get deleted by that flag).

I know that with that example I'm making something out to apply to current option list, but the reason I do that is that in the next step you could easily apply that to potential --remove mybranch or --remove my.dvc, unlike the old one, where syntax is focused on what to keep and not what to remove. From discussions on ODS and Discord, people always ask how to remove something specific, and the new gc is allowing for that expansion of functionality in the future, unlike old gc. Plus you could always add --force flag there, if someone absolutely needs to get rid of cache for one specific dvc file and he really doesn't care if it is used somewhere else in the history - again, unlike with old gc.

That being said, it is clear that we also need to improve user awareness, which should be solved by some stats command/flag/behaviour, that would tell you how many data sources you have, how many branches, what is their unique space usage, how much you could free up if you delete cache for them, whether or not cache is already on the cloud and so on and so forth. That needs a separate discussion, but even now it seems pretty clear that whatever we come up with will be mostly focused on remove and not on keep aspect, as you will need to tell gc what you no longer need.

So I think we should bite the bullet, switch to the safest behaviour and start iterating on this with a list of options that we pretty much already agree on like preserving data sources, freeing up local cache if there is stuff on the cloud already, separating cloud and local gc better, introducing history depth option etc. If there will be errors, so be it, we could always improve or switch behaviour, at least we will get more experience and will know better what we want and what we need, which will fuel next iterations. Otherwise, we will be "saving the effort" indefinitely and won't move anywhere, not gaining any new feedback/info and so being stuck in the discussion forever flying in our own fantasies.

The history of the current gc ticket is a clear illustration that we are stuck, we need to stop talking the talk and finally walk the walk, to iterate and collect new feedback.

Suor · 2020-01-30T06:28:39Z

@efiop I don't agree that most people want to remove something specific. People usually want to simply free up some space, which dvc gc now does. If we switch to --remove-* flags then people will ask how they can remove everything but X, i.e. keep X, so that switch doesn't really solve anything but safety, which people don't really care about.

Bottom lines:

we are putting effort into this, while nobody cares that much.
current iteration is one step forward, two step backwards.

I understand that there is an invested effort/code/time here, I was pushing this for a long time too, but now I think it's not worth it, both overall and this particular PR.

EDIT. TLDR. We've spawned a monster.

efiop · 2020-01-30T15:05:06Z

@Suor I have never ever heard any user ask how to only keep something and delete the rest in all of these years, have you? Maybe @shcheklein or @dmpetrov have a different experience.

shcheklein · 2020-01-30T17:21:17Z

@efiop I think there is a natural bias - gc options are made right now with a "keep" philosophy in mind, so users asks "how to remove" something. If we change it - we will start getting questions "how to keep" instead.

The best way to answer here is to try to understand what actually users want in their workflow. Probably again, not one can come up with a 100% correct answer, unfortunately. I think both approaches would be valid in different cases.

My 2cs here - keep semantics is more user friendly if you can put it that way. I care about certain things from my domain (experiments, models, datasets and whatnot) to stay on my machine because I still iterate with them. And I don't care about anything else.

Remove feels more advanced - I need to understand now what exactly do I want to remove (either do a negation from what I want to keep) or recall what takes most of the space and go remove it.

I definitely recognize the lack of a "pinpoint" remove operation - few users have been asking - how to un-add something, for example. But it feels it should be solved separately?

efiop · 2020-01-31T01:03:21Z

@shcheklein How I imagine this. I'm a DS and I have created a new branch mybranch, that I've added some new things to, produced some intermediate files. Then I realise that I no longer need it and I would like to get rid of that data. How do I do that with the current "keep" terminology? Do I list all the other stuff that I have that I might need? But that would not be trivial, as no one keeps a full list of tags/branches in their mind. With the one that I suggest user would be able to git branch -D branch and dvc gc(no options), that will automatically get rid of the unused garbage, while not affecting the rest. Or it might potentially be dvc gc --remove mybranch in the future, that would eliminate the need for git branch -D. Again, I see no good way to implement that in the current keep approach.

People seem to rely on remotes as a backup and just push stuff as they go and nuke the whole local cache whenever it is taking too much space, as they don't understand current dvc gc. Yes, remotes are usually limitless compared to the local cache capasity, but the time will come that you will need to clean it up as well, at which point you will have to dive into dvc gc options and new dvc gc would be a good no-brainer no-bad-consequences command to run.

shcheklein · 2020-01-31T07:37:52Z

Again, I see no good way to implement that in the current keep approach.

how about dvc gc --all-commits in this case? it will keep all the stuff you have, except the removed branch? Granted, this operation gonna be slower.

as they don't understand current dvc gc

so, we change it, right? we provide all-commits, last N commits, branches, tags, etc. Next step would be to provide filters on tags and branches - --filter *-experiments - and again this is symmetrical and needed for all commands. The same as you need -n for metrics, pull, push, etc.

would love to better explorer the "they don't understand" part, for now I tend to agree that there are not that much evidence for this.

saying all that, I understand that there is not one-fits-all solution in this case - some might prefer keep, some remove semantics. In some case it would easier to write in one, in other cases in another. But, I'm not sure we have strong enough evidence that "remove" is superior to the extent to jump into breaking the symmetry, default behavior and redoing all options.

May be we should try to come with more examples to start feeling this better? (but again, it can be an argument to slightly change it for now, not changing the whole semantics of it?)

efiop · 2020-01-31T08:52:16Z

how about dvc gc --all-commits in this case? it will keep all the stuff you have, except the removed branch? Granted, this operation gonna be slower.

If you delete the branch prior to that - yes, but it is not a "no-brainer" command. And it doesn't extend to the aforementioned removal of particular dvc-file.

so, we change it, right? we provide all-commits, last N commits, branches, tags, etc. Next step would be to provide filters on tags and branches - --filter *-experiments - and again this is symmetrical and needed for all commands. The same as you need -n for metrics, pull, push, etc.

No, it doesn't have to be symmetrical with metrics/pull/push, really, as they have different purposes. dvc gc that assumes --all-commits is good, while dvc push/pull that assumes --all-commits is bad.

would love to better explorer the "they don't understand" part, for now I tend to agree that there are not that much evidence for this.

Wasn't it the previously constantly discussed argument about current dvc gc? That backwards logic of keep was not intuitive?

saying all that, I understand that there is not one-fits-all solution in this case - some might prefer keep, some remove semantics. In some case it would easier to write in one, in other cases in another. But, I'm not sure we have strong enough evidence that "remove" is superior to the extent to jump into breaking the symmetry, default behavior and redoing all options.

Well, it has been a couple of years already, and the feedback we've received was about gc being dangerous and people asking how to delete something in particular, at least in my experience. I think we need to iterate based on what we have and not freeze this ticket again for another couple of years.

May be we should try to come with more examples to start feeling this better? (but again, it can be an argument to slightly change it for now, not changing the whole semantics of it?)

Well, we could forbid dvc gc with no arguments as you've suggested, but then we will need to introduce a compatible option for the current behaviour, which will be dvc gc --workspace in current keep terminology or something like that. Is that really better to introduce another bump in the road for the user? I think dvc gc should be usable without arguments and provide a no-brainer low-consequence behaviour.

I'll try to prepare more examples, but looks like we are already out of focus here and this is turning back into a large discussion... Hope we will agree on some action points...

dmpetrov · 2020-01-31T19:21:52Z

Interesting discussion... and it is more than one year long 😄 A lot of good points for each of the options.

we are lacking a complete solution here, which was expected, as this change was only meant to make the first iteration

My major concern with GC - I have a strong feeling that we are still lacking a big picture and don't fully understand the scenarios. The only thing that we can say with confidence - the current GC does not solve users' problems.

Without the big pictures, it is quite dangerous to move forward with the current approach because it is quite opinionated and is based on the assumption that we can implement all possible scenarios (with probably -n X option which still looks like a workaround, not a solution).

From discussions on ODS and Discord, people always ask how to remove something specific

It might be more efficient and much safer to implement a very basic, low-level GC-option for removing specific cache items. I can imagine a command dvc gc data.csv (with an optional --rev abcd123) which removes data for a specific cache item with surgical precision.

With this specific GC-commands, users will be well-equipped for implementing their own GC-scenarios. So, we can identify common GC-scenarios and implement them later as high-level options like --remove-all-<something>.

I'd suggest reverting the current change and implementing the low-level GC-option.

dmpetrov · 2020-01-31T19:33:43Z

Also, I quite don’t understand why we care about a safe approach so much in the context of GC. The core propose of GC is removing data. It is not safe. I think it was a mistake (my mistake btw…. 😄) to call this command GC. dvc clean or dvc cache clean might be a better name for it. We can consider renaming - any ideas are welcome!

Today dvc gc is safer than it looks like. It asks the question all the time in prompt mode: “Are you sure you want to proceed? [y/n]”.

PS:
We might need more command in the future to manage the cache:

how many versions of this dataset do we keep in local cache? ... in data-remote?
how much space this file takes among the whole history?
how much data repo imports from outside?

So, introducing a dvc cache umbrella with dvc cache clean/gc option can be a good move.

Suor · 2020-02-03T12:02:23Z

Granted, this operation gonna be slower.

Not really it would be the same speed. You can't really delete whatever is referenced from some branch because anything referenced from there might also be referenced from somewhere else. So the implementation is always the same:

collect all the checksums, which should be kept
walk through cache, removing anything not in the set above

This is why a proper unadd is such a complicated operation.

That backwards logic of keep was not intuitive?

The issue always was that it is not safe, not that it's non-intuitive. At least this is how I understood that all the way.

The only thing that we can say with confidence - the current GC does not solve users' problems.

I doubt this. It looks like most users are ok with it, some people worry occasionally about non-safe defaults, but they don't follow up on their worries, which makes me think the issue is theoritical and we are the only ones really pushing it.

dmpetrov · 2020-02-04T00:51:19Z

It looks like different people have a different understanding and expectations of GC command. Some people (@efiop and I'm) trying to clean the cache and cloud/remote to solve user issues with space. While others (@Suor and @shcheklein) are trying to preserve more "keep" semantic which makes sense for GC name.

To my mind, the actual problem we need to focus on is cleaning space (local and cloud) and we should give up with the GC name. Let's introduce a new subcommand to dvc cache (today it has only dvc cache dir).

With this command, we can:

keep the current GC (or the safe version) - dvc cache gc
introduce pinpoint remove - dvc cache remove users.csv file.pkl
introduce "remove" strategies - dvc cache remove-all-branches etc
introduce "keep" strategies - dvc cache clean -n 10
*in the future we might need additional cache commands like history/log/du and others.

UPDATED cmd name:
For cleaning up space in remote an option dvc cache -c can be used or implement this in the remote command dvc remote gc/remove/clean.

efiop · 2020-02-04T13:28:36Z

@dmpetrov Thanks for the feedback! I really like your dvc cache gc/clean/remove(depending on which name(s) we decide to go with) idea. Let me give it a thought and come up with a proposal based on it. So far it is clear that current dvc gc is here to stay for now and might be obsoleted after dvc cache ... cmds introduction, so clearly no need to adjust it right now and this PR will be closed soon. I'll send a bugfix for --all-commits separately so it is not lost and could be quickly merged.

Suor · 2020-02-11T08:40:15Z

From my 1-1 with @efiop:

The remove specific branch or file commands won't care if same checksum is referenced anywhere else in the history, they will assume a user knows what he or she is doing.

efiop · 2020-02-21T11:17:51Z

Closing this fo

skshetry self-assigned this Jan 21, 2020

skshetry force-pushed the gc-change-defaults branch 4 times, most recently from 8679cd4 to c3b44f7 Compare January 21, 2020 19:36

efiop reviewed Jan 22, 2020

View reviewed changes

dvc/command/gc.py Outdated Show resolved Hide resolved

efiop reviewed Jan 22, 2020

View reviewed changes

dvc/command/gc.py Outdated Show resolved Hide resolved

efiop reviewed Jan 22, 2020

View reviewed changes

dvc/command/gc.py Outdated Show resolved Hide resolved

skshetry force-pushed the gc-change-defaults branch from c3b44f7 to 907263b Compare January 22, 2020 10:14

skshetry commented Jan 22, 2020

View reviewed changes

dvc/repo/gc.py Outdated Show resolved Hide resolved

skshetry commented Jan 22, 2020

View reviewed changes

dvc/command/gc.py Show resolved Hide resolved

efiop reviewed Jan 22, 2020

View reviewed changes

dvc/command/gc.py Outdated Show resolved Hide resolved

efiop reviewed Jan 22, 2020

View reviewed changes

skshetry mentioned this pull request Jan 22, 2020

brancher: do not get tree if there's no revs #3211

Merged

3 tasks

gc: change option to use single remove flags

2412016

skshetry force-pushed the gc-change-defaults branch from 907263b to 7708de1 Compare January 23, 2020 04:31

efiop reviewed Jan 23, 2020

View reviewed changes

dvc/repo/gc.py Outdated Show resolved Hide resolved

fixup! brancher: do not get tree if there's no revs

b9838b7

skshetry force-pushed the gc-change-defaults branch from 7708de1 to b9838b7 Compare January 23, 2020 07:36

Suor suggested changes Jan 23, 2020

View reviewed changes

dvc/command/gc.py Outdated Show resolved Hide resolved

dvc/repo/gc.py Outdated Show resolved Hide resolved

temp: try to exclude some hashes

13c3338

weekly-digest bot mentioned this pull request Jan 26, 2020

Weekly Digest (19 January, 2020 - 26 January, 2020) #3237

Closed

gc: rework implementation

3c07c42

efiop self-assigned this Jan 26, 2020

efiop reviewed Jan 26, 2020

View reviewed changes

Suor suggested changes Jan 28, 2020

View reviewed changes

weekly-digest bot mentioned this pull request Feb 2, 2020

Weekly Digest (26 January, 2020 - 2 February, 2020) #3272

Closed

weekly-digest bot mentioned this pull request Feb 9, 2020

Weekly Digest (2 February, 2020 - 9 February, 2020) #3289

Closed

weekly-digest bot mentioned this pull request Feb 16, 2020

Weekly Digest (9 February, 2020 - 16 February, 2020) #3342

Closed

efiop closed this Feb 21, 2020

skshetry deleted the gc-change-defaults branch May 26, 2020 05:04

[WIP] gc: change option to use single remove flags #3207

[WIP] gc: change option to use single remove flags #3207

Conversation

skshetry commented Jan 21, 2020

efiop Jan 22, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

efiop commented Jan 24, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Suor Jan 28, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Suor commented Jan 28, 2020 • edited Loading

shcheklein commented Jan 29, 2020

Suor commented Jan 29, 2020

efiop commented Jan 29, 2020

Suor commented Jan 30, 2020 • edited Loading

efiop commented Jan 30, 2020

shcheklein commented Jan 30, 2020

efiop commented Jan 31, 2020

shcheklein commented Jan 31, 2020

efiop commented Jan 31, 2020 • edited Loading

dmpetrov commented Jan 31, 2020 • edited Loading

dmpetrov commented Jan 31, 2020 • edited Loading

Suor commented Feb 3, 2020 • edited Loading

dmpetrov commented Feb 4, 2020 • edited Loading

efiop commented Feb 4, 2020

Suor commented Feb 11, 2020 • edited Loading

efiop commented Feb 21, 2020

efiop Jan 22, 2020 •

edited

Loading

Suor Jan 28, 2020 •

edited

Loading

Suor commented Jan 28, 2020 •

edited

Loading

Suor commented Jan 30, 2020 •

edited

Loading

efiop commented Jan 31, 2020 •

edited

Loading

dmpetrov commented Jan 31, 2020 •

edited

Loading

dmpetrov commented Jan 31, 2020 •

edited

Loading

Suor commented Feb 3, 2020 •

edited

Loading

dmpetrov commented Feb 4, 2020 •

edited

Loading

Suor commented Feb 11, 2020 •

edited

Loading