Fast inclusion operation on hashmaps and hashsets #282

svenkeidel · 2020-07-12T19:31:28Z

Dear Johan, Dear David,

First of all, thanks for putting all this hard work into this library. We use it heavily in Sturdy to implement static code analyses in Haskell. To this end, we need a fast subset operation on hashmaps and hashsets. Unfortunately, while a naive implementation is easy to understand, it is rather slow because it compares keys with different hashes:

subset :: (Eq k, Hashable k, Eq v) => HashMap k v -> HashMap k v -> Bool
subset m1 m2 = keys m1 ⊆ keys m2 && and [ v1 == v2 | (k1,v1) <- toList m1; let v2 = m2 ! k1 ]

Instead, I worked on an implementation that only compares keys with the same hash. I had to extend the lookupCont function with the number of the subkey to avoid code duplication. I tested this implementation with a few QuickCheck properties. What do you think?

I would be delighted if you would add this function to the library.

Regards,
Sven

treeowl · 2020-07-12T19:41:37Z

Thanks for all this work! Can you add an explanation of what you mean by a "subkey"?

treeowl · 2020-07-12T20:08:14Z

The not-fun part: how does this change affect benchmarks? Also also, do you have ideas for implementing #226? That would presumably subsume this.

sjakobi

Thanks!

I have a few comments regarding the consistency with the containers API.

Data/HashMap/Base.hs

Data/HashSet/Base.hs

svenkeidel · 2020-07-12T20:52:17Z

treeowl: how does this change affect benchmarks?

You mean the performance of lookupCont (the only existing function I changed) and functions that use lookupCont? I have not tested this yet. The generated code should be the same though, since lookupCont is inlined and its recursive helper function go already uses a subkey offset.

treeowl: do you have ideas for implementing #226? That would presumably subsume this.

You mean something like subset = mergeA (const False) (const True) (==)? I guess this would work. I don't know if this implementation would be as efficient as what I implemented right now. In particular, mergeA ... m1 m2 traverses over keys that are in m2 but not in m1. My subset implementation tries to avoid this. Also containers does not implement subset with merge (see submap')

Furthermore, I don't know how to implement this. I presume an implementation of merge is closer to union than to subset.

sjakobi: I have a few comments regarding the consistency with the containers API.

Thanks, I will change these function names.

svenkeidel · 2020-07-12T20:55:00Z

One thing that is missing is a proper benchmark for this operation. Any ideas for what would be a good benchmark?

svenkeidel · 2020-07-13T08:34:38Z

@treeowl, I compared the performance of subset2 vs master and calculated the speedup (slowdown): performance.xlsx

The speedup is within 0.97 and 1.04, which I would attribute to measuring inaccuracies.

svenkeidel · 2020-07-13T08:37:02Z

@sjakobi, I renamed the functions for compatibility with the containers API in https://github.com/svenkeidel/unordered-containers/commit/ffa499e7c8a087a44442992af9d266c412e962c0. Github did not show this commit here because of an outage.

sjakobi · 2020-07-13T10:01:47Z

@sjakobi, I renamed the functions for compatibility with the containers API in svenkeidel@ffa499e. Github did not show this commit here because of an outage.

Could you try pushing it again? The outage is supposed to be over: https://www.githubstatus.com/

sjakobi

The comments in isSubmapOfBy are very helpful! :)

tests/HashMapProperties.hs

Data/HashMap/Base.hs

Data/HashSet/Base.hs

sjakobi

Great progress!

Data/HashMap/Base.hs

treeowl · 2020-07-13T16:16:14Z

I think mathematical symbols are fine, but I like them presented as MathJax, not Unicode in the source.

svenkeidel · 2020-07-13T16:43:41Z

@sjakobi, @treeowl thanks for the feedback. Except for this last issue (#282 (comment)), I addressed all feedback.

sjakobi

A few more comments

I really like the examples BTW! :)

Data/HashSet/Base.hs

Data/HashMap/Base.hs

svenkeidel · 2020-07-16T06:57:56Z

Thanks again for your feedback. I addressed all further comments. Unfortunately, this is about all time I can and want to spend on this PR.

sjakobi

I'm quite happy with the code at this stage.

I'm curious how much faster this is than a naive implementation though. The complexity seems to be the same I think.

Could you give us an indication of that @svenkeidel?

I don't feel like requiring that proper benchmarks be included in this PR. I think that task could be left for someone who wants to make the code even faster than it is now.

sjakobi · 2020-07-16T13:01:24Z

Data/HashMap/Base.hs

+-- | /O(min n m))/ Checks if a bitmap indexed node is a submap of another.
+submapBitmapIndexed :: (HashMap k v1 -> HashMap k v2 -> Bool) -> Bitmap -> A.Array (HashMap k v1) -> Bitmap -> A.Array (HashMap k v2) -> Bool
+submapBitmapIndexed comp b1 ary1 b2 ary2 = subsetBitmaps && go 0 0 (b1Orb2 .&. negate b1Orb2)
+  where
+    go :: Int -> Int -> Bitmap -> Bool
+    go i j m
+      | m > b1Orb2 = True
+
+      -- In case a key is both in ary1 and ary2, check ary1[i] <= ary2[j] and
+      -- increment the indices i and j.
+      | b1Andb2 .&. m /= 0 = comp (A.index ary1 i) (A.index ary2 j) &&
+                             go (i+1) (j+1) (m `unsafeShiftL` 1)
+
+      -- In case a key occurs in ary1, but not ary2, only increment index j.
+      | b2 .&. m /= 0 = go i (j+1) (m `unsafeShiftL` 1)
+
+      -- In case a key neither occurs in ary1 nor ary2, continue.
+      | otherwise = go i j (m `unsafeShiftL` 1)
+
+    b1Andb2 = b1 .&. b2
+    b1Orb2  = b1 .|. b2
+    subsetBitmaps = b1Orb2 == b2


This bit goes way over my head. Can I rely on you, @treeowl, to review this?

If it helps, I copied some of this code from unionArrayBy:

unordered-containers/Data/HashMap/Base.hs

Lines 1642 to 1657 in 7e8fc9e

let ba = b1 .&. b2

go !i !i1 !i2 !m

| m > b' = return ()

| b' .&. m == 0 = go i i1 i2 (m `unsafeShiftL` 1)

| ba .&. m /= 0 = do

x1 <- A.indexM ary1 i1

x2 <- A.indexM ary2 i2

A.write mary i $! f x1 x2

go (i+1) (i1+1) (i2+1) (m `unsafeShiftL` 1)

| b1 .&. m /= 0 = do

A.write mary i =<< A.indexM ary1 i1

go (i+1) (i1+1) (i2 ) (m `unsafeShiftL` 1)

| otherwise = do

A.write mary i =<< A.indexM ary2 i2

go (i+1) (i1 ) (i2+1) (m `unsafeShiftL` 1)

go 0 0 0 (b' .&. negate b') -- XXX: b' must be non-zero

This bit goes way over my head. Can I rely on you, @treeowl, to review this?

Mine eyes glazeth over when it comes to bit fiddling. I can try. Mostly, I rely on testing.

sjakobi · 2020-07-16T13:05:36Z

Data/HashMap/Base.hs

+-- | /O(n*m)/ Check if the first array is a subset of the second array.
+subsetArray :: Eq k => (v1 -> v2 -> Bool) -> A.Array (Leaf k v1) -> A.Array (Leaf k v2) -> Bool
+subsetArray cmpV ary1 ary2 = A.length ary1 <= A.length ary2 && A.all inAry2 ary1
+  where
+    inAry2 (L k1 v1) = lookupInArrayCont (\_ -> False) (\v2 _ -> cmpV v1 v2) k1 ary2
+    {-# INLINE inAry2 #-}


@svenkeidel, @treeowl What did you think about my idea to record the indices of any found elements in ary2 and to skip these when checking the remaining elements of ary1?

I'm mostly wondering whether that's an idea worth recording – I wouldn't insist that it be included in this PR.

It is hard to say if this is going to make a big difference performance wise. For each key you still would need to iterate over ary2 anyways. The only thing you would save are a few equality checks of keys. It could be worth working on this optimization, but before we need some benchmarks to verify that we actually improve the performance.

I wouldn't record indices. But we could thaw one of the arrays and play mutation games. Say we have

1 2 3 4

and we search for 3. Then we can mutate the array to

undefined 2 1 4

That is, swap the 3 to the front and replace it with undefined. Then start the next scan in the second slot.

@treeowl That's a neat idea! :)

If we don't do this in this PR, I'll record it in an issue.

svenkeidel · 2020-07-16T19:40:28Z

@sjakobi, about benchmarks. My collaborator Tobias Hombücher is currently working on a static analysis for Scheme that uses this hashmaps and hashsets extensively (https://gitlab.rlp.net/plmz/sturdy/-/tree/PowersetT/scheme). We have a benchmark suit with a few scheme programs (https://gitlab.rlp.net/plmz/sturdy/-/blob/PowersetT/scheme/bench/FixpointBench.hs). Once Tobias gets the analysis working, this would be a great real-world benchmark for the unordered-containers library, because it uses the insert, delete, union and isSubmapOf functions.

That being said, a micro benchmark for the isSubmapOf operation would still be valuable and I still want to implement it.

svenkeidel · 2020-07-16T20:53:35Z

I added a few benchmarks. There is still some room for improvement.

benchmarked Map/isSubmapOf/String
time                 153.7 μs   (152.7 μs .. 155.5 μs)
                     0.999 R²   (0.999 R² .. 1.000 R²)
mean                 154.0 μs   (153.6 μs .. 154.7 μs)
std dev              1.609 μs   (953.4 ns .. 2.458 μs)

benchmarked Map/isSubmapOf/ByteString
time                 80.00 μs   (78.90 μs .. 81.17 μs)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 79.74 μs   (79.42 μs .. 80.96 μs)
std dev              1.673 μs   (632.2 ns .. 3.537 μs)

benchmarked hashmap/Map/isSubmapOf/String
time                 135.3 μs   (134.5 μs .. 136.2 μs)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 136.4 μs   (135.9 μs .. 137.5 μs)
std dev              2.301 μs   (1.123 μs .. 3.720 μs)

benchmarked hashmap/Map/isSubmapOf/ByteString
time                 78.98 μs   (78.52 μs .. 79.67 μs)
                     1.000 R²   (0.999 R² .. 1.000 R²)
mean                 78.60 μs   (78.42 μs .. 78.92 μs)
std dev              703.4 ns   (428.3 ns .. 1.199 μs)

benchmarked IntMap/isSubmapOf
time                 128.5 ns   (126.6 ns .. 131.1 ns)
                     0.997 R²   (0.994 R² .. 1.000 R²)
mean                 128.8 ns   (128.1 ns .. 130.1 ns)
std dev              2.688 ns   (1.236 ns .. 4.315 ns)

benchmarked HashMap/isSubmapOf/String
time                 199.8 μs   (198.7 μs .. 200.7 μs)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 202.5 μs   (201.5 μs .. 204.4 μs)
std dev              4.169 μs   (2.111 μs .. 6.863 μs)

benchmarked HashMap/isSubmapOf/ByteString
time                 101.0 μs   (100.0 μs .. 102.1 μs)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 100.5 μs   (100.3 μs .. 101.2 μs)
std dev              1.285 μs   (683.3 ns .. 2.341 μs)

benchmarked HashMap/isSubmapOf/Int
time                 83.80 ns   (80.81 ns .. 86.37 ns)
                     0.963 R²   (0.884 R² .. 0.999 R²)
mean                 85.79 ns   (84.39 ns .. 90.96 ns)
std dev              6.445 ns   (1.616 ns .. 13.01 ns)
variance introduced by outliers: 36% (moderately inflated)

benchmarked HashMap/isSubmapOfNaive/String
time                 218.3 μs   (214.8 μs .. 222.4 μs)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 221.3 μs   (220.1 μs .. 223.4 μs)
std dev              5.203 μs   (3.151 μs .. 8.042 μs)

benchmarked HashMap/isSubmapOfNaive/ByteString
time                 104.8 μs   (103.6 μs .. 105.8 μs)
                     0.992 R²   (0.970 R² .. 1.000 R²)
mean                 106.6 μs   (105.6 μs .. 110.1 μs)
std dev              5.373 μs   (967.1 ns .. 11.16 μs)
variance introduced by outliers: 28% (moderately inflated)

benchmarked HashMap/isSubmapOfNaive/Int
time                 84.29 ns   (82.98 ns .. 85.34 ns)
                     0.999 R²   (0.998 R² .. 0.999 R²)
mean                 83.93 ns   (83.48 ns .. 84.41 ns)
std dev              1.231 ns   (941.8 ps .. 1.615 ns)

treeowl · 2020-07-16T20:57:19Z

Data/HashMap/Base.hs

+-- False
+isSubmapOf :: (Eq k, Hashable k, Eq v) => HashMap k v -> HashMap k v -> Bool
+isSubmapOf = isSubmapOfBy (==)
+{-# INLINE isSubmapOf #-}


Shouldn't this be INLINABLE?

Whats wrong with and INLINE pragma on isSubmapOf? The function is nice and small.

We never need this to be inlined at a call site. But it is very valuable to specialize to particular key and value types. Once those types are known, I believe we want GHC to create a specialization that inlines the comparison function into isSubmapOfBy.

treeowl · 2020-07-16T20:57:46Z

Data/HashMap/Base.hs

+    -- Collision and Full nodes always contain at least two entries. Hence it
+    -- cannot be a map of a leaf.
+    go _ (Collision {}) (Leaf {}) = False
+    go _ (Full {}) (Leaf {}) = False


Should this be INLINE?

Isn't isSubmapOfBy a bit large for INLINE? This would create a lot of code bloat for every call of isSubmapOfBy. Why not INLINABLE? Then users can still specialize it within their code base.

I'm not sure. We want to be sure to inline it into isSubmapOf; does that happen if this is just marked INLINABLE (and that is too)?

We can collapse these three Leaf cases into one and put it up above, matching the approach we take to Empty:

go s (Leaf h1 (L k1 v1)) t2 = lookupCont (\_ -> False) (\v2 _ -> comp v1 v2) h1 k1 s t2 go _ _ (Leaf {}) = False

This makes no difference in the compiled code, but the source is shorter.

I'm not sure. We want to be sure to inline it into isSubmapOf; does that happen if this is just marked INLINABLE (and that is too)?

Another option is to mark both functions INLINABLE and then inline the isSubmapOfBy function at the isSubmapOf call site with the inline function.

Data/HashMap/Base.hs

treeowl · 2020-07-16T21:32:45Z

General question: would it be useful to implement a lookup-by-hash function that returns the Leaf or Collision node? That should help with Collision cases here, and would likely also be useful elsewhere.

Data/HashMap/Base.hs

svenkeidel · 2020-07-17T10:08:12Z

General question: would it be useful to implement a lookup-by-hash function that returns the Leaf or Collision node? That should help with Collision cases here, and would likely also be useful elsewhere.

Yes, this function would allow us to remove a bit of code duplication. Furthermore, we can probably implement the regular lookup function with a lookup-by-hash function.

treeowl · 2020-07-17T14:41:02Z

How do the benchmarks look with the simplifications in place?

svenkeidel · 2020-07-17T20:43:29Z

The simplifications only affect negative cases where isSubmapOf m1 m2 == False. Therefore the performance of the benchmarks did not change, since they are a positive test.

Data/HashMap/Base.hs

svenkeidel · 2020-07-18T17:05:09Z

Something is fishy with the benchmarks. When I run only a single benchmark with stack bench --ba='--match=prefix HashMap/isSubmapOf/String', I get

benchmarked HashMap/isSubmapOf/String
time                 167.6 μs   (166.8 μs .. 169.1 μs)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 168.4 μs   (167.9 μs .. 169.4 μs)
std dev              2.105 μs   (1.232 μs .. 4.253 μs)

When I run multiple benchmarks with stack bench --ba='--match=pattern isSubmapOf', I get

benchmarked HashMap/isSubmapOf/String
time                 205.8 μs   (204.8 μs .. 207.4 μs)
                     0.999 R²   (0.998 R² .. 1.000 R²)
mean                 206.6 μs   (206.0 μs .. 207.8 μs)
std dev              2.561 μs   (1.487 μs .. 4.375 μs)

Either garbage collection kicks in, or there is something wrong with the benchmark environment Env. Eitherway, it should not make a difference when you run a benchmark alone or in a group.

treeowl · 2020-07-18T17:09:08Z

Definitely an issue. Could caching play a role too? I don't know enough about these things.

sjakobi · 2020-07-18T17:16:16Z

@svenkeidel You might be running into haskell/criterion#166?!

You can try the --measure-with option: vincenthz/hs-gauge#3

svenkeidel · 2020-07-18T17:26:27Z

@sjakobi, --measure-with fixed it. Thanks!

Nonetheless, I think it is a bad idea to share a single benchmark environment across all benchmarks. It is reallocated for every benchmark group. We should split the environment into smaller environments for each benchmark group to reduce memory pressure.

svenkeidel · 2020-07-22T14:45:13Z

Ok, I rebased everything on master, removed the changes to the benchmark suite and squashed everything into one commit on this branch: https://github.com/svenkeidel/unordered-containers/tree/subset-squashed

sjakobi · 2020-07-22T14:53:27Z

@svenkeidel Is it OK if we finish the review in this PR, so the commit to master is associated with the discussion, or do you want to open a separate PR from that branch?

svenkeidel · 2020-07-22T15:11:21Z

The former makes most sense to me.

treeowl · 2020-07-22T15:28:01Z

I haven't reviewed them specifically, but rewrite rules in the benchmark suite make me very nervous in principle, because that takes the suite further from realistic code.

svenkeidel · 2020-07-22T15:30:01Z

@treeowl, don't worry about them, I removed them from the PR.

sjakobi

I have some questions about the INLIN[ABL]E pragmas – mostly because I'm somewhat confused about where to prefer each. I hope @treeowl can give some principled advice.

How do the new definitions stack up against the naive ones in the benchmark now, BTW?

Data/HashMap/Internal.hs

sjakobi · 2020-07-22T15:16:10Z

Data/HashSet/Internal.hs

+-- >>> fromList [1,2] `isSubsetOf` fromList [1,3]
+-- False
+isSubsetOf :: (Eq a, Hashable a) => HashSet a -> HashSet a -> Bool
+isSubsetOf s1 s2 = H.isSubmapOfBy (\_ _ -> True) (asMap s1) (asMap s2)


Suggested change

isSubsetOf s1 s2 = H.isSubmapOfBy (\_ _ -> True) (asMap s1) (asMap s2)

isSubsetOf s1 s2 = H.isSubmapOfBy (\_ _ -> True) (asMap s1) (asMap s2)

{-# INLINABLE isSubsetOf #-}

If INLINABLE helps with specialization, it would be good to use here, right?

The best thing to do is always to write some sample code and pore over the Core. But that's not a hard requirement when adding new functions.

Data/HashMap/Internal.hs

sjakobi · 2020-07-22T15:47:40Z

Oh, if my questions about the pragmas cannot be answered with the current state of the benchmark suite, then never mind.

It's fine for me if this is addressed in follow-up patches.

sjakobi

LGTM! Tweaks can happen later.

(I'd suggest a squashed commit here.)

sjakobi · 2020-07-29T09:00:39Z

@treeowl Are you done with reviewing this PR?

treeowl · 2020-07-29T09:05:36Z

I want to make sure someone's verified that this doesn't hurt lookup or alterF performance.

sjakobi · 2020-07-29T09:42:54Z

I want to make sure someone's verified that this doesn't hurt lookup or alterF performance.

@svenkeidel What did your benchmark spreadsheet say about this? I unfortunately can't find it in this thread anymore.

svenkeidel · 2020-07-30T06:21:27Z

#282 (comment)

treeowl · 2020-07-30T06:28:49Z

Please squash to one or two logical commits.

svenkeidel · 2020-07-30T06:39:58Z

@treeowl, see #282 (comment)

treeowl · 2020-07-30T06:55:31Z

I'll let you and @sjakobi work out the merge mechanics.

svenkeidel · 2020-07-30T12:58:12Z

@treeowl, @sjakobi, thanks for all the reviewing work and merging this PR 👍

sjakobi · 2020-07-30T13:09:53Z

You're welcome, @svenkeidel!

And thanks for the feedback on the test and benchmark setup! :) Any further help with that would be very much appreciated.

Would you mind if I'd prepare a release now, or do you see anything that needs to be done before that?

svenkeidel · 2020-07-30T13:29:19Z

Would you mind if I'd prepare a release now, or do you see anything that needs to be done before that?

Not at all. Sounds good 👍

sjakobi reviewed Jul 12, 2020

View reviewed changes

Data/HashMap/Base.hs Outdated Show resolved Hide resolved

Data/HashMap/Base.hs Outdated Show resolved Hide resolved

Data/HashSet/Base.hs Outdated Show resolved Hide resolved

svenkeidel changed the title ~~Fast subset operation on hashmaps and hashsets~~ Fast inclusion operation on hashmaps and hashsets Jul 13, 2020

sjakobi mentioned this pull request Jul 13, 2020

Rename internal modules and expose them #283

Merged

sjakobi reviewed Jul 13, 2020

View reviewed changes

Data/HashMap/Base.hs Outdated Show resolved Hide resolved

Data/HashMap/Base.hs Outdated Show resolved Hide resolved

Data/HashMap/Base.hs Outdated Show resolved Hide resolved

sjakobi reviewed Jul 13, 2020

View reviewed changes

Data/HashSet/Base.hs Outdated Show resolved Hide resolved

Data/HashMap/Base.hs Outdated Show resolved Hide resolved

Data/HashMap/Base.hs Outdated Show resolved Hide resolved

Data/HashMap/Base.hs Outdated Show resolved Hide resolved

sjakobi reviewed Jul 16, 2020

View reviewed changes

treeowl reviewed Jul 16, 2020

View reviewed changes

svenkeidel commented Jul 17, 2020

View reviewed changes

Data/HashMap/Base.hs Outdated Show resolved Hide resolved

treeowl reviewed Jul 17, 2020

View reviewed changes

Data/HashMap/Base.hs Outdated Show resolved Hide resolved

Data/HashMap/Base.hs Outdated Show resolved Hide resolved

Data/HashMap/Base.hs Outdated Show resolved Hide resolved

svenkeidel added 6 commits July 22, 2020 16:35

delete unused property

3ca2f77

fix error in isSubmapOf based on wrong assumption

dd079a3

add benchmarks

012188b

change a few recursive isSubmap cases to False

7e2de75

add strictness annotations

2d79571

make isSubmapOf and isSubmapOfBy INLINABLE

e4de203

sjakobi reviewed Jul 22, 2020

View reviewed changes

sjakobi approved these changes Jul 24, 2020

View reviewed changes

sjakobi merged commit 352591a into haskell-unordered-containers:master Jul 30, 2020

sjakobi mentioned this pull request Jul 30, 2020

Optimization idea for subsetArray #291

Open

sjakobi mentioned this pull request Jul 30, 2020

Real-world benchmarks (sturdy) #292

Open

This was referenced Jul 30, 2020

Splitting the benchmark Env #293

Open

Add more detailed data structure information #161

Open

sjakobi mentioned this pull request May 29, 2021

Add isSubsetOf #127

Closed

	let ba = b1 .&. b2
	go !i !i1 !i2 !m
	\| m > b' = return ()
	\| b' .&. m == 0 = go i i1 i2 (m `unsafeShiftL` 1)
	\| ba .&. m /= 0 = do
	x1 <- A.indexM ary1 i1
	x2 <- A.indexM ary2 i2
	A.write mary i $! f x1 x2
	go (i+1) (i1+1) (i2+1) (m `unsafeShiftL` 1)
	\| b1 .&. m /= 0 = do
	A.write mary i =<< A.indexM ary1 i1
	go (i+1) (i1+1) (i2 ) (m `unsafeShiftL` 1)
	\| otherwise = do
	A.write mary i =<< A.indexM ary2 i2
	go (i+1) (i1 ) (i2+1) (m `unsafeShiftL` 1)
	go 0 0 0 (b' .&. negate b') -- XXX: b' must be non-zero

	isSubsetOf s1 s2 = H.isSubmapOfBy (\_ _ -> True) (asMap s1) (asMap s2)
	isSubsetOf s1 s2 = H.isSubmapOfBy (\_ _ -> True) (asMap s1) (asMap s2)
	{-# INLINABLE isSubsetOf #-}

Fast inclusion operation on hashmaps and hashsets #282

Fast inclusion operation on hashmaps and hashsets #282

Conversation

svenkeidel commented Jul 12, 2020

treeowl commented Jul 12, 2020

treeowl commented Jul 12, 2020

sjakobi left a comment

Choose a reason for hiding this comment

svenkeidel commented Jul 12, 2020 • edited Loading

svenkeidel commented Jul 12, 2020

svenkeidel commented Jul 13, 2020

svenkeidel commented Jul 13, 2020

sjakobi commented Jul 13, 2020

sjakobi left a comment

Choose a reason for hiding this comment

sjakobi left a comment

Choose a reason for hiding this comment

treeowl commented Jul 13, 2020

svenkeidel commented Jul 13, 2020 • edited Loading

sjakobi left a comment

Choose a reason for hiding this comment

svenkeidel commented Jul 16, 2020 • edited Loading

sjakobi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

svenkeidel Jul 16, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjakobi Jul 16, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

svenkeidel commented Jul 16, 2020 • edited Loading

svenkeidel commented Jul 16, 2020

Choose a reason for hiding this comment

svenkeidel Jul 17, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

treeowl Jul 18, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

treeowl commented Jul 16, 2020

svenkeidel commented Jul 17, 2020

treeowl commented Jul 17, 2020

svenkeidel commented Jul 17, 2020 • edited Loading

svenkeidel commented Jul 18, 2020

treeowl commented Jul 18, 2020

sjakobi commented Jul 18, 2020

svenkeidel commented Jul 18, 2020 • edited Loading

svenkeidel commented Jul 22, 2020

sjakobi commented Jul 22, 2020

svenkeidel commented Jul 22, 2020

treeowl commented Jul 22, 2020

svenkeidel commented Jul 22, 2020

sjakobi left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sjakobi commented Jul 22, 2020

sjakobi left a comment

Choose a reason for hiding this comment

sjakobi commented Jul 29, 2020

treeowl commented Jul 29, 2020

sjakobi commented Jul 29, 2020

svenkeidel commented Jul 30, 2020

treeowl commented Jul 30, 2020

svenkeidel commented Jul 30, 2020

treeowl commented Jul 30, 2020

svenkeidel commented Jul 30, 2020

sjakobi commented Jul 30, 2020

svenkeidel commented Jul 30, 2020

svenkeidel commented Jul 12, 2020 •

edited

Loading

svenkeidel commented Jul 13, 2020 •

edited

Loading

svenkeidel commented Jul 16, 2020 •

edited

Loading

svenkeidel Jul 16, 2020 •

edited

Loading

sjakobi Jul 16, 2020 •

edited

Loading

svenkeidel commented Jul 16, 2020 •

edited

Loading

svenkeidel Jul 17, 2020 •

edited

Loading

treeowl Jul 18, 2020 •

edited

Loading

svenkeidel commented Jul 17, 2020 •

edited

Loading

svenkeidel commented Jul 18, 2020 •

edited

Loading