[WIP] Support run GHC's test from hadrian. #495

sighingnow · 2018-01-17T11:52:40Z

Necessary command line arguments to run test driver.
- --test-only=<TESTS>
- --skip-perf
- --summary=<TEST_SUMMARY>
- --junit=<TEST_SUMMARY_JUNIT>
- --config=<EXTRA_TEST_CONFIG>
Synchronize configurations from test.mk.
Synchronize GHC's compilation flags from test.mk (that's very important).

WIP

I'm not sure if it's fine to add these command line arguments for running test.
Document about how to use hadrian to run one or more test cases.

1. Necessary command line arguments to run test driver. + `--test-only=<TEST_CASE>` + `--test-skip-perf` + `--test-summary=<SUMMARY_FILE>` + `--test-junit=<SUMMARY_FILE>` + `--test-config=<EXTRA_TEST_CONFIG>` 2. Synchronize configurations from test.mk. 3. Synchronize GHC's compilation flags from test.mk (that's very important). Signed-off-by: HE, Tao <[email protected]>

sighingnow · 2018-01-17T12:01:45Z

Another question is that I have noticed that currently hadrian just call sytem's make to run the validate target. Should we move to make to run test target or call the runtest.py script from hadrian directly ?

snowleopard · 2018-01-17T18:25:04Z

@sighingnow Many thanks for this. To answer your questions:

I'm not sure if it's fine to add these command line arguments for running test.

I think that's fine. Otherwise there is really no way to conveniently use the built-in test rule, because one often wants to test just a single test case. However, I think we don't need the test- prefixes, because that would lead to a bit of duplication in command lines, e.g. hadrian test --test-only=123. Let's try to keep the flags exactly as they are currently in the driver script, e.g. hadrian test --only=123. This also has the benefit that GHC developers do not need to learn any new flags.

Another question is that I have noticed that currently hadrian just call sytem's make to run the validate target. Should we move to make to run test target or call the runtest.py script from hadrian directly ?

At the moment we run validate via make because that's the simplest thing to do. However, in future I think it would be best to get rid of make completely, and possibly even rewrite the runtest logic in Hadrian directly. Does this answer your question?

snowleopard · 2018-01-17T18:30:56Z

src/Hadrian/Utilities.hs

@@ -453,3 +460,21 @@ renderUnicorn ls =
    ponyPadding = "                                            "
    boxLines :: [String]
    boxLines = ["", "", ""] ++ (lines . renderBox $ ls)
+
+-- | These arguments are used by the `test` target.
+data TestArgs = TestArgs


I'm not sure why TestArgs is defined here, let's move it to src/CommandLine.hs.

Everything in src/Hadrian is supposed to be generally useful for build systems, i.e. not GHC-specific, whereas flags passed to the GHC's testsuite are very much GHC-specific.

snowleopard · 2018-01-17T18:35:16Z

src/Settings.hs

@@ -66,3 +66,7 @@ findPackageByName name = find (\pkg -> pkgName pkg == name) knownPackages
 -- | Install's DESTDIR setting.
 getDestDir :: Action FilePath
 getDestDir = fromMaybe "" <$> cmdInstallDestDir
+
+-- | Arguments to run GHC's test script.
+getTestArgs :: Action TestArgs


Not clear why need another name for the same function. I'd suggest to drop it.

snowleopard · 2018-01-17T18:36:07Z

src/Rules/Test.hs

+        let timeout_prog = "testsuite/timeout/install-inplace/bin/timeout"
+
+        quietly . cmd "python3" $
+            [ "testsuite/driver/runtests.py" ] ++


We probably should be constructing this command line using the abstractions available in Hadrian. For example:

https://github.com/snowleopard/hadrian/blob/master/src/Settings/Builders/Haddock.hs

sighingnow · 2018-01-18T01:36:31Z

Thanks for your reply and review. I will fix these problem this weekends. I'll work on the validate rule after finishing the test rule.

alpmestan · 2018-01-18T09:47:36Z

I recently got started tackling the very same thing, but you went through much more trouble to integrate this properly with command-line flags etc. Can this run .T tests properly too, or more generally can we just run all tests and get the same result that we get with the make build system? Some tests have Makefiles that rely on some env vars being set etc.

I got started looking into this in the context of #445 but if you do most of the work, that would be great and I'll then just have to tweak a couple of things to make it work with that branch. So... thanks! :)

simonmar · 2018-01-18T09:48:05Z

@alpmestan aren't you also working on this? cc @bgamari

alpmestan · 2018-01-18T09:51:33Z

@simonmar I am indeed, I left a comment above to ask a question or two. This patch is most likely saving me quite some work.

snowleopard · 2018-01-18T09:57:42Z

@alpmestan Ah, hopefully there wasn't too much work duplication. Note that this PR grew out of solving a bug in the test rule: #494. Perhaps, we should create a more general issue to coordinate the work on test build rules in Hadrian.

alpmestan · 2018-01-18T10:16:19Z

@snowleopard Well if this PR gets us to a point where we can run tests with the same results that we get with the make build system, then I don't think there's much more to do for now, hence my questions above :)

I haven't done a lot of duplicate work, but I did some. It's not all lost though, as this allowed me to look at and understand the testing infra of GHC as well as more code in Hadrian.

sighingnow · 2018-01-18T10:20:01Z

@alpmestan Currently I just have taken flags from test.mk to the test rule to execute the python script.

Make hadrian's test and validate rules fully support GHC's testsuite needs working with GHC team to translate the Makefile script in some tests to hadrian/shake's rule. I have run hadrian validate and many testcases failed to run their Makefile scripts. test summary log

I'm willing to take the job to integrate hadrian to GHC's test and validate.

alpmestan · 2018-01-18T10:26:43Z

src/Rules/Test.hs

+            , "-e", "config.in_tree_compiler=True"          -- Use default value, see https://github.com/ghc/ghc/blob/master/testsuite/mk/boilerplate.mk
+
+            , "--config-file=testsuite/config/ghc"
+            , "--config", "compiler="     ++ show (top -/- compiler)


These two lines have been a problem for me, as the config file seems to take priority and it also defines a config.compiler field. @sighingnow Can you confirm this is not the case for you and the correct GHC is picked?

I have confirm that this line let me execute the right ghc (inplace/bin/ghc-stage2) to compile the test.

Without this config, the ghc in $PATH will be called.

OK, somehow the --config options were not working for me, glad you got that to work!

alpmestan · 2018-01-18T10:28:24Z

src/Rules/Test.hs

-            -- , "--config", "gs=$(call quote_path,$(GS))"
-            -- , "--config", "timeout_prog=$(call quote_path,$(TIMEOUT_PROGRAM))"
+        -- See: https://github.com/ghc/ghc/blob/master/testsuite/mk/test.mk#L291
+        let timeout_prog = "testsuite/timeout/install-inplace/bin/timeout"


Does this therefore rely on having the timeout prog make rules executed before running hadrian's test rule? I do have a rule or two to handle the timeout program in my branch. I don't properly implement the Windows logic yet, I always pick the python script, but that's a start.

If the timeout_prog is not set, I get a [Error 13] Permission denied in python script (inside the runCmd method).

I'm working with Bash on Windows, it's a Ubuntu 16.04 environment.

We have the following code in testlib.py:

r = subprocess.Popen([timeout_prog, timeout, cmd], stdin=stdin_file, stdout=subprocess.PIPE, stderr=hStdErr, env=ghc_env)

Right, so this indeed requires that you build it through the make build system first. I have a bit of code that works on Linux for wrapping the python version of the program in a shell script and using that for the tests. It's a bit ugly but it does the trick. You'd need to replicate the Windows logic to make this work on your Windows machine, but as you can see here it's not scary enough to prevent us from doing it all in hadrian, I think. Here's the code that I have for the non-Windows case (that is, the part that replicates this bit of the Makefile).

-- FIXME: add ".exe" extension on windows timeoutProgPath :: FilePath timeoutProgPath = "test" -/- "bin" -/- "timeout" timeoutPyPath :: FilePath timeoutPyPath = "test" -/- "bin" -/- "timeout.py" -- and then in `testRules`... root <- buildRootRules root -/- timeoutPyPath ~> do -- FIXME: run only when not on Windows copyFile "testsuite/timeout/timeout.py" (root -/- timeoutPyPath) root -/- timeoutProgPath ~> do need [ root -/- timeoutPyPath ] let script = unlines [ "#!/usr/bin/env sh" , "exec python3 $0.py \"$@\"" ] liftIO $ do writeFile (root -/- timeoutProgPath) script cmd "chmod" [ "+x", root -/- timeoutProgPath ]

If we can get ride of current python's test driver, we could do the timeout directly in Haskell.

I'm afraid the python test driver is not going anytime soon, and we might want to aim for a more reasonable goal of just replicating that tiny Makefile's logic in hadrian, so that we can at least properly build/wrap the appropriate timeout program but also need it in any rule that will run tests. Really just replicating what's done in the makefile I linked to but in hadrian, so that we don't introduce any (implicit) dependency on them make build system for running the test rules.

alpmestan · 2018-01-18T10:39:20Z

@sighingnow I see! So we are both facing the same problem. One idea that I had yesterday to handle this problem without touching GHC's codebase at all might do the trick but requires going through a few hacky hoops.

What if we copied over the makefiles for all the tests that have one, along with the files needed for the test, and put them somewhere under the build root while following the same hierarchy. Except that we could edit the makefiles on the fly. As far as I can tell, many (most? all?) makefiles define TOP, include boilerplate.mk and test.mk and then define rules for the tests, which end up running simple commands like $(TEST_HC) $(TEST_HC_OPTS) [...] foo.hs -o foo. What if we simply stripped of the problematic bits (TOP and the includes) and instead defined TEST_HC, TEST_HC_OPTS to point to the GHC we've built and our config etc? I don't know how many such variables have to be defined, but I was looking around yesterday and I've really just seen these 2. I haven't looked everywhere though of course.

It sounds to me like such an approach could work and has the huge advantage that this doesn't require a single change to GHC's codebase. I'm not sure defining TEST_HC and what not on the spot is the best option, but I do think it is necessary to work with variations on those Makefiles that don't rely on half of the make based build system, therefore the suggestion to strip off the includes. If we don't do this, then we don't have to just hook into the python driver correctly but also into the make build system, and I suspect this is something we want to avoid as much as possible, or even entirely.

sighingnow · 2018-01-18T11:00:21Z

Yeah, we just need to remove the include in Makefile scripts and do some tricks in python's run_command to set necessary variables, for example, $TOP.

We could leave the .T file as it is and only modify the Makefile in every test cases (remove these include lines).

alpmestan · 2018-01-18T11:09:29Z

Yeah, we just need to remove the include in Makefile scripts and do some tricks in python's run_command to set necessary variables.

Either that or define those variables in our modified Makefiles, or in the environment in which the python script is executed by Hadrian, if we want to leave GHC's testsuite driver code untouched (which would be a good thing IMO, but it's absolutely not necessary).

We could leave the .T file as it is and only modify the Makefile in every test cases.

Right, but I think the makefiles rely on having the .hs (and others) files in the same directory, so we would probably have to move those over as well, without changing them though.

Alright @sighingnow, are you willing to give a shot at implementing this or should I?

sighingnow · 2018-01-18T11:30:04Z

I still don't think we need to move these files, before run_command the python driver has already changed to the right directory. For example:

cd "testsuite/tests/printer/Ppr017.run" && $MAKE -s --no-print-directory ppr017

are you willing to give a shot at implementing this or should I?

Do you mean modify the test cases' Makefile and hack the environment variables before run test ?

I think we can pass necessary variables via system's environment variable. Define these variables in Makefile is not a good idea since we may want to run test with another ghc instance.

sighingnow · 2018-01-18T11:41:53Z

are you willing to give a shot at implementing this or should I?

@alpmestan I'm really not familiar with hadrian's codebase so it would be better if you could submit such a patch :)

I'll try to follow your work.

alpmestan · 2018-01-18T11:42:10Z

Do you mean modify the test cases' Makefile and hack the environment variables before run test ?

Well, anything that gets all the tests to be executed properly, really :) If you want to implement a plan you have in mind, that's fine too!

alpmestan · 2018-01-18T11:44:52Z

Just to clarify: I'm by no means an expert on Hadrian's code. I've just been working on/with it for several few weeks. I took the liberty to give some feedback and suggestions on ways to fix the test rules because I'm myself facing the same problem. I'm nowhere saying my suggestions are better, just trying to figure out a way to get hadrian to be a fine companion for running tests.

snowleopard · 2018-01-18T15:32:44Z

I'm really not familiar with hadrian's codebase
I'm by no means an expert on Hadrian's code

@sighingnow @alpmestan I'm happy to be your go-to expert on Hadrian :) Feel free to submit half-cooked PRs for review and I'll suggest how to make the best use of Hadrian infrastructure and/or do some polishing after merging. Thank you for helping!

alpmestan · 2018-01-18T17:09:05Z

@snowleopard For the record, @sighingnow and I had a little chat by email, and he's willing to take a stab at implementing something along the lines of what we have been suggesting above. By combining everyone's knowledge, fixes and experiments, we will get that testsuite running smoothly sooner or later :)

(While you're around Andrey, I think it might be good that you and (Moritz and/or me) have a little discussion about #445 and how we should go about merging most/all of that work -- should I start an email thread? Sorry for the off-topic question on this PR.)

snowleopard · 2018-01-19T09:17:13Z

@alpmestan Awesome!

Sure, feel free to start an email thread to discuss #445. There is already a bit of a discussion in the PR itself, so let's include all participants. We could also setup a Skype chat to speed things up.

Signed-off-by: HE, Tao <[email protected]>

sighingnow · 2018-01-19T16:38:34Z

Update summary:

Move TestArg definition to CommandLine.hs.
Add runTestBuilderArgs under the Settings/Builders directory to collect all necessary arguments to run python driver.
Add timeout program target in testRules.
Set environment variables TEST_HS and TEST_HC_OPTS before run python script, Now the tests which use Makefile script can work after deleting include lines in their Makefile, (for example, T2578 and T14626 under codeGen/should_compile), without touching or moving the .T file.

Help wanted:

I'm really not sure if I do the right things in runTestBuilderArgs in the newly created RunTest.hs.
Currently I need to compute ghcFlags twice, in runTestBuilderArgs and testRules. I need to pass these flags to python driver, as well as set these flags to TEST_HC_OPTS environment variables. How could I avoid the boilerplate code ?

/cc @snowleopard @alpmestan

alpmestan · 2018-01-21T00:44:20Z

Regarding question number 1: the code so far looks reasonable. It is much nicer to have a proper "builder" for running the tests. If I were you I would not worry much more about that code at least until the tests all run properly. :)

Regarding question number 2, I think @snowleopard or someone else will have a better answer.

snowleopard · 2018-01-21T01:18:18Z

src/Rules/Test.hs

+        liftIO $ do
+            setEnv "MAKE" makePath
+            setEnv "TEST_HC" ghcPath
+            setEnv "TEST_HC_OPTS" ghcFlags


We typically set the environment as follows:

https://github.com/snowleopard/hadrian/blob/master/src/Rules/Gmp.hs#L41-L44

I have tried

-- Set environment variables for test's Makefile. env <- sequence [ builderEnvironment "MAKE" $ Make "" , builderEnvironment "TEST_HC" $ Ghc CompileHs Stage2 , AddEnv "TEST_HC_OPTS" <$> runTestGhcFlags ] -- Execute the test target. buildWithCmdOptions env $ target (vanillaContext Stage2 compiler) RunTest [] []

But it doesn't work. The env variables $MAKE, TEST_HC and TEST_HC_OPTS are still undefined when the python driver invoke the $MAKE command.

Hmm, strange. Thanks for leaving a TODO message, I also suggest to open an issue so we don't forget about this -- the ability to set an environment for a builder is important and we should figure out the best approach for it.

snowleopard · 2018-01-21T01:20:01Z

src/Rules/Test.hs

+        build $ target (vanillaContext Stage2 compiler) RunTest [] []
+
+timeoutPyPath :: FilePath
+timeoutPyPath = "test" -/- "bin" -/- "timeout.py"


In such cases we are usually more direct in Hadrian: test/bin/timeout.py. As far as I know, this doesn't cause any compatibility issues across different OSs.

snowleopard · 2018-01-21T01:21:25Z

src/Settings.hs

@@ -66,3 +66,23 @@ findPackageByName name = find (\pkg -> pkgName pkg == name) knownPackages
 -- | Install's DESTDIR setting.
 getDestDir :: Action FilePath
 getDestDir = fromMaybe "" <$> cmdInstallDestDir
+
+-- | Arguments to run GHC's test script.
+getTestArgs :: Args


Why is this defined here and not in Settings.Builders.RunTest together with the rest of the flags?

I define this method in src/Setting.hs to translate the arguments from command line to Args type.

Without this function I'll need to import module CommandLine into Settings.Builders.RunTest to do the transformation. I don't sure if it's the right way (import CommandLine into the builder module).

The getTestArgs returns the arguments provided by user from command line. In Settings.Builders.RunTest we only process the arguments decided by current GHC's configuration.

We use Settings.Builders.X to define all X-specific arguments. This seems to be relevant only for RunTest.

I don't sure if it's the right way (import CommandLine into the builder module).

Yes, that's fine, as long as this doesn't create nasty import cycles. In fact, we do something very similar in Settings.Builders.Make where we rely on shakeThreads <$> expr getShakeOptions which essentially comes from Hadrian's command line (but captured by Shake).

I have moved the getTestArgs from Settings into Settings.Builders.RunTest and deleted the unused cmdTestArgs function.

snowleopard · 2018-01-21T01:22:49Z

src/Settings/Builders/RunTest.hs

+            , pure "-dno-debug-output"
+            ]
+
+        timeout_prog <- (-/- "test" -/- "bin" -/- "timeout") <$> expr buildRoot


This would be more typical: expr buildRoot <&> "test/bin/timeout".

snowleopard · 2018-01-21T01:25:38Z

src/Settings/Builders/RunTest.hs

+
+        let ifMinGhcVer ver opt = do v <- expr ghcCanonVersion
+                                     if ver <= v then pure opt
+                                                 else pure ""


Isn't this simply (ver <= v) ? pure opt?

Now I move this port to src/Rules/Tests.hs. I still use if ... then ... else because operator ? is only available for Expr monad, can't be used in Action monad.

snowleopard · 2018-01-21T01:29:56Z

I'm really not sure if I do the right things in runTestBuilderArgs in the newly created RunTest.hs.

@sighingnow This looks good to me.

Currently I need to compute ghcFlags twice

Can you simply export ghcFlags from Settings.Builders.RunTest? That's what I would do. I hope that doesn't lead to an import cycle.

Great job! I've left a few other minor comments above. In general, I think you sometimes could shorten the code by using the operator ? for adding certain command line flags conditionally.

Signed-off-by: HE, Tao <[email protected]>

sighingnow · 2018-01-21T15:02:57Z

Can you simply export ghcFlags from Settings.Builders.RunTest? That's what I would do. I hope that doesn't lead to an import cycle.

Now I define ghcFlags in Rules.Test and import it in Settings.Builders.RunTest to make it less boilerplate.

Signed-off-by: HE, Tao <[email protected]>

snowleopard · 2018-01-23T02:21:45Z

@sighingnow Many thanks! I think we are getting close to merging this. My apologies for slow responses, I am travelling this week.

Signed-off-by: HE, Tao <[email protected]>

snowleopard · 2018-01-26T11:43:07Z

src/Builder.hs

             | Sphinx SphinxMode
             | Tar TarMode
             | Unlit
+             | Validate


This seems to be irrelevant for this particular PR? I suggest to drop this to avoid confusion.

snowleopard · 2018-01-26T11:45:24Z

src/Rules/Test.hs

+                ]
+        liftIO $ do
+            writeFile (root -/- timeoutProgPath) script
+            cmd "chmod" [ "+x", root -/- timeoutProgPath ]


There is a utility called makeExecutable that does this.

snowleopard · 2018-01-26T11:47:35Z

src/Rules/Test.hs

+timeoutPyPath = "test/bin/timeout.py"
+
+timeoutProgPath :: FilePath
+timeoutProgPath = "test/bin/timeout"  -- TODO: `.exe` suffix for windows.


You can do <.> exe.

<.> exe will add .exe extension to the file[1], however on linux/unix system we usually don't use specific extension for executable.

http://hackage.haskell.org/package/filepath-1.4.2/docs/System-FilePath-Windows.html#v:-60-.-62-

It's OS-specific exactly for this reason: https://hackage.haskell.org/package/shake-0.16/docs/Development-Shake-FilePath.html#v:exe

That is, on Linux "foo" <.> exe should remain "foo".

Oops, my bad.

snowleopard · 2018-01-26T11:48:14Z

src/Settings/Builders/RunTest.hs

+                ]
+
+    -- TODO
+    , builder Validate ? pure [] ]


Not sure what should be done here, so I suggest to drop this bit.

snowleopard · 2018-01-26T11:50:33Z

@sighingnow I've left a few more comments.

At the top of the PR one of your tasks is to add docs describing how to run tests. Are you going to add these docs as part of this PR? Or do you prefer to do it in a separate PR?

Thank you!

Signed-off-by: HE, Tao <[email protected]>

sighingnow · 2018-01-26T12:14:54Z

I have read the task list in #187 . I prefer to add doc after #187 get fixed.

Many TODOS still exists in this PR. I need more time to understanding other stuffs in GHC's test script, such as --way=... and finish hadrain's 'test' and validate rules.

Signed-off-by: HE, Tao <[email protected]>

sighingnow · 2018-01-26T12:22:54Z

I suggest to use the Squash and merge option when merge this PR to make a clean diff for others who also is interested in this work. Thanks!

@snowleopard

snowleopard · 2018-01-26T13:11:35Z

@sighingnow Many thanks, I've squashed & merged this. Looking forward to seeing more PRs from you :)

snowleopard reviewed Jan 17, 2018

View reviewed changes

alpmestan reviewed Jan 18, 2018

View reviewed changes

sighingnow added 2 commits January 19, 2018 22:26

The RunTest builder and test rule to run GHC's test.

c760850

Signed-off-by: HE, Tao <[email protected]>

Timeout rules.

3833928

Signed-off-by: HE, Tao <[email protected]>

snowleopard reviewed Jan 21, 2018

View reviewed changes

Reduce boilerplate.

01cc4cd

Signed-off-by: HE, Tao <[email protected]>

Fix warning.

c216285

Signed-off-by: HE, Tao <[email protected]>

Move getTestArgs into Settings.Builders.RunTest.

149bcba

Signed-off-by: HE, Tao <[email protected]>

snowleopard reviewed Jan 26, 2018

View reviewed changes

sighingnow added 2 commits January 26, 2018 19:58

Drop validate related code to avoid confusion.

b754d3b

Signed-off-by: HE, Tao <[email protected]>

Replace explicit chmod +x with makeExecutable.

504e9cf

Signed-off-by: HE, Tao <[email protected]>

Fix executable's extension.

5792f53

Signed-off-by: HE, Tao <[email protected]>

snowleopard merged commit 63a5563 into snowleopard:master Jan 26, 2018

alpmestan mentioned this pull request Feb 19, 2018

Run ghc's test can't work #494

Closed

snowleopard mentioned this pull request Apr 24, 2018

build test failed because of python subprocess "Permission denied" #361

Closed

sighingnow mentioned this pull request May 23, 2018

Add options for running testsuite tests #605

Closed

sighingnow deleted the test-rule branch June 17, 2018 07:08

[WIP] Support run GHC's test from hadrian. #495

[WIP] Support run GHC's test from hadrian. #495

Conversation

sighingnow commented Jan 17, 2018 • edited Loading

sighingnow commented Jan 17, 2018

snowleopard commented Jan 17, 2018

snowleopard Jan 17, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sighingnow commented Jan 18, 2018

alpmestan commented Jan 18, 2018

simonmar commented Jan 18, 2018

alpmestan commented Jan 18, 2018

snowleopard commented Jan 18, 2018

alpmestan commented Jan 18, 2018

sighingnow commented Jan 18, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

alpmestan commented Jan 18, 2018

sighingnow commented Jan 18, 2018 • edited Loading

alpmestan commented Jan 18, 2018

sighingnow commented Jan 18, 2018

sighingnow commented Jan 18, 2018

alpmestan commented Jan 18, 2018 • edited Loading

alpmestan commented Jan 18, 2018

snowleopard commented Jan 18, 2018

alpmestan commented Jan 18, 2018

snowleopard commented Jan 19, 2018 • edited Loading

sighingnow commented Jan 19, 2018 • edited Loading

alpmestan commented Jan 21, 2018

Choose a reason for hiding this comment

sighingnow Jan 21, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

snowleopard commented Jan 21, 2018 • edited Loading

sighingnow commented Jan 21, 2018 • edited Loading

snowleopard commented Jan 23, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

snowleopard commented Jan 26, 2018

sighingnow commented Jan 26, 2018

sighingnow commented Jan 26, 2018

snowleopard commented Jan 26, 2018

sighingnow commented Jan 17, 2018 •

edited

Loading

snowleopard Jan 17, 2018 •

edited

Loading

sighingnow commented Jan 18, 2018 •

edited

Loading

alpmestan commented Jan 18, 2018 •

edited

Loading

snowleopard commented Jan 19, 2018 •

edited

Loading

sighingnow commented Jan 19, 2018 •

edited

Loading

sighingnow Jan 21, 2018 •

edited

Loading

snowleopard commented Jan 21, 2018 •

edited

Loading

sighingnow commented Jan 21, 2018 •

edited

Loading