Documented Bulk IO issues #2340

Amruta-Ranade · 2018-01-09T20:05:59Z

Because the PR has multiple reviewers, I have pointed out the sections to be reviewed for each reviewer. Hope that helps!

cockroach-teamcity · 2018-01-09T20:06:04Z

This change is

cockroach-teamcity · 2018-01-09T20:07:23Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/c305a6b4ddec749e364b266e72862bbac4c74bfe/

Amruta-Ranade · 2018-01-09T20:07:34Z

v2.0/backup.md

-{{site.data.alerts.callout_info}}Because CockroachDB is a distributed system, you cannot meaningfully store backups "locally" on nodes. The entire backup file must be stored in a single location, so attempts to store backups locally must point to an NFS drive to be useful.{{site.data.alerts.end}}
+- The backup location parameters often contain special characters that need to be URI-encoded. Use Javascript's [encodeURIComponent](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent) function or Go language's [url.QueryEscape](https://golang.org/pkg/net/url/#QueryEscape) function to URI-encode the parameters. Other languages provide similar functions to URI-encode special characters.
+- [<sup>1</sup>](#import-file-temp-directory-urls) Because CockroachDB is a distributed system, you cannot meaningfully store backups "locally" on nodes. The entire backup file must be stored in a single location, so attempts to store backups locally must point to an NFS drive to be useful.
+- [<sup>2</sup>](#import-file-temp-directory-urls) <span class="version-tag">New in v2.0</span> The file system backup location on a local node or NFS drive is relative to the path specified by the `--external-io-dir` flag set while [starting the node](start-a-node.html). If the flag is set to `disabled`, then backups to local directories and NFS drives are disabled. 


@dt Please review the bullet point.

This is correct, though we could include in the last one (which defaults a subdirectory of the first configured store) or similar if we wanted to.

We could also mention that, if you do not want to restart your nodes to change this flag, you can create symlinks to the locations you want to use from within the extern dir to read/write to paths outside the extern dir

@dt Does this sound right: "The file system backup location on a local node or NFS drive is relative to the path specified by the --external-io-dir flag set while starting the node. If the --external-io-dir flag is set to disabled, then backups to local directories and NFS drives are disabled. By default, the flag is set to the extern subdirectory of the first configured store. To avoid restarting your nodes to set the --external-io-dir flag to the locations you want to use, create symlinks to the desired locations from within the extern directory."

Amruta-Ranade · 2018-01-09T20:07:52Z

v2.0/backup.md


-{{site.data.alerts.callout_info}}Because CockroachDB is a distributed system, you cannot meaningfully store backups "locally" on nodes. The entire backup file must be stored in a single location, so attempts to store backups locally must point to an NFS drive to be useful.{{site.data.alerts.end}}
+- The backup location parameters often contain special characters that need to be URI-encoded. Use Javascript's [encodeURIComponent](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent) function or Go language's [url.QueryEscape](https://golang.org/pkg/net/url/#QueryEscape) function to URI-encode the parameters. Other languages provide similar functions to URI-encode special characters.


@benesch Please review this paragraph.

Amruta-Ranade · 2018-01-09T20:08:10Z

v2.0/import.md

@@ -172,14 +172,17 @@ URLs for the file you want to import and your temp directory must use the follow
 | Google Cloud [<sup>1</sup>](#notes) | `gs` | Bucket name | None |
 | HTTP [<sup>2</sup>](#notes) | `http` | Remote host | N/A |
 | NFS/Local [<sup>3</sup>](#notes) | `nodelocal` | File system location | N/A |
+| S3-compatible services | `s3` | Bucket name | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`, `AWS_ENDPOINT` |


@mjibson Please review this row.

Amruta-Ranade · 2018-01-09T20:08:40Z

v2.0/restore.md

@@ -109,8 +109,13 @@ The URL for your backup's locations must use the following format:
 | Google Cloud Storage | `gs` | Bucket name | None––currently only supports instance auth, but we can build non-instance auth at a customer's request |
 | HTTP | `http` | Remote host | N/A |
 | NFS | `nodelocal` | File system location | N/A |
+| S3-compatible services | `s3` | Bucket name | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`, `AWS_ENDPOINT` |


@mjibson Please review this row.

Amruta-Ranade · 2018-01-09T20:08:55Z

v2.0/restore.md

-{{site.data.alerts.callout_info}}Backups stored on NFSes work only if each node in the cluster accesses the drive in the same way.{{site.data.alerts.end}}
+#### Considerations
+
+- The backup location parameters often contain special characters that need to be URI-encoded. Use Javascript's [encodeURIComponent](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent) function or Go language's [url.QueryEscape](https://golang.org/pkg/net/url/#QueryEscape) function to URI-encode the parameters. Other languages provide similar functions to URI-encode special characters.


@benesch Please review this section

Amruta-Ranade · 2018-01-09T20:09:08Z

v2.0/restore.md

+#### Considerations
+
+- The backup location parameters often contain special characters that need to be URI-encoded. Use Javascript's [encodeURIComponent](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent) function or Go language's [url.QueryEscape](https://golang.org/pkg/net/url/#QueryEscape) function to URI-encode the parameters. Other languages provide similar functions to URI-encode special characters.
+- Backups stored on NFSes work only if each node in the cluster accesses the drive in the same way.


@dt Please review this section

The --external-io-dir mitigates this limitation to some extent -- previously, since the path was relative to /, every node had to mount the NFS share in the same place, like /mnt/nfs/backups or whatever. Now that the backup path is relative to the node's configured external-io-dir, nodes can have the NFS mount at whatever location they want as long as they all point to the same share using their external-io-dir flag.. e.g. one node might be started with --external-io-dir=/mnt/nfs/backups while another might use --external-io-dir=/Volumes/backups or whatever.

The important thing is that all nodes use --external-io-dir to point to the same shared storage.

@dt So I can delete the second bullet (Backups stored on NFSes...), right?

@Amruta-Ranade, @dt, has this conversation been resolved?

Amruta-Ranade · 2018-01-09T20:09:21Z

v2.0/start-a-node.md

@@ -42,6 +42,7 @@ Flag | Description
 `--background` | Set this to start the node in the background. This is better than appending `&` to the command because control is returned to the shell only once the node is ready to accept requests.
 `--cache` | The total size for caches, shared evenly if there are multiple storage devices. This can be a percentage or any bytes-based unit, for example: <br><br>`--cache=25%`<br>`--cache=1000000000 ----> 1000000000 bytes`<br>`--cache=1GB ----> 1000000000 bytes`<br>`--cache=1GiB ----> 1073741824 bytes` <br><br><strong>Note:</strong> If you enter the cache size as a percentage, you might need to escape the `%` sign, for instance, while configuring CockroachDB through systemd service files.<br><br><span class="version-tag">Changed in v1.1: </span>**Default:** `128MiB`<br><br>The default cache size is reasonable for local development clusters. For production deployments, this should be increased to 25% or higher. See [Recommended Production Settings](recommended-production-settings.html#cache-and-sql-memory-size-changed-in-v1-1) for more details.
 `--certs-dir` | The path to the [certificate directory](create-security-certificates.html). The directory must contain valid certificates if running in secure mode.<br><br>**Default:** `${HOME}/.cockroach-certs/`
+`--external-io-dir` | <span class="version-tag">New in v2.0</span> The path of the external IO directory with which the local file access paths are prefixed while performing backup and restore operations using local node directories or NFS drives. If set to `disabled`, backups and restores using local node directories and NFS drives are disabled.<br><br>**Default:** `extern` directory on the first storage device specified by the `--store` flag. 


@dt Please review this section

I think we just call them stores? though we generally only advise configuring one per physical storage device. To be a bit pedantic, since we default to cockroach-data, it isn't always explicitly specified by --store.

I might go for something like:

extern subdirectory of the first configured store (see the --store flag).

Amruta-Ranade · 2018-01-09T20:09:54Z

v2.0/backup.md

 | Backup Location | scheme | host | parameters |
 |-----------------|--------|------|------------|
 | Amazon S3 | `s3` | Bucket name | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` |
 | Azure | `azure` | Container name | `AZURE_ACCOUNT_KEY`, `AZURE_ACCOUNT_NAME` |
 | Google Cloud Storage | `gs` | Bucket name | None––currently only supports instance auth, but we can build non-instance auth at a customer's request |
 | HTTP | `http` | Remote host | N/A |
-| NFS | `nodelocal` | File system location | N/A |
+| NFS [<sup>1</sup>](#considerations) | `nodelocal` | File system location [<sup>2</sup>](#considerations) | N/A |
+| S3-compatible services | `s3` | Bucket name | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`, `AWS_ENDPOINT` |


@mjibson Please review this row.

Amruta-Ranade · 2018-01-09T20:10:23Z

v2.0/import.md


 #### Notes

 [<sup>1</sup>](#import-file-temp-directory-urls) Only supports instance auth.

 [<sup>2</sup>](#import-file-temp-directory-urls) You can easily create your own HTTP server with [Caddy or nginx](create-a-file-server.html).

-[<sup>3</sup>](#import-file-temp-directory-urls) If using NFS for your temp directory, each node in the cluster must have access to the NFS using the same URL.
+[<sup>3</sup>](#import-file-temp-directory-urls) If using NFS for your temp directory, each node in the cluster must have access to the NFS using the same URL. The file system backup location on the NFS drive is relative to the path specified by the `--external-io-dir` flag set while [starting the node](start-a-node.html). If the flag is set to `disabled`, then imports from local directories and NFS drives are disabled. 


@dt Please review this bullet point

First sentence of this is no longer quite true (due to the second).
Now it is important only that each node in your cluster configure its --external-io-dir flag to point to its local mount point of the that NFS share (or, as an alternative, symlink from the default extern subdirectory to the NFS-backed directory).

Amruta-Ranade · 2018-01-09T20:10:36Z

v2.0/import.md

-[<sup>3</sup>](#import-file-temp-directory-urls) If using NFS for your temp directory, each node in the cluster must have access to the NFS using the same URL.
+[<sup>3</sup>](#import-file-temp-directory-urls) If using NFS for your temp directory, each node in the cluster must have access to the NFS using the same URL. The file system backup location on the NFS drive is relative to the path specified by the `--external-io-dir` flag set while [starting the node](start-a-node.html). If the flag is set to `disabled`, then imports from local directories and NFS drives are disabled. 
+
+The location parameters often contain special characters that need to be URI-encoded. Use Javascript's [encodeURIComponent](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent) function or Go language's [url.QueryEscape](https://golang.org/pkg/net/url/#QueryEscape) function to URI-encode the parameters. Other languages provide similar functions to URI-encode special characters.


@benesch Please review this paragraph

maddyblue

My parts LGTM

benesch · 2018-01-10T18:04:25Z

Reviewed 4 of 4 files at r1.
Review status: all files reviewed at latest revision, 10 unresolved discussions, some commit checks failed.

Comments from Reviewable

cockroach-teamcity · 2018-01-11T18:22:07Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/4bbcb7bc712f1b08632cdfc3dba2e445e290bd93/

dt · 2018-01-16T15:44:36Z

Review status: 1 of 4 files reviewed at latest revision, 10 unresolved discussions, some commit checks failed.

v2.0/backup.md, line 135 at r1 (raw file):

Previously, Amruta-Ranade (Amruta Ranade) wrote…

@dt Does this sound right: "The file system backup location on a local node or NFS drive is relative to the path specified by the --external-io-dir flag set while starting the node. If the --external-io-dir flag is set to disabled, then backups to local directories and NFS drives are disabled. By default, the flag is set to the extern subdirectory of the first configured store. To avoid restarting your nodes to set the --external-io-dir flag to the locations you want to use, create symlinks to the desired locations from within the extern directory."

👍 that last sentence could be prefixed with Note: or something or have a you can or some indication that it is an optional suggestion, not a strong recommendation.

v2.0/restore.md, line 117 at r1 (raw file):

Previously, Amruta-Ranade (Amruta Ranade) wrote…

@dt So I can delete the second bullet (Backups stored on NFSes...), right?

yes.

Comments from Reviewable

dt · 2018-01-16T15:44:44Z

Review status: 1 of 4 files reviewed at latest revision, 10 unresolved discussions, some commit checks failed.

Comments from Reviewable

cockroach-teamcity · 2018-01-19T15:07:28Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/4bbcb7bc712f1b08632cdfc3dba2e445e290bd93/

jseldess

Sorry for the delay, @Amruta-Ranade. Looks good overall. Just a few comments/questions.

jseldess · 2018-01-19T15:16:21Z

v2.0/backup.md

-{{site.data.alerts.callout_info}}Because CockroachDB is a distributed system, you cannot meaningfully store backups "locally" on nodes. The entire backup file must be stored in a single location, so attempts to store backups locally must point to an NFS drive to be useful.{{site.data.alerts.end}}
+- The backup location parameters often contain special characters that need to be URI-encoded. Use Javascript's [encodeURIComponent](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent) function or Go language's [url.QueryEscape](https://golang.org/pkg/net/url/#QueryEscape) function to URI-encode the parameters. Other languages provide similar functions to URI-encode special characters.
+- [<sup>1</sup>](#import-file-temp-directory-urls) Because CockroachDB is a distributed system, you cannot meaningfully store backups "locally" on nodes. The entire backup file must be stored in a single location, so attempts to store backups locally must point to an NFS drive to be useful.
+- [<sup>2</sup>](#import-file-temp-directory-urls) <span class="version-tag">New in v2.0</span> The file system backup location on a local node or NFS drive is relative to the path specified by the `--external-io-dir` flag set while [starting the node](start-a-node.html). If the `--external-io-dir` flag is set to `disabled`, then backups to local directories and NFS drives are disabled. By default, the flag is set to the `extern` subdirectory of the first configured [`store`](#store). To avoid restarting your nodes to set the `--external-io-dir` flag to the locations you want to use, create symlinks to the desired locations from within the `extern` directory.


The New in v2.0 doesn't behave well will bulleted lists. Renders weird:

So let's just remove it here. It's called out as new in the cockroach start docs, which is sufficient.

(#store) should be (start-a-node.html#store).

In the last sentence, "To avoid restarting your nodes to set the --external-io-dir flag to the locations you want to use` is confusing me. Can you rephrase?

jseldess · 2018-01-19T15:19:58Z

v2.0/import.md

@@ -172,14 +172,17 @@ URLs for the file you want to import and your temp directory must use the follow
 | Google Cloud [<sup>1</sup>](#notes) | `gs` | Bucket name | None |
 | HTTP [<sup>2</sup>](#notes) | `http` | Remote host | N/A |
 | NFS/Local [<sup>3</sup>](#notes) | `nodelocal` | File system location | N/A |
+| S3-compatible services | `s3` | Bucket name | `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY`, `AWS_REGION`, `AWS_ENDPOINT` |

 #### Notes


Since this is called Considerations in backup.md, perhaps use that heading here as well? Also use bullets like in backup.md?

jseldess · 2018-01-19T15:23:33Z

v2.0/restore.md

+#### Considerations
+
+- The backup location parameters often contain special characters that need to be URI-encoded. Use Javascript's [encodeURIComponent](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent) function or Go language's [url.QueryEscape](https://golang.org/pkg/net/url/#QueryEscape) function to URI-encode the parameters. Other languages provide similar functions to URI-encode special characters.
+- Backups stored on NFSes work only if each node in the cluster accesses the drive in the same way.


@Amruta-Ranade, @dt, has this conversation been resolved?

jseldess · 2018-01-19T15:23:35Z

v2.0/restore.md

+
+- The backup location parameters often contain special characters that need to be URI-encoded. Use Javascript's [encodeURIComponent](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent) function or Go language's [url.QueryEscape](https://golang.org/pkg/net/url/#QueryEscape) function to URI-encode the parameters. Other languages provide similar functions to URI-encode special characters.
+- Backups stored on NFSes work only if each node in the cluster accesses the drive in the same way.
+- <span class="version-tag">New in v2.0</span> The file system backup location on a local node or NFS drive is relative to the path specified by the `--external-io-dir` flag set while [starting the node](start-a-node.html). If the flag is set to `disabled`, then backups and restores from local directories and NFS drives are disabled. 


Same feedback as for backup.md: Let's remove New in v2.0 here.

jseldess · 2018-01-19T15:28:57Z

v2.0/start-a-node.md

@@ -42,6 +42,7 @@ Flag | Description
 `--background` | Set this to start the node in the background. This is better than appending `&` to the command because control is returned to the shell only once the node is ready to accept requests.
 `--cache` | The total size for caches, shared evenly if there are multiple storage devices. This can be a percentage or any bytes-based unit, for example: <br><br>`--cache=25%`<br>`--cache=1000000000 ----> 1000000000 bytes`<br>`--cache=1GB ----> 1000000000 bytes`<br>`--cache=1GiB ----> 1073741824 bytes` <br><br><strong>Note:</strong> If you enter the cache size as a percentage, you might need to escape the `%` sign, for instance, while configuring CockroachDB through systemd service files.<br><br><span class="version-tag">Changed in v1.1: </span>**Default:** `128MiB`<br><br>The default cache size is reasonable for local development clusters. For production deployments, this should be increased to 25% or higher. See [Recommended Production Settings](recommended-production-settings.html#cache-and-sql-memory-size-changed-in-v1-1) for more details.
 `--certs-dir` | The path to the [certificate directory](create-security-certificates.html). The directory must contain valid certificates if running in secure mode.<br><br>**Default:** `${HOME}/.cockroach-certs/`
+`--external-io-dir` | <span class="version-tag">New in v2.0</span> The path of the external IO directory with which the local file access paths are prefixed while performing backup and restore operations using local node directories or NFS drives. If set to `disabled`, backups and restores using local node directories and NFS drives are disabled.<br><br>**Default:** `extern` subdirectory of the first configured `store` (see [`store`](#store) below).<br><br>To avoid restarting your nodes to set the `--external-io-dir` flag to the locations you want to use, create symlinks to the desired locations from within the `extern` directory. 


Add colon to "new" callout, i.e., New in v2.0:.

Let's remove "see store below" and just make the previous instance of store a link, i.e.,

the first configured store.

As I mentioned in my review of backup.md, I don't understand what this is saying:

To avoid restarting your nodes to set the --external-io-dir flag to the locations you want to use, create symlinks to the desired locations from within the extern directory.

Can you rephrase, or explain it to me in other terms so I can figure out what I'm not getting?

cockroach-teamcity · 2018-01-22T17:06:40Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/b5ee4ae0dc2fc2071576c65a9702f83ad9b5cc59/

jseldess · 2018-01-22T17:29:00Z

LGTM

cockroach-teamcity · 2018-01-22T17:31:10Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/87a3214ce872ae0ec05b56f6f3253b08226b9eb5/

…to bulk_io_small_issues

cockroach-teamcity · 2018-01-22T18:46:52Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/47798f0ba170cf2a619aa07894c0216c987e5cf4/

cockroach-teamcity · 2018-01-22T19:17:30Z

http://cockroach-docs-review.s3-website-us-east-1.amazonaws.com/47798f0ba170cf2a619aa07894c0216c987e5cf4/

jseldess · 2018-01-22T19:49:57Z

TC is failing on a link to www.bottlecaps.de/rr/ui, which is an attribution at the bottom of our full sql grammar page. Merging in the expectation that the site comes back eventually.

Documented bulk io issues

c305a6b

Amruta-Ranade commented Jan 9, 2018

View reviewed changes

Amruta-Ranade requested review from benesch, dt and maddyblue January 9, 2018 20:12

maddyblue approved these changes Jan 9, 2018

View reviewed changes

Incorporated David's comments

4bbcb7b

Amruta-Ranade requested a review from jseldess January 11, 2018 18:52

jseldess reviewed Jan 19, 2018

View reviewed changes

Incorporated review comments

b5ee4ae

Merge branch 'master' into bulk_io_small_issues

87a3214

Amruta-Ranade added 2 commits January 22, 2018 13:44

Fixed broken links

728154e

Merge branch 'bulk_io_small_issues' of github.com:cockroachdb/docs in…

47798f0

…to bulk_io_small_issues

jseldess merged commit 7e8e7e6 into master Jan 22, 2018

jseldess deleted the bulk_io_small_issues branch January 22, 2018 19:50

This was referenced Jan 22, 2018

URL-encoding AWS credentials specified as query parameters #2238

Closed

storageccl,cli: add external IO dir cli flag #2254

Closed

storageccl,sqlccl: make nodelocal paths relative to external-io-dir #2253

Closed

storageccl: support S3 endpoints #2255

Closed


		{{site.data.alerts.callout_info}}Because CockroachDB is a distributed system, you cannot meaningfully store backups "locally" on nodes. The entire backup file must be stored in a single location, so attempts to store backups locally must point to an NFS drive to be useful.{{site.data.alerts.end}}
		- The backup location parameters often contain special characters that need to be URI-encoded. Use Javascript's [encodeURIComponent](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent) function or Go language's [url.QueryEscape](https://golang.org/pkg/net/url/#QueryEscape) function to URI-encode the parameters. Other languages provide similar functions to URI-encode special characters.

Documented Bulk IO issues #2340

Documented Bulk IO issues #2340

Conversation

Amruta-Ranade commented Jan 9, 2018 • edited Loading

cockroach-teamcity commented Jan 9, 2018

cockroach-teamcity commented Jan 9, 2018

Amruta-Ranade Jan 9, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Amruta-Ranade Jan 9, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dt Jan 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dt Jan 10, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Amruta-Ranade Jan 9, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Amruta-Ranade Jan 9, 2018 • edited Loading

Choose a reason for hiding this comment

maddyblue left a comment

Choose a reason for hiding this comment

benesch commented Jan 10, 2018

cockroach-teamcity commented Jan 11, 2018

dt commented Jan 16, 2018

dt commented Jan 16, 2018

cockroach-teamcity commented Jan 19, 2018

jseldess left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

cockroach-teamcity commented Jan 22, 2018

jseldess commented Jan 22, 2018

cockroach-teamcity commented Jan 22, 2018

cockroach-teamcity commented Jan 22, 2018

cockroach-teamcity commented Jan 22, 2018

jseldess commented Jan 22, 2018

Amruta-Ranade commented Jan 9, 2018 •

edited

Loading

Amruta-Ranade Jan 9, 2018 •

edited

Loading

Amruta-Ranade Jan 9, 2018 •

edited

Loading

dt Jan 10, 2018 •

edited

Loading

dt Jan 10, 2018 •

edited

Loading

Amruta-Ranade Jan 9, 2018 •

edited

Loading

Amruta-Ranade Jan 9, 2018 •

edited

Loading