+ +If you have any questions, concerns or ideas, let us know +[here](https://dvc.org/support) and our stellar team will get back to you in no +time. diff --git a/content/blog/2019-04-18-april-19-dvc-heartbeat.md b/content/blog/2019-04-18-april-19-dvc-heartbeat.md new file mode 100644 index 00000000000..5f98edd1da2 --- /dev/null +++ b/content/blog/2019-04-18-april-19-dvc-heartbeat.md @@ -0,0 +1,264 @@ +--- +title: April ’19 DVC❤️Heartbeat +date: 2019-04-18 +description: | + DVC creator Dmitry Petrov is giving a talk on PyCon 2019 🎤, new DVC logo + design, new Discord discussions, interesting reads that caught our eye, and + everything along the way. +descriptionLong: | + Every month we are sharing here our news, findings, interesting reads, + community takeaways, and everything along the way. + + Some of those are related to our brainchild [DVC](https://dvc.org) and its + journey. The others are a collection of exciting stories and ideas centered + around ML best practices and workflow. +picture: ../../static/uploads/images/2019-04-18/post-image.jpeg +author: ../authors/svetlana_grinchenko.md +commentsUrl: https://discuss.dvc.org/t/april-19-dvc-heartbeat/292 +tags: + - Heartbeat + - Discord Gems + - PyCon +--- + +## News and links + +We have some exciting news to share this month! + +DVC is going to [PyCon 2019](https://us.pycon.org/2019/)! It is the first +conference that we attend as a team. When we say ‘team’ — we mean it. Our +engineers are flying from all over the globe to get together offline and catch +up with fellow Pythonistas. + +The [speaker pipeline](https://us.pycon.org/2019/schedule/talks/list/) is +amazing! DVC creator Dmitry Petrov is giving a talk on +[Machine learning model and dataset versioning practices](https://us.pycon.org/2019/schedule/presentation/176/). + +Stop by our booth at the Startup Row on Saturday, May 4, reach out and let us +know that you are willing to chat, or simply find a person with a huge DVC owl +on their shirt! + +Speaking of the owls — DVC has done some rebranding recently and we love our new +logo. Special thanks to [99designs.com](https://99designs.com/) for building a +great platform for finding trusted designers. + + + +DVC is moving fast (almost as fast as my two-year-old). We do our best to keep +up and totally love all the buzz in our community channels lately! + +Here is a number of interesting reads that caught our eye: + +- **[A walkthrough of DVC](https://blog.codecentric.de/en/2019/03/walkthrough-dvc/) + by [Bert Besser](https://www.linkedin.com/in/bert-besser-284564182/)** + +
+ +## Discord gems + +There are lots of hidden gems in our Discord community discussions. Sometimes +they are scattered all over the channels and hard to track down. + +We are sifting through the issues and discussions and share with you the most +interesting takeaways. + +### Q: [What are the system requirements to install DVC (type of operating system, dependencies of another application (as GIT), memory, cpu, etc).](https://discordapp.com/channels/485586884165107732/485596304961962003/552098155861114891) + +- It supports Windows, Mac, Linux. Python 2 and 3. + +- No specific CPU or RAM requirements — it’s a lightweight command line tool and + should be able run pretty much everywhere you can run Python. + +- It depends on a few Python libraries that it installs as dependencies (they + are specified in the + [`requirements.txt`](https://github.com/iterative/dvc/blob/master/requirements.txt)). + +- It does not depend on Git and theoretically could be run without any SCM. + Running it on top of a Git repository however is recommended and gives you an + ability to actually save history of datasets, models, etc (even though it does + not put them into Git directly). + +### Q: [Do I have to buy a server license to run DVC, do you have this?](https://discordapp.com/channels/485586884165107732/485596304961962003/560212552638791706) + +No server licenses for DVC. It is 100% free and open source. + +### Q: [What is the storage limit when using DVC?](https://discordapp.com/channels/485586884165107732/485596304961962003/560154903331340289) + +I am trying to version control datasets and models with >10 GB (Potentially even +bigger). Can DVC handle this? + +There is no limit. None enforced by DVC itself. It depends on the size of your +local or [remote storages](https://dvc.org/doc/commands-reference/remote). You +need to have some space available on S3, your SSH server or other storage you +are using to keep these data files, models and their version, which you would +like to store. + +### Q: [How does DVC know the sequence of stages to run](https://discordapp.com/channels/485586884165107732/485596304961962003/553731815228178433)? + +How does it connect them? Does it see that there is a dependency which is +outputted from the first run? + +DVC figures out the pipeline by looking at the dependencies and outputs of the +stages. For example, having the following: + +`gist:SvetaGr/a2a28fbc9db0a675422785bc5f925e14#heartbeat-dvc-run-2019-04.sh` + +you will end up with two stages: `download.dvc` and `duplicate.dvc`. The +download one will have `joke.txt` as an output . The duplicate one defined +`joke.txt` as a dependency, as it is the same file. DVC detects that and creates +a pipeline by joining those stages. + +You can inspect the content of each stage file +[here](https://dvc.org/doc/user-guide/dvc-file-format) (they are human +readable). + +### Q: [Is it possible to use the same data of a remote in two different repositories?](https://discordapp.com/channels/485586884165107732/485596304961962003/560022999848321026) + +(e.g. in one repo `run dvc pull -r my_remote` to pull some data and running the +same command in a different git repo should also pull the same) + +Yes! It’s a frequent scenario for multiple repos to share remotes and even local +cache. DVC file serves as a link to the actual data. If you add the same DVC +file (e.g. `data.dvc`) to the new repo and do `dvc pull -r remotename data.dvc`- +it will fetch data. You have to use `dvc remote add` first to specify the +coordinates of the remote storage you would like to share in every project. +Alternatively (check out the question below), you could use `--global` to +specify a single default remote (and/or cache dir) per machine. + +### Q: [Could I set a global remote server, instead of config in each project?](https://discordapp.com/channels/485586884165107732/485586884165107734/559653121228275727) + +Use `--global` when you specify the remote settings. Then remote will be visible +for all projects on the same machine. `--global` — saves remote configuration to +the global config (e.g. `~/.config/dvc/config`) instead of a per project one — +`.dvc/config`. See more details +[here](https://dvc.org/doc/commands-reference/remote-add). + +### Q: [How do I version a large dataset in S3 or any other storage?](https://discordapp.com/channels/485586884165107732/485596304961962003/554679392823934977) + +We would recommend to skim through our +[get started](https://dvc.org/doc/get-started) tutorial, to summarize the data +versioning process of DVC: + +- You create stage (aka DVC) files by adding, importing files (`dvc add` / + `dvc import`) , or run a command to generate files: + +```dvc +$ dvc run --out file.csv "wget https://example.com/file.csv" +``` + +- This stage files are tracked by `git` + +- You use git to retrieve previous stage files (e.g. `git checkout v1.0`) + +- Then use `dvc checkout` to retrieve all the files related by those stage files + +All your files (with each different version) are stored in a `.dvc/cache` +directory, that you sync with a remote file storage (for example, S3) using the +`dvc push` or `dvc pull` commands (analogous to a `git push` / `git pull`, but +instead of syncing your `.git`, you are syncing your `.dvc` directory) on a +remote repository (let’s say an S3 bucket). + +### Q: [How do I move/rename a DVC-file?](https://discordapp.com/channels/485586884165107732/485596304961962003/558216007684980736) + +If you need to move your dvc file somewhere, it is pretty easy, even if done +manually: + +`gist:SvetaGr/b25a5b45773bf94d36e60d48462502f4#heartbeat-dvc-rename.sh` + +### Q: [I performed `dvc push` of a file to a remote. On the remote there is created a directory called `8f` with a file inside called `2ec34faf91ff15ef64abf3fbffa7ee`. The original CSV file doesn’t appear on the remote. Is that expected behaviour?](https://discordapp.com/channels/485586884165107732/485596304961962003/555431645402890255) + +This is an expected behavior. DVC saves files under the name created from their +checksum in order to prevent duplication. If you delete “pushed” file in your +project directory and perform `dvc pull`, DVC will take care of pulling the file +and renaming it to “original” name. + +Below are some details about how DVC cache works, just to illustrate the logic. +When you add a data source: + +`gist:SvetaGr/b69fa8ce36bcce00ecd69e7f2d7ccd2e#heartbeat-remote-file-naming.sh` + +It computes the (md5) checksum of the file and generates a DVC file with related +information: + +`gist:SvetaGr/110ae76df929654ec573ea9e4b1e1980#heartbeat-dvc-file-2019-04.yaml` + +The original file is moved to the cache and a link or copy (depending on your +filesystem) is created to replace it on your working space: + +`gist:SvetaGr/133cb93e5a21c6f21a86f8709ed39ea9#heartbeat-cache-structure-2019-04.sh` + +### Q: [Is it possible to integrate dvc with our in-house tools developed in Python?](https://discordapp.com/channels/485586884165107732/485586884165107734/553570391000481802) + +Absolutely! There are three ways you could interact with DVC: + +1. Use [subprocess](https://docs.python.org/3/library/subprocess.html) to launch + DVC + +2. Use `from dvc.main import main` and use it with regular CLI logic like + `ret = main(‘add’, ‘foo’)` + +3. Use our internal API (see `dvc/repo` and `dvc/command` in our source to get a + grasp of it). It is not officially public yet, and we don’t have any special + docs for it, but it is fairly stable and could definitely be used for a POC. + We’ll add docs and all the official stuff for it in the not-so-distant + future. + +### Q: [Can I still track the linkage between data and model without using `dvc run`](https://discordapp.com/channels/485586884165107732/485586884165107734/555750217522216990) and a graph of tasks? Basically what would like extremely minimal DVC invasion into my GIT repo for an existing machine learning application? + +There are two options: + +1. Use `dvc add` to track models and/or input datasets. It should be enough if + you use `git commit` on DVC files produced by `dvc add`. This is the very + minimum you can get with DVC and it does not require using DVC run. Check the + first part (up to the Pipelines/Add transformations section) of the DVC + [get started](https://dvc.org/doc/get-started). + +2. You could use `--no-exec` in `dvc run` and then just `dvc commit` and + `git commit` the results. That way you’ll get your DVC files with all the + linkages, without having to actually run your commands through DVC. + +If you have any questions, concerns or ideas, let us know +[here](https://dvc.org/support) and our stellar team will get back to you in no +time. diff --git a/content/blog/2019-04-23-dvc-project-ideas-for-google-summer-of-docs-2019.md b/content/blog/2019-04-23-dvc-project-ideas-for-google-summer-of-docs-2019.md new file mode 100644 index 00000000000..62985559a99 --- /dev/null +++ b/content/blog/2019-04-23-dvc-project-ideas-for-google-summer-of-docs-2019.md @@ -0,0 +1,226 @@ +--- +title: DVC project ideas for Google Season of Docs 2019 +date: 2019-04-23 +description: | + DVC.org is applying for Google Season of Docs — a new and unique program + sponsored by Google that pairs technical writers with open source projects to + collaborate on the open source project documentation. +descriptionLong: | + [DVC.org](https://dvc.org) is applying for + [Google Season of Docs](https://developers.google.com/season-of-docs/) — a new + and unique program sponsored by Google that pairs technical writers with open source projects to + collaborate on the open source project documentation. + + It’s happening for the first time in 2019 and we are excited about the + opportunity to be a part of it! +picture: ../../static/uploads/images/2019-04-23/post-image.png +author: ../authors/svetlana_grinchenko.md +commentsUrl: https://discuss.dvc.org/t/dvc-project-ideas-for-google-season-of-docs-2019/291 +tags: + - Google Season of Docs + - Python + - Documentation +--- + +We strongly believe that well-shaped documentation is key for making the product +truly open. We have been investing lots of time and energy in improving our docs +lately. Being a team of 90% engineers we are eager to welcome the writers into +our team and our community. We are happy to share our experience, introduce them +to the world of open source and machine learning best practices, guide through +the OS contribution process and work together on improving our documentation. + +DVC was started in late 2017 by a data scientist and an engineer. It is now +growing pretty fast and though our in-house team is quite small, we have to +thank our contributors (more than 80 in both code and docs) for developing DVC +with us. When working with DVC the technical writer will not only get lots of +hands-on experience in writing technical docs, but will also immerse into DVC +community — a warm and welcoming gathering of ML and DS enthusiasts and an +invaluable source of inspiration and expertise in ML engineering. + +### About DVC + +DVC is a brainchild of a data scientist and an engineer, that was created to +fill in the gaps in the ML processes tooling and evolved into a successful open +source project. + +ML brings changes in development and research processes. These ML processes +require new tools for data versioning, ML pipeline versioning, resource +management for model training and others that haven’t been formalized. The +traditional software development tools do not fully cover ML team’s needs but +there are no good alternatives. It makes engineers to custom develop a new +toolset to manage data files, keep track of ML experiments and connect data and +source code together. The ML process becomes very fragile and requires tons of +tribal knowledge. + +We have been working on [DVC](http://DVC.org) by adopting best ML practices and +turning them into Git-like command line tool. DVC versions multi-gigabyte +datasets and ML models, make them shareable and reproducible. The tool helps to +organize a more rigorous process around datasets and the data derivatives. Your +favorite cloud storage (S3, GCS, or bare metal SSH server) could be used with +DVC as a data file backend. + +If you are interested in learning a little bit more about DVC and its journey, +here is a great interview with DVC creator in the Episode 206 of +Podcast.**init**. Listen to it +[HERE ](https://www.pythonpodcast.com/data-version-control-episode-206/)or read +the transcript +[HERE.](https://towardsdatascience.com/data-version-control-with-dvc-what-do-the-authors-have-to-say-3c3b10f27ee) + +### The state of DVC documentation + +DVC is a pretty young project, developed and maintained solely by engineers. As +many OS projects we started from the bottom and for a long time our +[documentation](https://dvc.org/doc) was a bunch of bits and pieces. Nowadays +improving documentation is one of our top priorities. We moved to the new +in-house built documentation engine and started working with several technical +writers. Certain parts have been tremendously improved recently, e.g. +[Get Started](https://dvc.org/doc/get-started) and +[certain parts of Commands Reference](https://dvc.org/doc/commands-reference/fetch) +. So far most of our documentation has been written majorly by the engineering +team and there is need for improving the overall structure and making some parts +more friendly from a new user perspective. We have mostly complete +[reference documentation](https://dvc.org/doc/commands-reference) for each +command, although some functions are missing good actionable examples. We also +have a [User Guide](https://dvc.org/doc/user-guide/dvc-files-and-directories), +however it is not in very good shape. We strive for making our documentation +clear and comprehensive for users of various backgrounds and proficiency levels +and this is where we do need some fresh perspective. + +### How DVC documentation is built + +We have an open Github Apache-2 licensed repository for the +[DVC website](https://github.com/iterative/dvc.org), the documentation engine +and the [documentation files](https://github.com/iterative/dvc.org). The website +is built with Node.js + React, including the documentation engine (built +in-house). + +Each documentation page is a static Markdown file in the repository, e.g. +[example here](https://github.com/iterative/dvc.org/blob/master/static/docs/get-started/example-versioning.md.). +It is rendered dynamically in the browser, no preprocessing is required. It +means that tech writers or contributors need to write/edit a Markdown file, +create a pull request and merge it into the master branch of the +[repository.](https://github.com/iterative/dvc.org) The complete +[documentation contributing guide](https://github.com/iterative/dvc.org/blob/master/README.md#contributing) +describes the directory structure and locations for the different documentation +parts. + +### DVC’s approach to documentation work + +Documentation tasks and issues are maintained on our doc’s GitHub +[issue tracker](https://github.com/iterative/dvc.org/issues). Changes to the +documentation are made via pull requests on GitHub, and go through our standard +review process which is the same for documentation and code. A technical writer +would be trained in working with our current development process. It generally +means that tech writers or contributors need to write/edit a Markdown file, use +git and Github to create a pull request and publish it. The documentation +[contributing guide](https://github.com/iterative/dvc.org/blob/master/README.md#contributing) +includes style conventions and other details. Documentation is considered of the +same importance as code. Engineering team has a policy to write or update the +relevant sections if something new is released. If it’s something too involved +engineers may create a ticket and ask for help. There is one maintainer who is +responsible for doing final reviews and merging the changes. In this sense, our +documentation is very similar to any other open source project. + +## Project ideas for GSoD’19 + +We identified a number of ideas to work on and there are two major topics these +ideas fall into. Both topics are pretty broad and we don’t expect we can +completely cover them during this GSoD but hopefully we can make certain +progress. + +First of all, we want to bring more structure and logic to our documentation to +improve user onboarding experience. The goal is for a new user to have a clear +path they can follow and understand what takeaways each part of the +documentation provides. In particular, improving how +[Get Started](https://dvc.org/doc/get-started), +[Tutorials](https://dvc.org/doc/tutorial) and +[Examples](https://dvc.org/doc/get-started/example-versioning) relate to each +other, restructuring the existing [User Guide](https://dvc.org/doc/user-guide) +to explain basic concepts, and writing more use cases that resonate with ML +engineers and data scientists. + +The other issue we would like to tackle is improving and expanding the existing +reference docs — commands descriptions, examples, etc. It involves filling in +the gaps and developing new sections, similar to +[this one](https://dvc.org/doc/commands-reference/fetch). We would also love to +see more illustrative materials. + +### Project 1: Improving and expanding User Guide + +**Description and details:** Reviewing, restructuring and filling major gaps in +the User Guide (introductory parts of the basic concepts of DVC), e.g. have a +look at [this ticket](https://github.com/iterative/dvc.org/issues/144) or +[this one](https://github.com/iterative/dvc.org/issues/53). + +**Mentors**: [@shcheklein](https://github.com/shcheklein) and +[@dmpetrov](https://github.com/dmpetrov) + +### Project 2: Expanding and developing new tutorials and use cases. + +**Description and details:** We already have some requests for more tutorials, +e.g. [this ticket](https://github.com/iterative/dvc.org/issues/96). Here is +another good [use case request](https://github.com/iterative/dvc.org/issues/194) +. If you are going to work on this project you would need some domain knowledge, +preferably some basic ML or data science experience. + +**Mentors**: [@shcheklein](https://github.com/shcheklein) and +[@dmpetrov](https://github.com/dmpetrov) + +### Project 3: Improving new user onboarding + +**Description and details:** Analyze and restructure user walkthrough across +[Get started](https://dvc.org/doc/get-started), +[Tutorials](https://dvc.org/doc/tutorial) and +[Examples](https://dvc.org/doc/get-started/example-versioning). These three have +one thing in common — hands-on experience with DVC. If you choose this project, +we will work together to come up with a better location for the Examples (to +move them out of the Get Started shadow), and a better location for the +Tutorials (to reference external tutorials that were developed by our community +members and published on different platforms). + +**Mentors**: [@shcheklein](https://github.com/shcheklein) and +[@dmpetrov](https://github.com/dmpetrov) + +### Project 4: Improving commands reference + +**Description and details:** We will work on improving our +[Commands reference](https://dvc.org/doc/commands-reference) section. This +includes expanding and filling in the gaps. One of the biggest pain points right +now are Examples. Users want them to be +[easy to run and try](https://github.com/iterative/dvc.org/issues/198) and here +is a lot to be done in terms of improvement. We have a good example of how is +should be done [here](https://dvc.org/doc/commands-reference/fetch). + +**Mentors**: [@shcheklein](https://github.com/shcheklein) and +[@dmpetrov](https://github.com/dmpetrov) + +### Project 5: Describe and integrate “DVC packages” + +**Description and details:** Describe the brand new feature “DVC packages” and +integrate it with the rest of the documentation. We have been working hard to +release a few new commands to help with datasets management (have a look at +[this ticket](https://github.com/iterative/dvc/issues/1487)). It’s a major +feature that deserves its place in the Get Started, Use cases, Commands +Reference, etc. + +**Mentors**: [@shcheklein](https://github.com/shcheklein) and +[@dmpetrov](https://github.com/dmpetrov) + +The ideas we outline above are just an example of what we can work on. We are +open for any other suggestions and would like to work together with the +technical writer to make the contribution experience both useful and enjoyable +for all parties involved. If you have any suggestions or questions we would love +to hear from you => DVC.org/support and our DMs on +[Twitter](https://twitter.com/DVCorg) are always open! + +
+ +Special thanks to the [NumFOCUS](https://numfocus.org/) for the ideas list +inspiration. + +If you are a tech writer — check the +[Technical writer guide](https://developers.google.com/season-of-docs/docs/tech-writer-guide). +From April 30, 2019 you can see the list of participating open source +organizations on the [Season of Docs website](https://g.co/seasonofdocs). The +application period for technical writers opens on **May 29, 2019** and ends on +June 28, 2019. diff --git a/content/blog/2019-05-21-may-19-dvc-heartbeat.md b/content/blog/2019-05-21-may-19-dvc-heartbeat.md new file mode 100644 index 00000000000..f3bc2762587 --- /dev/null +++ b/content/blog/2019-05-21-may-19-dvc-heartbeat.md @@ -0,0 +1,300 @@ +--- +title: May ’19 DVC❤️Heartbeat +date: 2019-05-21 +description: | + DVC accepted into Google Season of Docs 🎉, Dmitry's talk at the O’Reilly AI + Conference, new portion of Discord gems, and articles either created or + brought to us by our community. +descriptionLong: | + Every month we are sharing here our news, findings, interesting reads, + community takeaways, and everything along the way. + + Some of those are related to our brainchild [DVC](https://dvc.org) and its + journey. The others are a collection of exciting stories and ideas centered + around ML best practices and workflow. +picture: ../../static/uploads/images/2019-05-21/post-image.jpeg +pictureComment: | + Kudos to [StickerMule.com](https://www.stickermule.com) for our amazing + stickers (and great customer service)! +author: ../authors/svetlana_grinchenko.md +commentsUrl: https://discuss.dvc.org/t/may-19-dvc-heartbeat/290 +tags: + - Heartbeat + - Discord Gems + - Google Season of Docs +--- + +## News and links + +This section of DVC Heartbeat is growing with every new Issue and this is +already quite a good piece of news! + +One of the most exciting things we want to share this month is acceptance of DVC +into the [Google Season of Docs](https://developers.google.com/season-of-docs/). +It is a new and unique program sponsored by Google that pairs technical writers +with open source projects to collaborate and improve the open source project +documentation. You can find the outline of DVC vision and project ideas in +[this dedicated blogpost](https://blog.dataversioncontrol.com/dvc-project-ideas-for-google-summer-of-docs-2019-defe3a73b248) +and check the +[full list of participating open source organizations](https://developers.google.com/season-of-docs/docs/participants/). +Technically the +[program is starting in a few months](https://developers.google.com/season-of-docs/docs/timeline), +but there is already a fantastic increase in the amount of commits and +contributors, and we absolutely love it! + +The other important milestone for us was the first offline meeting with our +distributed remote team. Working side by side and having non-Zoom meetings with +the team was amazing. Joining our forces to prepare for the upcoming conferences +turned out to be the most valuable, educating and uniting experience for the +whole team. + +It’s a shame that our tech lead was unable to join us it due to another visa +denial. We do hope he will finally make it to the USA for the next big +conference. + + + +While we were busy finalizing all the PyCon 2019 prep, our own +[Dmitry Petrov](https://twitter.com/FullStackML) flew to New York to speak at +the +[O’Reilly AI Conference](https://conferences.oreilly.com/artificial-intelligence/ai-ny) +about the +[Open Source tools for Machine Learning Models and Datasets versioning](https://www.oreilly.com/library/view/artificial-intelligence-conference/9781492050544/video324691.html). +Unfortunately the video is available for the registered users only (with a free +trial option) but you can have a look at Dmitry’s slides +[here](https://www.slideshare.net/DmitryPetrov15/dvc-oreilly-artificial-intelligence-conference-2019-new-york). + + + +We renamed our Twitter! Our old handle was a bit misleading and we moved from +@Iterativeai to [@DVCorg](https://twitter.com/DVCorg) (yet keep the old one for +future projects). + +Our team is so happy every time we discover an article featuring DVC or +addressing one of the burning ML issues we are trying to solve. Here are some of +our favorite links from the past month: + +- **[Version Control For Your Machine Learning Projects — Episode 206](https://www.pythonpodcast.com/data-version-control-episode-206/)** + by **[Tobias Macey](https://www.linkedin.com/in/tmacey/)** + +
+ +## Discord gems + +There are lots of hidden gems in our Discord community discussions. Sometimes +they are scattered all over the channels and hard to track down. + +We are sifting through the issues and discussions and share with you the most +interesting takeaways. + +### Q: This might be [a favourite gem of ours ](https://discordapp.com/channels/485586884165107732/485598848111083531/572960640122224640) — our engineers are so fast that someone assumed they were bots. + +We feared that too until we met them in person. They appeared to be real (unless +bots also love Ramen now)! + + + +### Q: [Is this the best way to track data with DVC when code and data are separate?](https://discordapp.com/channels/485586884165107732/485596304961962003/572974117351849997) Having being burned by this a couple of times, i.e accidentally pushing large files to GitHub, I now keep my code and data separate. + +Every time you run `dvc add` to start tracking some data artifact, its path is +automatically added to the `.gitignore` file, as a result it is hard to commit +it to git by mistake — you would need to explicitly modify the `.gitignore` +first. The feature to track some external data is called +[external outputs](https://dvc.org/doc/user-guide/external-outputs) (if all you +need is to track some data artifacts). Usually it is used when you have some +data on S3 or SSH and don’t want to pull it into your working space, but it’s +working even when your data is located on the same machine outside of the +repository. + +### Q: [How do I wrap a step that downloads a file/directory into a DVC stage?](https://discordapp.com/channels/485586884165107732/485596304961962003/571342592508428289) I want to ensure that it runs only if file has no been downloaded yet + +Use `dvc import` to track and download the remote data first time and next time +when you do dvc repro if data has changed remotely. If you don’t want to track +remote changes (lock the data after it was downloaded), use `dvc run` with a +dummy dependency (any text file will do you do not touch) that runs an actual +wget/curl to get the data. + +### Q: [How do I show a pipeline that does not have a default Dvcfile?](https://discordapp.com/channels/485586884165107732/485596304961962003/570943786151313408) (e.g. I assigned all files names manually with `-f` in the `dvc run` command and I just don’t have `Dvcfile` anymore) + +Almost any command in DVC that deals with pipelines (set of DVC-files) accepts a +single stage as a target, for example: + +```dvc +$ dvc pipeline show — ascii model.dvc +``` + +### Q: [DVC hangs or I’m getting `database is locked` issue](https://discordapp.com/channels/485586884165107732/485596304961962003/570843482218823682) + +It’s a well known problem with NFS, CIFS (Azure) — they do not support file +locks properly which is required by the SQLLite engine to operate. The easiest +workaround — don’t create a DVC project on network attached partition. In +certain cases a fix can be made by changing mounting options, check +[this discussion](https://discordapp.com/channels/485586884165107732/485596304961962003/570276668694855690) +for the Azure ML Service. + +### Q: [How do I use DVC if I use a separate drive to store the data and a small/fast SSD to run computations?](https://discordapp.com/channels/485586884165107732/485596304961962003/570091809594671126) I don’t have enough space to bring data to my working space. + +An excellent question! The short answer is: + +```dvc +# To move your data cache to a big partition +$ dvc cache dir --local /path/to/an/external/partition + +# To enable symlinks/harldinks to avoid actual copying +$ dvc config cache.type reflink, hardlink, symlink, copy + +# To protect the cache +$ dvc config cache.protected true +``` + +The last one is highly recommended to make links in your working space read-only +to avoid corrupting the cache. Read more about different link types +[here](https://dvc.org/doc/user-guide/large-dataset-optimization). + +To add your data first time to the DVC cache, do a clone of the repository on a +big partition and run `dvc add` to add your data. Then you can do `git pull`, +`dvc pull` on a small partition and DVC will create all the necessary links. + +### Q: [Why I’m getting `Paths for outs overlap` error when I run `dvc add` or `dvc run`?](https://discordapp.com/channels/485586884165107732/485596304961962003/571335064374345749) + +Usually it means that a parent directory of one of the arguments for `dvc add` / +`dvc run` is already tracked. For example, you’ve added the whole datasets +directory already. And now you are trying to add a subdirectory, which is +already tracked as a part of the datasets one. No need to do that. You could +`dvc add datasets` or `dvc repro datasets.dvc` to save changes. + +### Q: [I’m getting `ascii codec can’t encode character` error on DVC commands when I deal with unicode file names](https://discordapp.com/channels/485586884165107732/485596304961962003/567310354766495747) + +[Check the locale settings you have](https://perlgeek.de/en/article/set-up-a-clean-utf8-environment) +(`locale` command in Linux). Python expects a locale that can handle unicode +printing. Usually it’s solved with these commands: `export LC_ALL=en_US.UTF-8` +and `export LANG=en_US.UTF-8`. You can place those exports into `.bashrc` or +other file that defines your environment. + +### Q: [Does DVC use the same logins `aws-cli` has when using an S3 bucket as its repo/remote storage](https://discordapp.com/channels/485586884165107732/485596304961962003/563149775340568576)? + +In short — yes, but it can be also configured. DVC is going to use either your +default profile (from `~/.aws/*`) or your env vars by default. If you need more +flexibility (e.g. you need to use different credentials for different projects, +etc) check out +[this guide](https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-profiles.html) +to configure custom aws profiles and then you could use them with DVC using +these +[remote options](https://dvc.org/doc/commands-reference/remote-add#options). + +### Q: [How can I output multiple metrics from a single file?](https://discordapp.com/channels/485586884165107732/485596304961962003/566000729505136661) + +Let’s say I have the following in a file: + +```json +{ + “AUC_RATIO”: + { + “train”: 0.8922748258797667, + “valid”: 0.8561602726251776, + “xval”: 0.8843431199314923 + } +} +``` + +How can I show both `train` and `valid` without `xval`? + +You can use `dvc metrics show` command `--xpath` option and provide multiple +attribute names to it: + +```dvc +$ dvc metrics show metrics.json \ + --type json \ + --xpath AUC_RATIO[train,valid] + metrics.json: + 0.89227482588 + 0.856160272625 +``` + +### Q: [What is the quickest way to add a new dependency to a DVC-file?](https://discordapp.com/channels/485586884165107732/485596304961962003/566314479499870211) + +There are a few options to add a new dependency: + +- simply opening a file with your favorite editor and adding a dependency there + without md5. DVC will understand that that stage is changed and will re-run + and re-calculate md5 checksums during the next DVC repro; + +- use `dvc run --no-exec` is another option. It will rewrite the existing file + for you with new parameters. + +### Q: [Is there a way to add a dependency to a python package, so it runs a stage again if it imported the updated library?](https://discordapp.com/channels/485586884165107732/485596304961962003/566315265646788628) + +The only recommended way so far would be to somehow make DVC know about your +package’s version. One way to do that would be to create a separate stage that +would be dynamically printing version of that specific package into a file, that +your stage would depend on: + +```dvc +$ dvc run -o mypkgver 'pip show mypkg > mypkgver’ +$ dvc run -d mypkgver -d ... -o .. mycmd +``` + +### Q: [Is there anyway to forcibly recompute the hashes of dependencies in a pipeline DVC-file?](https://discordapp.com/channels/485586884165107732/485596304961962003/564807276146458624) + +E.g. I made some whitespace/comment changes in my code and I want to tell DVC +“it’s ok, you don’t have to recompute everything”. + +Yes, you could `dvc commit -f`. It will save all current checksum without +re-running your commands. + +### Q: [I have projects that use data that’s stored in S3. I never have data locally to use `dvc push`, but I would like to have this data version controlled.](https://discordapp.com/channels/485586884165107732/485596304961962003/563352000281182218) Is there a way to use the features of DVC in this use case? + +Yes! This DVC features is called +[external outputs](https://dvc.org/doc/user-guide/external-outputs) and +[external dependencies](https://dvc.org/doc/user-guide/external-dependencies). +You can use one of them or both to track, process, and version your data on a +cloud storage without downloading it locally. + +
+ +If you have any questions, concerns or ideas, let us know +[here](https://dvc.org/support) and our stellar team will get back to you in no +time! diff --git a/content/blog/2019-06-26-june-19-dvc-heartbeat.md b/content/blog/2019-06-26-june-19-dvc-heartbeat.md new file mode 100644 index 00000000000..f602b772029 --- /dev/null +++ b/content/blog/2019-06-26-june-19-dvc-heartbeat.md @@ -0,0 +1,233 @@ +--- +title: June ’19 DVC❤️Heartbeat +date: 2019-06-26 +description: | + First DVC user survey, sharing our PyCon experience, new portion of Discord + discussions, and articles either created or brought to us by our community. +descriptionLong: | + Every month we are sharing here our news, findings, interesting reads, + community takeaways, and everything along the way. + + Some of those are related to our brainchild [DVC](https://dvc.org) and its + journey. The others are a collection of exciting stories and ideas centered + around ML best practices and workflow. +picture: ../../static/uploads/images/2019-06-26/post-image.png +pictureComment: | + Thanks to the amazing [Signaturit Tech](https://twitter.com/SignaturitTech) + team for this + [photo](https://twitter.com/SignaturitTech/status/1127927520140120065?s=20)! +author: ../authors/svetlana_grinchenko.md +commentsUrl: https://discuss.dvc.org/t/june-19-dvc-heartbeat/289 +tags: + - PyCon + - Heartbeat + - Discord Gems +--- + +## News and links + +We want to start by saying to our users, contributors, and community members how +grateful we are for the fantastic work you are doing contributing to DVC, giving +talks about DVC, sharing your feedback, use cases and your concerns. A huge +thank you to each of you from the DVC team! + +We would love to give back and support any positive initiative around DVC — just +let us know [here](https://dvc.org/support) and we will send you a bunch of cool +swag, connect to a tech expert or find another way to support your project. Our +[DMs on Twitter](https://twitter.com/DVCorg) are open, too. + +**And if you have 4 minutes to spare, we are conducting out first +[DVC user survey](https://docs.google.com/forms/d/1tmn8YHLUkeSi5AIq4DGJi28iZy9HTazl6DWKe3Hxpnc/edit?ts=5cfc47c2) +and would love to hear from you!** + +Aside from admiring great DVC-related content from our users we have one more +reason to particularly enjoy the past month — DVC team went to Cleveland to +attend [PyCon 2019](https://us.pycon.org/2019/about/) and it was a blast! + + _Amazing +[Jennifer](https://github.com/sureL) and her artwork for our +[SupportOpenSource](https://twitter.com/hashtag/SupportOpenSource) contest_ + +We had it all. Running our first ever conference booth, leading an impromptu +unconference discussion and arranging some cool +[#SupportOpenSource](https://twitter.com/hashtag/SupportOpenSource?src=hashtag_click) +activities was great! Last-minute accommodation cancellations, booth equipment +delivery issues, and being late for our very own talk was not so great. Will be +sharing more about it in a separate blogpost soon. + +https://youtu.be/jkfh2PM5Sz8 + +Here is [Dmitry Petrov](https://twitter.com/FullStackML)’s PyCon +[talk](https://www.youtube.com/watch?v=jkfh2PM5Sz8) and +[slides](https://docs.google.com/presentation/d/1CYt0w8WoZAXiQEtVDVDsTnQumzdZx91v32MwEK20R-E/edit) +on Machine learning model and dataset versioning practices. + +We absolutely loved being at PyCon and can’t wait for our next conference! + +
+ +Our team is so happy every time we discover an article featuring DVC or +addressing one of the burning ML issues we are trying to solve. Here are some of +the links that caught our eye past month: + +- **[The Rise of DataOps (from the ashes of Data Governance)](https://towardsdatascience.com/the-rise-of-dataops-from-the-ashes-of-data-governance-da3e0c3ac2c4) + by [Ryan Gross](https://towardsdatascience.com/@ryanwgross).** + +A brilliant comprehensive read on the current data management issues. It might +be the best article we have ever read on this subject. Every word strongly +resonates with our vision and ideas behind DVC. Highly recommended by DVC team! + +
+ +## Discord gems + +There are lots of hidden gems in our Discord community discussions. Sometimes +they are scattered all over the channels and hard to track down. + +We are sifting through the issues and discussions and share with you the most +interesting takeaways. + +### Q: [Does DVC support Azure Data Lake Gen1?](https://discordapp.com/channels/485586884165107732/563406153334128681/575655655629651968) + +Azure data lake is HDFS compatible. And DVC supports HDFS remotes. Give it a try +and let us know if you hit any problems [here](https://dvc.org/chat). + +### Q: [An excellent discussion on versioning tabular (SQL) data.](https://discordapp.com/channels/485586884165107732/563406153334128681/575681811401801748) Do you know of any tools that deal better with SQL-specific versioning? + +It’s a wide topic. The actual solution might depend on a specific scenario and +what exactly needs to be versioned. DVC does not provide any special +functionality on top of databases to version their content. + +Depending on your use case, our recommendation would be to run SQL and pull the +result file (CSV/TSV file?) that then can be used to do analysis. This file can +be taken under DVC control. Alternatively, in certain cases source files (that +are used to populate the databases) can be taken under control and we can keep +versions of them, or track incoming updates. + +Read the +[discussion](https://discordapp.com/channels/485586884165107732/563406153334128681/575681811401801748) +to learn more. + +### Q: [How does DVC do the versioning between binary files?](https://discordapp.com/channels/485586884165107732/563406153334128681/575686711821205504) Is there a binary diff, similar to git? Or is every version stored distinctly in full? + +DVC is just saving every file as is, we don’t use binary diffs right now. There +won’t be a full directory (if you added just a few files to a 10M files +directory) duplication, though, since we treat every file inside as a separate +entity. + +### Q: [Is there a way to pass parameters from e.g. `dvc repro` to stages?](https://discordapp.com/channels/485586884165107732/563406153334128681/576160840701575169) + +The simplest option is to create a config file — json or whatnot — that your +scripts would read and your stages depend on. + +### Q: [What is the best way to get cached output files from different branches simultaneously?](https://discordapp.com/channels/485586884165107732/563406153334128681/577852740034625576) For example, cached tensorboard files from different branches to compare experiments. + +There is a way to do that through our (still not officially released) API pretty +easily. Here is an +[example script](https://cdn.discordapp.com/attachments/563406153334128681/577894682722304030/dvc_get_output_files.py) +how it could be done. + +### Q: [Docker and DVC.](https://discordapp.com/channels/485586884165107732/563406153334128681/583949033685516299) To being able to push/pull data we need to run a git clone to get DVC-files and remote definitions — but we worry that would make the container quite heavy (since it contains our entire project history). + +You can do `git clone — depth 1`, which will not download any history except the +latest commits. + +### Q: [After DVC pushing the same file, it creates multiple copies of the same file. Is that how it’s supposed to work?](https://discordapp.com/channels/485586884165107732/485596304961962003/574133734136086559) + +If you are pushing the same file, there are no copies pushed or saved in the +cache. DVC is using checksums to identify files, so if you add the same file +once again, it will detect that cache for it is already in the local cache and +wont copy it again to cache. Same with dvc push, if it sees that you already +have cache file with that checksum on your remote, it won’t upload it again. + +### Q: [How do I uninstall DVC on Mac (installed via `pkg` installer)?](https://discordapp.com/channels/485586884165107732/485596304961962003/574941227624169492) + +Something like this should work: + +```dvc +$ which dvc +/usr/local/bin/dvc -> /usr/local/lib/dvc/dvc + +$ ls -la /usr/local/bin/dvc +/usr/local/bin/dvc -> /usr/local/lib/dvc/dvc + +$ sudo rm -f /usr/local/bin/dvc +$ sudo rm -rf /usr/local/lib/dvc +$ sudo pkgutil --forget com.iterative.dvc +``` + +### Q: [How do I pull from a public S3 bucket (that contains DVC remote)?](https://discordapp.com/channels/485586884165107732/485596304961962003/575236576309674024) + +Just add public URL of the bucket as an HTTP endpoint. See +[here](https://github.com/iterative/example-get-started/blob/master/.dvc/config) +for an example. +[https://remote.dvc.org/get-started](https://remote.dvc.org/get-started) is made +to redirect to the S3 bucket anyone can read from. + +### Q: [I’m getting the same error over and over about locking:](https://discordapp.com/channels/485586884165107732/485596304961962003/575535709490905101) `ERROR: failed to lock before running a command — cannot perform the cmd since DVC is busy and locked. Please retry the command later.` + +Most likely it happens due to an attempt to run DVC on NFS that has some +configuration problems. There is a +[well known problem with DVC on NFS](https://github.com/iterative/dvc/issues/1918) +— sometimes it hangs on trying to lock a file. The usual workaround for this +problem is to allocate DVC cache on NFS, but run the project (git clone, DVC +metafiles, etc) on the local file system. Read +[this answer](https://discuss.dvc.org/t/share-nas-data-in-server/180/4?u=shcheklein) +to see how it can be setup. + +
+ +If you have any questions, concerns or ideas, let us know in the comments below +or connect with DVC team [here](https://dvc.org/support). Our +[DMs on Twitter](https://twitter.com/DVCorg) are open, too. diff --git a/content/blog/2019-08-01-july-19-dvc-heartbeat.md b/content/blog/2019-08-01-july-19-dvc-heartbeat.md new file mode 100644 index 00000000000..8df4729ee1a --- /dev/null +++ b/content/blog/2019-08-01-july-19-dvc-heartbeat.md @@ -0,0 +1,212 @@ +--- +title: July ’19 DVC❤️Heartbeat +date: 2019-08-01 +description: | + As we continue to grow DVC together with our fantastic contributors, we enjoy + more and more insights, discussions, and articles either created or brought to + us by our community. +descriptionLong: | + Every month we are sharing here our news, findings, interesting reads, + community takeaways, and everything along the way. + + Some of those are related to our brainchild [DVC](https://dvc.org) and its + journey. The others are a collection of exciting stories and ideas centered + around ML best practices and workflow. +picture: ../../static/uploads/images/2019-08-01/post-image.png +pictureComment: | + Special edition + [DVC shirt](https://twitter.com/rkuprieiev/status/1144298339200098306?s=20). + We made this one for [Ruslan](https://github.com/efiop) — DVC maintainer and + the best tech lead. +author: ../authors/svetlana_grinchenko.md +commentsUrl: https://discuss.dvc.org/t/july-19-dvc-heartbeat/288 +tags: + - Heartbeat + - Open Source Summit + - Discord Gems +--- + +## News and links + +As we continue to grow DVC together with our fantastic contributors, we enjoy +more and more insights, discussions, and articles either created or brought to +us by our community. We feel it is the right time to start sharing more of your +news, your stories and your discoveries. New Heartbeat is here! + +Speaking of our own news — next month DVC team is going to the +[Open Source North America Summit](https://events.linuxfoundation.org/events/open-source-summit-north-america-2019/). +It is taking place in San Diego on August 21–23. +[Dmitry](https://ossna19.sched.com/speaker/dmitry35) and +[Sveta](https://ossna19.sched.com/speaker/svetlanagrinchenko) will be giving +talks and we will run a booth. So looking forward to it! Stop by for a chat and +some cool swag. And if you are in San Diego on those days and want to catch up — +please let us know [here](http://dvc.org/support) or on Twitter! + +
[D] How do you manage your machine learning experiments? from r/MachineLearning+ +
+ +## Discord gems + +There are lots of hidden gems in our Discord community discussions. Sometimes +they are scattered all over the channels and hard to track down. + +We are sifting through the issues and discussions and share with you the most +interesting takeaways. + +### Q: I have within one git repository different folders with very different content (basically different projects, or content I want to have different permissions to), and I thought about using different buckets in AWS as remotes. [I’m not sure if it’s possible with DVC to store some files in some remote, and some other files in some other remote, is it?](https://discordapp.com/channels/485586884165107732/485596304961962003/575718048330416158) + +You can definitely add more than one remote (see +[dvc remote add](https://dvc.org/doc/commands-reference/remote-add)) and then +[dvc push](https://dvc.org/doc/commands-reference/push) has a `-R` option to +pick which one to send the cached data files (deps, outs, etc) to. We would not +recommend doing this though. It complicates the commands you have to run — you +will need to remember to specify a remote name for every command that deals with +data — `push`, `pull`, `gc`, `fetch`, `status`, etc. Please, leave a comment in +the relevant issue [here](https://github.com/iterative/dvc/issues/2095) if this +case is important for you. + +### Q: [Is that possible with DVC to have multiple (few) metric files and compare them all at once?](https://discordapp.com/channels/485586884165107732/485596304961962003/578532350221352987) For example, we’d like to consider as metrics the loss of a neural network training process (loss as a `-M` output of a training stage), and also apart knowing the accuracy of the NN on a test set (another `-M` output of eval stage). + +Yes, it is totally fine to use `-M` in different stages. `dvc metrics show` will +just show both metrics. + +### Q: [I have a scenario where an artifacts (data) folder is created by the dvc run command via the `-o` flag. I have manually added another file into or modified the artifacts folder but when I do `dvc push` nothing happens, is there anyway around this?](https://discordapp.com/channels/485586884165107732/485596304961962003/577362750443880449) + +Let’s first do a quick recap on how DVC handles data files (you can definitely +find more information on the [DVC documentation site](http://dvc.org/docs)). + +- When you do `dvc add`, `dvc run` or `dvc import` DVC puts artifacts (in case + of `dvc run` artifacts == outputs produced by the command) into `.dvc/cache` + directory (default cache location). You don’t see this happening because + [DVC keeps links](https://dvc.org/doc/user-guide/large-dataset-optimization) + (or in certain cases creates a copy) to these files/directories. + +- `dvc push` does not move files from the workspace (that what you see) to the + remote storage, it always moves files/directories that are already in cache + (default is .dvc/cache). + +- So, now you’ve added a file manually, or made some other modifications. But + these files are not in cache yet. The analogy would be `git commit`. You + change the file, you do `git commit`, only after that you can push something + to Git server (Github/Gitlab, etc). The difference is that DVC is doing commit + (moves files to cache) automatically in certain cases — `dvc add`, `dvc run`, + etc. + +There is an explicit command — `dvc commit` - that you should run if you want to +enforce the change to the output produced by `dvc run`. This command will update +the corresponding DVC- files (.dvc extension) and will move data to cache. After +that you should be able to run `dvc push` to save your data on the external +storage. + +Note, when you do an explicit commit like this you are potentially “breaking” +the reproducibility. In a sense that there is no guarantee now that your +directory can be produced by `dvc run`/`dvc repro` — since you changed it +manually. + +### Q: [I’d like to transform my dataset in-place to avoid copying it, but I can’t use `dvc run` to do this because it doesn’t allow the same directory as an output and a dependency.](https://discordapp.com/channels/485586884165107732/485596304961962003/578898899469729796) + +You could do this in one step (one stage). So that getting your data and +modifying it, is one stage. So you don’t depend on the data folder. You just +could depend on your download + modifying script. + +### Q: [Can anyone tell me what this error message is about?](https://discordapp.com/channels/485586884165107732/485596304961962003/579283950778712076) “To avoid unpredictable behavior, rerun command with non overlapping outs paths.” + +Most likely it means that there is a DVC-file that have the same output twice. +Or there two DVC-files that share the same output file. + +### Q: [I’m getting “No such file or directory” error when I do `dvc run` or `dvc repro`](https://discordapp.com/channels/485586884165107732/485596304961962003/580176327701823498). The command runs find if I don’t use DVC. + +That happens because dvc run is trying to ensure that your command is the one +creating your output and removes existing outputs before executing the command. +So that when you run `dvc repro` later, it will be able to fully reproduce the +output. So you need to make the script create the directory or file. + +### Q: [I’m implementing a CI/CD and I would like to simplify my CI/CD or even my training code (keeping them cloud agnostic) by using `dvc pull` inside my Docker container when initializing a training job. ](https://discordapp.com/channels/485586884165107732/485596304961962003/581256265234251776) Can DVC be used in this way? + +Yes, it’s definitely a valid case for DVC. There are different ways of +organizing the storage that training machines are using to access data. From the +very simple — using local storage volume and pulling data from the remote +storage every time — to using NAS or EFS to store a shared DVC cache. + +### Q: [I was able to follow the getting started examples, however now I am trying to push my data to Github, I keep getting the following error: “ERROR: failed to push data to the cloud — upload is not supported by https remote”.](https://discordapp.com/channels/485586884165107732/563406153334128681/598866528984891403) + +HTTP remotes do not support upload yet. Example Get Started repository is using +HTTP to keep it read-only and abstract the actual storage provider we are using +internally. If you actually check the remote URL, you should see that it is an +S3 bucket and AWS provides an HTTP end-point to read data from it. + +### Q: I’m looking to configure AWS S3 as a storage for DVC. I’ve set up the remotes and initialized dvc in the git repository. I tried testing it by pushing a dataset in the form of an excel file. The command completed without any issues but this is what I’m seeing in S3. [DVC seems to have created a subdirectory in the intended directory called “35” where it placed this file with a strange name.](https://discordapp.com/channels/485586884165107732/485596304961962003/585967551708921856) + +This is not an issue, it is an implementation detail. There’s no current way to +upload the files with the original filename (In this case, the S3 bucket will +have the file `data.csv` but with another name `20/893143…`). The reason behind +this decision is because we want to store a file only once no matter how many +dataset versions it’s used in. Also, it’s a reliable way to uniquely identify +the file. You don’t have to be afraid that someone decided to create a file with +the same name (path) but a different content. + +### Q: [Is it possible to only have a shared ‘local’ cache and no remote?](https://discordapp.com/channels/485586884165107732/563406153334128681/587730054893666326) I’m trying to figure out how to use this in a 40 node cluster which already has very fast NFS storage across all the nodes. Not storing everything twice seems desirable. Esp. for the multi-TB input data + +Yes and it’s one of the very common use case, actually. All you need to do is to +use dvc cache dir command to setup an external cache. There are few caveats +though. Please, read +[this link](https://discuss.dvc.org/t/share-nas-data-in-server/180/4?u=shcheklein) +for an example of the workflow. + +
+ +If you have any questions, concerns or ideas, let us know in the comments below +or connect with DVC team [here](https://dvc.org/support). Our +[DMs on Twitter](https://twitter.com/DVCorg) are always open, too. diff --git a/content/blog/2019-09-26-september-19-dvc-heartbeat.md b/content/blog/2019-09-26-september-19-dvc-heartbeat.md new file mode 100644 index 00000000000..f9e1c8c5df0 --- /dev/null +++ b/content/blog/2019-09-26-september-19-dvc-heartbeat.md @@ -0,0 +1,355 @@ +--- +title: September ’19 DVC❤️Heartbeat +date: 2019-09-26 +description: | + Announcing our first meetup in San Francisco, kicking off Google Season of + Docs program, sharing Open Source Summit experience, and more news, links, and + gems. +descriptionLong: | + Every month we are sharing here our news, findings, interesting reads, + community takeaways, and everything along the way. + + Some of those are related to our brainchild [DVC](https://dvc.org) and its + journey. The others are a collection of exciting stories and ideas centered + around ML best practices and workflow. +picture: ../../static/uploads/images/2019-09-26/post-image.jpeg +author: ../authors/svetlana_grinchenko.md +commentsUrl: https://discuss.dvc.org/t/september-19-dvc-heartbeat/287 +tags: + - Discord Gems + - Heartbeat + - Meetup + - Open Source Summit +--- + +## News and links + +We are super excited to co-host our very first +**[meetup in San Francisco on October 10](https://www.meetup.com/San-Francisco-Machine-Learning-Meetup/events/264846847/)**! +We will gather at the brand new Dropbox HQ office at 6:30 pm to discuss +open-source tools to version control ML models and experiments. +[Dmitry Petrov](https://twitter.com/FullStackML) is teaming up with +[Daniel Fischetti](https://www.linkedin.com/in/daniel-fischetti-4a6592bb/) from +[Standard Cognition](https://standard.ai/) to discuss best ML practices. Join us +and save your spot now: + +
+ +Here are some of the great pieces of content around DVC and ML ops that we +discovered in July and August: + +- ** Great insightful discussion on Twitter about versioning ML projects started + by [Nathan Benaich](https://medium.com/@NathanBenaich).** + + https://twitter.com/NathanBenaich/status/1151815916512010242 + +- **[Our Machine Learning Workflow: DVC, MLFlow and Training in Docker Containers](https://medium.com/ixorthink/our-machine-learning-workflow-dvc-mlflow-and-training-in-docker-containers-5b9c80cdf804) + by [Ward Van Laer](https://medium.com/@ward.vanlaer).** + +> It is possible to manage your work flow using open-source and free tools. + +
+ +## Discord gems + +There are lots of hidden gems in our Discord community discussions. Sometimes +they are scattered all over the channels and hard to track down. + +We are sifting through the issues and discussions and share with you the most +interesting takeaways. + +### Q: I’m getting an error message while trying to use AWS S3 storage: `ERROR: failed to push data to the cloud — Unable to locate credentials.` [Any ideas what’s happening?](https://discordapp.com/channels/485586884165107732/563406153334128681/587792932061577218) + +Most likely you haven’t configured your S3 credentials/AWS account yet. Please, +read the full documentation on the AWS website. The short version of what should +be done is the following: + +- [Create your AWS account.](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) + +- Log in to your AWS Management Console. + +- Click on your user name at the top right of the page. + +- Click on the Security Credentials link from the drop-down menu. + +- Find the Access Credentials section, and copy the latest `Access Key ID`. + +- Click on the Show link in the same row, and copy the `Secret Access Key`. + +Follow +[this link](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-configure.html) +to setup your environment. + +### Q: I added data with `dvc add` or `dvc run` and see that it takes twice what it was before (with `du` command). [Does it mean that DVC copies data that is added under its control? How do I prevent this from happening?](https://discordapp.com/channels/485586884165107732/563406153334128681/595402051203235861) + +To give a short summary — by default, DVC copies the files from your working +directory to the cache (this is for safety reasons, it is better to duplicate +the data). If you have reflinks (copy-on-write) enabled on your file system, DVC +will use that method — which is as safe as copying. You can also configure DVC +to use hardlinks/symlinks to save some space and time, but it will require +enabling the protected mode (making data files in workspace read-only). Read +more details [here](https://dvc.org/doc/user-guide/large-dataset-optimization). + +### Q: [How concurrent-friendly is the cache? And different remotes? Is it safe to have several containers/nodes fill the same cache at the same time?](https://discordapp.com/channels/485586884165107732/563406153334128681/599345778703597568) + +It is safe and a very common use case for DVC to have a shared cache. Please, +check [this thread](https://discuss.dvc.org/t/share-nas-data-in-server/180/12), +for example. + +### Q:[What is the proper way to exit the ASCII visualization?](https://discordapp.com/channels/485586884165107732/563406153334128681/603890677176336394) (when you run `dvc pipeline show` command). + +See this +[document](https://dvc.org/doc/commands-reference/pipeline/show#options). To +navigate, use arrows or W, A, S, D keys. To exit, press Q. + +### Q: [Is there an issue if I set my `cache.s3` external cache to my default remote?](https://discordapp.com/channels/485586884165107732/563406153334128681/606197026488844338) I don’t quite understand what an external cache is for other than I have to have it for external outputs. + +Short answer is that we would suggest keeping them separately to avoid possible +checksum overlaps. Checksum on S3 might theoretically overlap with our checksums +(with the content of the file being different), so it could be dangerous. The +chances of losing data are pretty slim, but we would not risk it. Right now, we +are working on making sure there are no possible overlapping. + +### Q: [What’s the right procedure to move a step .dvc file around the project?](https://discordapp.com/channels/485586884165107732/563406153334128681/606425815139221504) + +Assuming the file was created with `dvc run`. There are few possible ways. +Obvious one is to delete the file and create a new one with +`dvc run --no-exec -f file/path/and/name.dvc`. Another possibility is to +rename/move and then edit manually. See +[this document](https://dvc.org/doc/user-guide/dvc-file-format) that describes +how DVC-files are organized. No matter what method you use, you can run +`dvc commit file.dvc` to save changes without running the command again. + +### Q: [`dvc status` doesn’t seem to report things that need to be dvc pushed, is that by design?](https://discordapp.com/channels/485586884165107732/563406153334128681/606917839688957952) + +You should try with dvc status `--cloud` or `dvc status --remote
+ +If you have any questions, concerns or ideas, let us know in the comments below +or connect with DVC team [here](https://dvc.org/support). Our +[DMs on Twitter](https://twitter.com/DVCorg) are always open, too. diff --git a/content/blog/2019-10-08-dvc-org-for-hacktoberfest-2019.md b/content/blog/2019-10-08-dvc-org-for-hacktoberfest-2019.md new file mode 100644 index 00000000000..b9e38784c55 --- /dev/null +++ b/content/blog/2019-10-08-dvc-org-for-hacktoberfest-2019.md @@ -0,0 +1,113 @@ +--- +title: DVC.org for Hacktoberfest 2019 +date: 2019-10-08 +description: | + Our favorite month of the year Hacktoberfest is already in full swing and we + at DVC.org are so excited to be a part of it! +descriptionLong: | + Our favorite month of the year + [Hacktoberfest](https://hacktoberfest.digitalocean.com/) is already in full + swing and we at [DVC.org](https://dvc.org) are so excited to be a part of it! +picture: ../../static/uploads/images/2019-10-08/post-image.png +author: ../authors/svetlana_grinchenko.md +commentsUrl: https://discuss.dvc.org/t/dvc-org-for-hacktoberfest-2019/286 +tags: + - Hacktoberfest +--- + +[Hacktoberfest](https://hacktoberfest.digitalocean.com/) is a monthly-long +program that celebrates open source and encourages you to contribute to open +source projects (and rewards you with stickers and a cool T-shirt!). Whether +you’re a seasoned contributor or looking for projects to contribute to for the +first time, you’re welcome to participate! + +It is the 6th season of Hacktoberfest and the 2d year of participating for +DVC.org team. We really enjoyed it in 2018 and this year we are upping the game +with our own cool stickers, special edition T-shirts and a +[collection of carefully picked tickets](https://github.com/iterative/dvc/labels/hacktoberfest). + +### How to participate? + +If you haven’t started your Hacktoberfest challenge yet, it is just the right +time, you have 3 weeks left to submit PRs and get your swag! Here are some +important details: + +- Hacktoberfest is open to everyone in the global community. + +- You can sign up anytime between October 1 and October 31. Make sure to sign up + on the + [official Hacktoberfest website](https://hacktoberfest.digitalocean.com/) for + your PRs to count. + +- To get a shirt, you must make 4 legit pull requests (PRs) between October 1–31 + in any time zone. + +- Pull requests can be made in any public GitHub-hosted repositories/projects, + not just the ones highlighted. + +And the special addition from DVC.org team: + +- Look through the list of + [DVC Hacktoberfest tickets](https://github.com/iterative/dvc/labels/hacktoberfest) + or the list of + [good DVC first issues](https://github.com/iterative/dvc/labels/good%20first%20issue). + +- Make a PR to DVC and get our stickers. + +- Close three issues for DVC and get a special DVC T-shirt. + +### Why contribute to DVC? + +[DVC](http://dvc.org) (Data Version Control) is a relatively young open source +project. It was started in late 2017 by a data scientist and an engineer to fill +in the gaps in the ML processes tooling. Nowadays DVC is growing pretty fast and +though our in-house team is quite small, we have to thank our contributors (more +than 100 in both code and docs) for developing DVC with us. + +DVC is participating in Hacktoberfest for 2 years in a row to bring more people +into open source, to learn from them and to give back by sharing our own +experience. This year we decided to focus on a single important topic for us — +improving UI/UX. + +As our contributors and maintainers were sifting through the feature requests, +bugs, and improvements to create a good +[list of Hacktoberfest tickets](https://github.com/iterative/dvc/labels/hacktoberfest), +we noticed that UI/UX label on Github is popping up again and again. DVC is a +command line tool, and improving UI/UX in our case means making decisions on how +to name command options, where and when to use +[confirmation prompts](https://github.com/iterative/dvc/issues/2498) and/or +where abort execution, what exactly user would expect to see in the output, how +to test it later, etc. + +Why improving UI/UX appears to be so important for DVC at this stage? Perhaps +because the project is more mature now and we are ready to spend more time on +polishing it. Or maybe because it is still too-engineering focused and we used +to disregard/de-prioritize all this ‘fancy’ stuff. Or it is because we just lack +experience in creating good CLI UI/UX! + +One or another, those are great reasons to focus on improving UI (in a broader +sense than just GUI), improving docs, creating powerful consistent experience +for our users and increasing accessibility of DVC. + +That’s how +[Heroku’s CLI style guide](https://devcenter.heroku.com/articles/cli-style-guide) +starts: + +> Heroku CLI plugins should provide a clear user experience, targeted primarily +> for human readability and usability, which delights the user, while at the +> same time supporting advanced users and output formats. This article provides +> a clear direction for designing delightful CLI plugins. + +At DVC we are building user experience in line with these principles too, but we +also have our own challenges. And here we turn for help to the global open +source community and all the contributors out there. + +For all of us who have a heart for open source — let’s discuss, contribute, +learn, take the technologies forward and build something great together! + +Happy hacking! + +
+ +We are happy to hear from you [here](https://dvc.org/support). Our +[DMs on Twitter](https://twitter.com/DVCorg) are always open, too! diff --git a/content/blog/2019-11-05-october-19-dvc-heartbeat.md b/content/blog/2019-11-05-october-19-dvc-heartbeat.md new file mode 100644 index 00000000000..92b76deced5 --- /dev/null +++ b/content/blog/2019-11-05-october-19-dvc-heartbeat.md @@ -0,0 +1,269 @@ +--- +title: October ’19 DVC❤️Heartbeat +date: 2019-11-05 +description: | + Every month we are sharing here our news, findings, interesting reads, + community takeaways, and everything along the way. +descriptionLong: | + Every month we are sharing here our news, findings, interesting reads, + community takeaways, and everything along the way. + + Some of those are related to our brainchild [DVC](https://dvc.org) and its + journey. The others are a collection of exciting stories and ideas centered + around ML best practices and workflow. +picture: ../../static/uploads/images/2019-11-05/post-image.png +author: ../authors/svetlana_grinchenko.md +commentsUrl: https://discuss.dvc.org/t/october-19-dvc-heartbeat/285 +tags: + - Meetup + - Heartbeat + - Hacktoberfest +--- + +## News and links + +Autumn is a great season for new beginnings and there is so much we love about +it this year. Here are some of the highlights: + +- Co-hosting our + [first ever meetup](https://www.meetup.com/San-Francisco-Machine-Learning-Meetup/events/264846847/)! + Our [Dmitry Petrov](https://twitter.com/FullStackML) partnering with + [Dan Fischetti](https://www.linkedin.com/in/daniel-fischetti-4a6592bb/) from + [Standard Cognition](https://twitter.com/standardAI) to discuss Open-source + tools to version control Machine Learning models and experiments. The + recording is available now here. + + https://youtu.be/RHQXK7EC0jI + +- [Getting ready for the Hacktoberfest](https://blog.dataversioncontrol.com/dvc-org-for-hacktoberfest-2019-ce5320151a0c) + and having the whole team get together to pick up and label nice issues and be + ready to support the contributors. + +- Discovering some really cool blogposts, talks and tutorials from our users all + over the world: check + [this blogpost in French](https://blog.octo.com/mise-en-application-de-dvc-sur-un-projet-de-machine-learning/) + or + [this tutorial in German](https://jupyter-tutorial.readthedocs.io/de/latest/reproduce/dvc/init.html)! + +- Having great time working with a [tech writer](https://github.com/dashohoxha) + brought to us by the + [Google Season of Docs](https://developers.google.com/season-of-docs) program. + Check out these + [interactive tutorials](https://dvc.org/doc/tutorials/interactive) we’ve + created together. + +- Having hot internal discussion about Discord vs Slack support/community + channels. If you are on the edge like us, have a look at + [this discussion](https://internals.rust-lang.org/t/exploring-new-communication-channels/7859) + in the Rust community, so helpful. + +- Seeing [Dmitry Petrov](https://twitter.com/FullStackML) being really happy one + day: + + https://twitter.com/FullStackML/status/1169403554290814976 + +
+ +We at [DVC.org](https://dvc.org) are so happy every time we discover an article +featuring DVC or addressing one of the burning ML issues we are trying to solve. +Here are some of the links that caught our eye past month: + +- **Continuous Delivery for Machine Learning by + [Danilo Sato](https://twitter.com/dtsato), + [Arif Wider](https://twitter.com/arifwider), + [Christoph Windheuser](https://twitter.com/intellification) and curated by + [Martin Fowler](https://martinfowler.com/).** + +> As Machine Learning techniques continue to evolve and perform more complex +> tasks, so is evolving our knowledge of how to manage and deliver such +> applications to production. By bringing and extending the principles and +> practices from Continuous Delivery, we can better manage the risks of +> releasing changes to Machine Learning applications in a safe and reliable way. + +
+ +## Discord gems + +There are lots of hidden gems in our Discord community discussions. Sometimes +they are scattered all over the channels and hard to track down. + +We are sifting through the issues and discussions and share with you the most +interesting takeaways. + +### Q: I’ve just run a `dvc run` step, and realised I forgot to declare an output file. [Is there a way to add an output file without rerunning the (computationally expensive) step/stage?](https://discordapp.com/channels/485586884165107732/485596304961962003/593743448020877323) + +If you’ve already ran it, you could just open created DVC-file with an editor +and add an entry to the outs field. After that, just run `dvc commit my.dvc` and +it will save the checksums and data without re-running your command. +`dvc run --no-exec` would also work with commit instead of modifying the +DVC-file by hand. + +### Q: [For metric files do I have to use dvc run to set a metric or can I do it some other way?](https://discordapp.com/channels/485586884165107732/485596304961962003/593869598651318282) Can I use metrics functionality without the need to setup and manage DVC cache and remote storage? + +Any file that is under DVC control (e.g. added with `dvc add` or an output in +`dvc run -o`) can be made a metric file with dvc metrics add file. Alternatively +a command `dvc run -M` file makes file a metric without caching it. It means dvc +metrics show can be used while file is still versioned by Git. + +### Q: [Is there a way not to add the full (Azure) connection string to the .dvc/config file that is being checked into Git for using dvc remotes](https://discordapp.com/channels/485586884165107732/485596304961962003/595586670498283520)? I think it’s quite unhealthy to have secrets checked in SCM. + +There are two options — use `AZURE_STORAGE_CONNECTION_STRING` environment +variable or use `--local` flag that will put it into the `.dvc/config.local` +that is added to the `.gitignore`, so you don’t track it with it and so won’t +expose secrets. + +### Q: [I would like to know if it is possible to manage files under DVC whilst keeping them in their original locations (e.g. on a network drive in a given folder structure)](https://discordapp.com/channels/485586884165107732/485596304961962003/601068667131920385)? [If I want to add a large file to be tracked by DVC, and it is in a bucket on S3 or GCS, can I do that without downloading it locally?](https://discordapp.com/channels/485586884165107732/485596304961962003/615278138896941101) + +Yes, you are probably looking for external dependencies and outputs. This is the +[link](https://dvc.org/doc/user-guide/managing-external-data) to the +documentation to start. + +### Q: [How do I setup DVC so that NAS (e.g. Synology) acts as a shared DVC cache?](https://discordapp.com/channels/485586884165107732/485596304961962003/606388040377565215) + +Using NAS (e.g. NFS) is a very common scenario for DVC. In short you use +`dvc cache dir` to setup a cache externally. Set cache type to use symlinks and +enable protected mode. We are preparing a +[document](https://github.com/iterative/dvc.org/blob/31c5d424c6530bb793af69c2af578d2b8a374d02/static/docs/use-cases/shared-storage-on-nfs.md) +how to setup the NFS as a shared cache, but I think it can be applied to any +NAS. + +### Q: So I have some data that is in the hundreds of gigs. [If I enable symlink, hardlink strategy and cache protecting, will DVC automatically choose this strategy over copying when trying to use dvc add](https://discordapp.com/channels/485586884165107732/485596304961962003/608013531010301952)? + +Yes, it will! Here is some clarification. So when you set those settings like +that, `dvc add` data will move data to your cache and then will create a +hardlink from your cache to your workspace. + +Unless your cache directory and your workspace are on different file systems, +move should be instant. Please, find more information +[here](https://dvc.org/doc/user-guide/large-dataset-optimization). + +### Q: My repo’s DVC is “busy and locked” and I’m not sure how it got that way and how to remove/diagnose the lock. [Any suggestions?](https://discordapp.com/channels/485586884165107732/485596304961962003/608392956679815168) + +DVC uses a lock file to prevent running two commands at the same time. The lock +[file](https://dvc.org/doc/user-guide/dvc-files-and-directories#dvc-files-and-directories) +is under the `.dvc` directory. If no DVC commands running and you are still +getting this error it’s safe to remove this file manually to resolve the issue. + +### Q: [I’m trying to understand how does DVC remote add work in case of a local folder and what is the best workflow when data is outside of your project root?](https://discordapp.com/channels/485586884165107732/485596304961962003/611209851757920266) + +When using DVC, in most cases we assume that your data will be somewhere under +project root. There is an option to use so called +[external dependencies](https://dvc.org/doc/user-guide/managing-external-data), +which is data that is usually too big to be stored under your project root, but +if you operate on data that is of some reasonable size, I would recommend +starting with putting data somewhere under project root. Remotes are usually +places where you store your data, but it is DVC task to move your data around. +But if you want to keep your current setup where you will have data in different +place than your project, you will need to refer to data with full paths. So, for +example: + +1. You are in `/home/gabriel/myproject` and you have initialized dvc and git + repository + +2. You have `featurize.py` in your project dir, and want to use data to produce + some features and than `train.py` to train a model. + +3. Run the command: + +```dvc +$ dvc run -d /research_data/myproject/videos \ + -o /research_data/myproject/features \ + python featurize.py +``` + +to tell DVC, that you use `/research_data/myproject/videos` to featurize, and +produce output to your features dir. Note that your code should be aware of +those paths, they can be hardcoded inside `featurize.py`, but point of `dvc run` +is just to tell DVC what artifacts belong to currently defined step of ML +pipeline. + +### Q: When I run `du` command to check how much space DVC project consumes I see that it duplicates/copies data. [It’s very space and time consuming to copy large data files, is there a way to avoid that?](https://discordapp.com/channels/485586884165107732/485596304961962003/613935477896249364) It takes too long to add large files to DVC. + +Yes! You don’t have to copy files with DVC. First of all, there are two reasons +when du can show that it takes double the space to store data under DVC control. +du can be inaccurate when the underlying file system supports reflinks (XFS on +Linux, APFS on Mac, etc). This is actually the best scenario since no copying is +happening and no changes are required to any DVC settings. Second, case means +that copy semantics is used by default. It can be turned off by providing cache +type `symlinks`, `hardlinks`. Please, read more on this +[here](https://dvc.org/doc/user-guide/large-dataset-optimization#file-link-types-for-the-dvc-cache). + +### Q: [How can I detach a file from DVC control?](https://discordapp.com/channels/485586884165107732/485596304961962003/615479227189559323) + +Just removing the corresponding DVC-file and running `dvc gc` after that should +be enough. It’ll stop tracking the data file and clean the local cache that +might still contain it. Note! Don’t forget to run `dvc unprotect` if you use +advanced[ DVC setup with symlinks and hardlinks](https://dvc.org/doc/user-guide/large-dataset-optimization) +(`cache.type` config option is not default). If `dvc gc` behavior is not +granular enough you can manually find the by its cache from the DVC-file in +`.dvc/cache` and remote storage. Learn +[here](https://dvc.org/doc/user-guide/dvc-files-and-directories#structure-of-cache-directory) +how they are organized. + +### Q: [I’m trying to understand if DVC is an appropriate solution for storing data under GDPR requirements.](https://discordapp.com/channels/485586884165107732/485596304961962003/621057268145848340) That means that permanent deletion of files with sensitive data needs to be fully supported. + +Yes, in this sense DVC is not very different from using bare S3, SSH or any +other storage where you can go and just delete data. DVC can give a bit of +overhead to locate a specific file to delete, but otherwise it’s all the same +you will be able to delete any file you want. See more details on how you +retrospectively can edit directories under DVC control +[here](https://discordapp.com/channels/485586884165107732/485596304961962003/621062105524862987). + +
+ +If you have any questions, concerns or ideas, let us know in the comments below +or connect with DVC team [here](https://dvc.org/support). Our +[DMs on Twitter](https://twitter.com/DVCorg) are always open, too. diff --git a/content/blog/2019-12-14-november-19-dvc-heartbeat.md b/content/blog/2019-12-14-november-19-dvc-heartbeat.md new file mode 100644 index 00000000000..17038ea4489 --- /dev/null +++ b/content/blog/2019-12-14-november-19-dvc-heartbeat.md @@ -0,0 +1,278 @@ +--- +title: November ’19 DVC❤️Heartbeat +date: 2019-12-14 +description: | + Co-hosting our first ever meetup, sharing our Hacktoberfest experience, 4K ⭐, + fresh Discord gems and other + news. +descriptionLong: | + Every month we are sharing here our news, findings, interesting reads, + community takeaways, and everything along the way. + + Some of those are related to our brainchild [DVC](https://dvc.org) and its + journey. The others are a collection of exciting stories and ideas centered + around ML best practices and workflow. +picture: ../../static/uploads/images/2019-12-14/post-image.jpeg +pictureComment: + How cool is this handmade swag from our community? We were in tears! +author: ../authors/svetlana_grinchenko.md +commentsUrl: https://discuss.dvc.org/t/november-19-dvc-heartbeat/284 +tags: + - Meetup + - Heartbeat + - Hacktoberfest +--- + +The past few months have been so busy and full of great events! We love how +involved our community is and can’t wait to share more with you: + +- We have organized our very first + [meetup](https://www.meetup.com/San-Francisco-Machine-Learning-Meetup/events/264846847/)! + So many great conversations, new use cases and insights! Many thanks to + [Dan Fischetti](https://www.linkedin.com/in/daniel-fischetti-4a6592bb/) from + [Standard Cognition](https://standard.ai/), who joined our Dmitry Petrov on + stage. Watch the recording here. + + https://youtu.be/RHQXK7EC0jI + +- [Hacktoberfest](https://blog.dataversioncontrol.com/dvc-org-for-hacktoberfest-2019-ce5320151a0c) + was a great exercise for DVC team on many levels and we really enjoyed + supporting new contributors. Kudos to + [Nabanita Dash](https://twitter.com/explorer_07) for organizing a cool + DVC-themed hackathon! + + https://twitter.com/psociiit/status/1185150096792535040 + +- We’ve crossed 4k stars mark on [Github](https://github.com/iterative/dvc)! + +- DVC was participating in the + [Devsprints](https://twitter.com/FossMec/status/1192866498324254720) (Thank + you [Kurian Benoy](https://twitter.com/kurianbenoy2) for the intro!) and we + were happy to jump in and help with some mentoring. + + https://twitter.com/FossMec/status/1192866498324254720 + +_Devsprints participants on our +[Discord](http://dvc.org/chat) channel_ + +- DVC became part of the default + [Homebrew formulae](https://formulae.brew.sh/formula/dvc)! So now you can + install it as easy as `brew install dvc`! + +- We helped 2 aspiring speakers deliver their very first conference talks. + [Kurian Benoy](https://twitter.com/kurianbenoy2/status/1183427495342694401?s=20) + was speaking at [PyconIndia](https://in.pycon.org/2019/) and + [Aman Sharma](https://www.linkedin.com/in/aman-sharma606/) was speaking at + [SciPyIndia](https://scipy.in/2019#speakers). **Supporting speakers is + something we are passionate about and if you ever wanted to give a talk on a + DVC-related topic — we are here to help, just + [let us know](https://dvc.org/support)!** + + https://youtu.be/Ipzf6oQqQpo + +- Our own [Dmitry Petrov](https://twitter.com/FullStackML) went to Europe to + speak at the + [Open Source Summit Europe](https://osseu19.sched.com/speaker/dmitry35) in + Lyon, [Highload++](https://www.highload.ru/moscow/2019/abstracts/6032) in + Moscow and made a stop in in Berlin to co-host a + [meetup](https://www.meetup.com/codecentric-Berlin/events/265555810/) with our + favourite AI folks from [Codecentric](https://www.codecentric.de/)! + +
+ +Here are some of the great pieces of content around DVC and ML ops that we +discovered in October and November: + +- **[Deploy Machine Learning Models with Django](https://www.deploymachinelearning.com/) + by Piotr Płoński.** + +> …building your ML system has a great advantage — it is tailored to your needs. +> It has all features that are needed in your ML system and can be as complex as +> you wish. This tutorial is for readers who are familiar with ML and would like +> to learn how to build ML web services. + +
+ +## Discord gems + +There are lots of hidden gems in our Discord community discussions. Sometimes +they are scattered all over the channels and hard to track down. + +We are sifting through the issues and discussions and share with you the most +interesting takeaways. + +### Q: When you do a `dvc import` you get the state of the data in the original repo at that moment in time from that repo, right? [The overall state of that repo (e.g. Git `commit id` (hash)) is not preserved upon import, right?](https://discordapp.com/channels/485586884165107732/563406153334128681/618744949277458462) + +On the contrary, DVC relies on Git `commit id` (hash) to determine the state of +the data as well as code. Git `commit id` (hash) is saved in DVC file upon +import, data itself is copied/downloaded into DVC repo cache but would not be +pushed to the remote — DVC does not create duplicates. There is a command to +advance/update it when it’s needed — `dvc update`. Git commit hash saved to +provide reproducibility. Even if the source repo `HEAD` has changed your import +stays the same until you run `dvc update` or redo `dvc import`. + +### Q: I’m trying to understand if DVC is an appropriate solution for storing data under GDPR requirements. [That means that permanent deletion of files with sensitive data needs to be fully supported.](https://discordapp.com/channels/485586884165107732/485596304961962003/621057268145848340) + +Yes, in this sense DVC is not very different from using bare S3, SSH or any +other storage where you can go and just delete data. DVC can give a bit of +overhead to locate a specific file to delete, but otherwise it’s all the same +you will be able to delete any file you want. Read more details in +[this discussion](https://discordapp.com/channels/485586884165107732/485596304961962003/621062105524862987). + +### Q: [Is there anyway to get the remote url for specific DVC-files?](https://discordapp.com/channels/485586884165107732/485596304961962003/621591769766821888) Say, I have a DVC-file `foo.png.dvc` — is there a command that will show the remote url, something like `dvc get-remote-url foo.png.dvc` which will return e.g. the Azure url to download. + +There is no special command for that, but if you are using Python, you could use +our API specifically designed for that: + +```python +from dvc.api import get_url + +url = get_url(path, + repo="https://github.com/user/proj", + rev="mybranch") +``` + +so, you could as well use this from CLI as a wrapper command. + +### Q: [Can DVC be integrated with MS Active Directory (AD) authentication for controlling access?](https://discordapp.com/channels/485586884165107732/563406153334128681/619244714071425035) The GDPR requirements would force me to use such a system to manage access. + +Short answer: no (as of the date of publishing this Heartbeat issue) Good news — +it should be very easy to add, so we would welcome a contribution :) Azure has a +connection argument for AD — quick googling shows this +[library](https://github.com/AzureAD/azure-activedirectory-library-for-python), +which is what probably needed. + +### Q: [How do I uninstall DVC from Mac installed as a package?](https://discordapp.com/channels/485586884165107732/485596304961962003/625124341201502209) + +When installing using `plain.pkg` it is a bit tricky to uninstall, so we usually +recommend using things like brew cask instead if you really need the binary +package. Try to run these commands: + +```dvc +$ sudo rm -rf /usr/local/bin/dvc +$ sudo rm -rf /usr/local/lib/dvc +$ sudo pkgutil --forget com.iterative.dvc +``` + +to uninstall the package. + +### Q: We are using SSH remote to store data, but the problem is that everyone within the project has different username on the remote machine and thus we cannot set it in the config file (that is committed to Git). [Is there a way to add just host and path, without the username?](https://discordapp.com/channels/485586884165107732/563406153334128681/619420070111608848) + +Yes, you should use `--local` or `--global` config options to set user per +project or per use machine without sharing (committing) them to Git: + +```dvc +$ dvc remote modify myremote —local user myuser +``` + +or + +```dvc +$ dvc remote modify myremote —global user myuser +``` + +### Q: [I still get the `SSL ERROR` when I try to perform a dvc push with or without `use_ssl = false`](https://discordapp.com/channels/485586884165107732/485596304961962003/628227197592797191)? + +A simple environment variable like this: + +```dvc +$ export AWS_CA_BUNDLE=/path/to/cert/cert.crt dvc push +``` + +should do the trick for now, we plan to fix the ca_bundle option soon. + +### Q: I have just finished a lengthy `dvc repro` and I’m happy with the result. However, I realized that I didn’t specify a dependency which I needed (and obviously is used in the computation). [Can I somehow fix it?](https://discordapp.com/channels/485586884165107732/563406153334128681/620572187841265675) + +Add the dependency to the stage file without rerunning/reproducing the stage. +This is not needed as this additional dependency hasn’t changed. + +You would need to edit the DVC-file. In the deps section add: + +```yaml +-path: not/included/file/path +``` + +and run `dvc commit file.dvc` to save changes w/o running the pipeline again. +See an example +[here](https://discordapp.com/channels/485586884165107732/563406153334128681/620641530075414570). + +### Q: For some reason [we need to always specify the remote name when doing a `dvc push`](https://discordapp.com/channels/485586884165107732/485596304961962003/629704961868955648) e.g., `dvc push -r upstream` as opposed to `dvc push` (mind no additional arguments). + +You can mark a “default” remote: + +```dvc +$ dvc remote add -d remote /path/to/my/main/remote +``` + +then, `dvc push` (and other commands like `dvc pull`) will know to push to the +default + +### Q: [If I want stage B to run after stage A, but the stage A has no output, can I specify A’s DVC-file as B’s dependency?](https://discordapp.com/channels/485586884165107732/563406153334128681/620715145374466048) + +No, at least at the time of publishing this. You could use a phony output +though. E.g. make the stage A output some dummy file and make B depend on it. +Please, consider creating or upvoting a relevant issue on our Github if you’d +this to be implemented. + +### Q: I’m just getting started with DVC, but I’d like to use it for multiple developers to access the data and share models and code. [I do own the server, but I’m not sure how to use DVC with SSH remote?](https://discordapp.com/channels/485586884165107732/563406153334128681/598867829785362452) + +Please, refer to +[this answer](https://discuss.dvc.org/t/how-do-i-use-dvc-with-ssh-remote/279/2) +on the DVC forum and check the documentation for the +[`dvc remote add`](https://dvc.org/doc/command-reference/remote/add) and +[`dvc remote modify`](https://dvc.org/doc/command-reference/remote/modify) +commands to see more options and details. + +
+ +If you have any questions, concerns or ideas, let us know in the comments below +or connect with DVC team [here](https://dvc.org/support). Our +[DMs on Twitter](https://twitter.com/DVCorg) are always open, too. diff --git a/content/blog/2020-01-17-january-20-dvc-heartbeat.md b/content/blog/2020-01-17-january-20-dvc-heartbeat.md new file mode 100644 index 00000000000..88459d325ab --- /dev/null +++ b/content/blog/2020-01-17-january-20-dvc-heartbeat.md @@ -0,0 +1,145 @@ +--- +title: January '20 DVC❤️Heartbeat +date: 2020-01-17 +description: | + Reaching 100 contributors, PyData LA, and more news from the DVC community. +descriptionLong: | + Every month we share news, findings, interesting reads, community takeaways, + and everything else along the way. Some of those are related to our brainchild + [DVC](https://dvc.org) and its journey. The others are a collection of + exciting stories and ideas centered around ML best practices and workflow. +picture: ../../static/uploads/images/2020-01-17/DVC_chalk_donuts.png +pictureComment: We spread the joys of version control and donuts at PyData LA. +author: ../authors/elle_obrien.md +commentsUrl: https://discuss.dvc.org/t/january-20-dvc-heartbeat/314 +tags: + - Heartbeat + - PyData +--- + +Welcome to the New Year! Time for a recap of the last few weeks of activity in +the DVC community. + +## News + +We were honored to be named a [Project of the Year](https://ods.ai/awards/2019/) +by Open Data Science, Russia's largest community of data scientists and machine +learning practitioners. Check out our ⭐️incredibly shiny trophy⭐️! + +https://twitter.com/DVCorg/status/1209544709930016768 + +DVC hit **100 individual contributors** on Github! To celebrate our +100th contributor, [Vera Sativa](https://github.com/verasativa/), we +sent her \$500 to use on any educational opportunity and her own DeeVee (that's +our rainbow owl). We also awarded educational mini-grants to two of DVC's +biggest contributors, [Vít Novotný](https://twitter.com/tweetiko), and +[David Příhoda](https://twitter.com/david_prihoda). + +_Vera (center, flashing a +peace sign) thanked us with this lovely picture of DeeVee and her team, +[Odd Industries](https://odd.co/en/). They are making some extremely neat tools +for construction teams using computer vision._ + +**We were at PyData LA!** Our fearless leader +[Dmitry gave a talk](https://www.youtube.com/watch?v=7Wsd6V0k4Oc) and we set up +a busy booth to meet with the Pythonistas of Los Angeles. It was a cold and +blustery day, but visitors kept showing up to our semi-outdoor booth. We're sure +they came for the open source version control and not the donuts. + + + _The DVC team and PyData +volunteers who heroically staffed our booth in the rain._ + +Our engineer and technical writer Jorge reported: + +> We were super happy to meet all kinds of data professionals and enthusiasts in +> several fields who are learning and adopting DVC with their teams – including +> several working with privacy-sensitive medical records, very cool! + +
+ +## From the community + +Here are some rumblings from the machine learning (ML) and data science +community that got us talking. + +**A machine learning software wishlist.** Computer scientist and writer +[Chip Huyen](https://twitter.com/chipro) tweeted about her ML software wishlist +and kicked off a big community discussion. + +https://twitter.com/chipro/status/1202815757593108480 + +Her tweet resonated with a lot of practitioners, who were eager to discuss the +solutions they'd tried. Among the many thoughtful replies and recommendations, +we were thrilled to see DVC mentioned. + +https://twitter.com/kristijan_ivanc/status/1202879739716870144 + +If you haven't already, definitely check out Chip's +[thread](https://twitter.com/chipro/status/1202815757593108480), and follow her +on Twitter for more excllent, accessible content about ML engineering. We're +thinking hard about these ideas and hope the discussion continues on- and +offline. + +**A gentle intro to DVC for data scientists.** Scientist +[Elle O'Brien](https://twitter.com/andronovhopf) published a code walkthrough +about using DVC to make an image classification project more reproducible. +Specifically, the blog is a case study about version control when a dataset +grows over time. If you're looking for a DVC tutorial geared for data +scientists, this might be up your alley. + +
_Skills required:_ Python, Git, experience with APIs
+ _Difficulty rating:_ Beginner-Medium
+ +2. **Introducing parallelism to DVC.** One of DVC’s features is the ability to + create pipelines, linking data repositories with code to process data, train + models, and evaluate model metrics. Once a DVC pipeline is created, the + pipeline can be shared and re-run in a systematic and entirely reproducible + way. Currently, DVC executes pipelines sequentially, even though some steps + may be run in parallel (such as data preprocessing). We would like to support + parallelization for pipeline steps specified by the user. Furthermore, we’ll + need to support building flags into DVC commands that specify the level of + parallelization (CPU, GPU or memory).
_Skills required:_ + Python, Git. Some experience with parallelization and/or scientific computing + would be helpful but not required.
_Difficulty rating:_ Advanced +
+ +3. **Developing use cases for data registries and ML model zoos.** A new DVC + functionality that we’re particularly excited about is `summon`, a method + that can turn remotely-hosted machine learning artifacts such as datasets, + trained models, and more into objects in the user’s local environment (such + as a Jupyter notebook). This is a foundation for creating data catalogs of + data-frames and machine learning model zoos on top of Git repositories and + cloud storages (like GCS or S3). We need to identify and implement model zoos + (think PyTorch Hub, the Caffe Model Zoo, or the TensorFlow DeepLab Model Zoo) + and data registries for types that are not supported by DVC yet. Currently, + we’ve tested `summon` with PyTorch image segmentation models and Pandas + dataframes. We’re looking for students to explore other possible use cases. +
_Skills required:_ Python, Git, and some machine learning or + data science experience
_Difficulty rating:_ Beginner-Medium
+ +4. **Continuous delivery for JetBrains TeamCity.** Continuous integration and + continuous delivery (CI/CD) for ML projects is an area where we see + [DVC make a big impact](https://martinfowler.com/articles/cd4ml.html)- + specifically, by delivering datasets and ML models into CI/CD pipelines. + While there are many cases when DVC is used inside GitHub Actions and GitLab + CI, you will be transferring this experience to another type of CI/CD system, + [JetBrains TeamCity](https://www.jetbrains.com/teamcity/). We're working to + integrate DVC's model and dataset versioning into TeamCity's CI/CD toolkit. + This project would be ideal for a student looking to explore the growing + field of MLOps, an offshoot of DevOps with the specifics of ML projects at + the center.
_Skills required:_ Python, Git, bash scripting. It + would be nice, but not necessary, to have some experience with CI/CD tools + and developer workflow automation.
_Difficulty rating:_ + Medium-Advanced
+ +5. **DVC performance testing framework.** Performance is a core value of DVC. We + will be creating a performance monitoring and testing framework where new + scenarios (e.g., unit testing)can be populated. The framework should reflect + all performance improvements and degradations for each of the DVC releases. + It would be especially compelling if testing could be integrated with our + GitHub workflow (CI/CD). This is a great opportunity for a student to learn + about DVC and versioning in-depth and contribute to its stability.
+
_Skills required:_ Python, Git, bash scripting.
_Difficulty + rating:_ Medium-Advanced
+ +## If you'd like to apply + +Please refer to the +[Google Summer of Code](https://summerofcode.withgoogle.com/) application guides +for specifics of the program. Students looking to know more about DVC, and our +worldwide community of contributors, will learn most by visiting our +[Discord channel](https://dvc.org/chat), +[GitHub repository](https://github.com/iterative/dvc), and +[Forum](https://discuss.dvc.org/). We are available to discuss project proposals +from interested students and can be reached by [email](mailto:support@dvc.org) +or on our Discord channel. diff --git a/content/blog/2020-02-10-february-20-dvc-heartbeat.md b/content/blog/2020-02-10-february-20-dvc-heartbeat.md new file mode 100644 index 00000000000..900165890da --- /dev/null +++ b/content/blog/2020-02-10-february-20-dvc-heartbeat.md @@ -0,0 +1,149 @@ +--- +title: February '20 DVC❤️Heartbeat +date: 2020-02-10 +description: | + DVC talks around the world, + new team members, and full-stack machine learning. +descriptionLong: | + Every month we share news, findings, interesting reads, community takeaways, + and everything else along the way. + + Look here for updates about [DVC](https://dvc.org), our journey as a startup, + projects by our users and big ideas about best practices in ML and data + science. +picture: ../../static/uploads/images/2020-02-10/heartbeat_black.png +pictureComment: + Just in time for Valentine's day, here's a seasonally-relevant DVC pipeline. +author: ../authors/elle_obrien.md +commentsUrl: https://discuss.dvc.org/t/dvc-heartbeat-feburary-20/318 +tags: + - Heartbeat + - Continuous Integration +--- + +Welcome to the February Heartbeat! This month's featured image is a DVC pipeline +[created by one of our users](https://medium.com/nlp-trend-and-review-en/use-dvc-to-version-control-ml-dl-models-bef61dbfe477), +which _we_ think resembles a valentine. Here are some more highlights from our +team and our community: + +## News + +**Our team is growing!** In early January, DVC gained two new folks: engineer +[Saugat Pachhai](https://github.com/skshetry) and data scientist +[Elle O'Brien](https://twitter.com/andronovhopf). Saugat, based in Nepal, will +be contributing to core DVC. Elle (that's me!), currently in San Francisco, will +be leading data science projects and outreach with DVC. + +We're **gearing up for a spring full of talks** about DVC projects, including +new up-and-coming features for data cataloging and continuous integration. Here +are just a few events that have been added to our schedule: + +
+ +### More resources + +You may want to check out a few more efforts to get at r/AmItheAsshole from a +data-scientific perspective, including +[topic modeling](https://medium.com/@tom.gonda/what-does-reddit-argue-about-28432b11ea26), +[visualizing voting patterns](http://www.nathancunn.com/2019-04-04-am-i-the-asshole/) +and +[growth of the subreddit](https://twitter.com/felipehoffa/status/1223278090958209025), +and +[classification](https://www.informatik.hu-berlin.de/de/forschung/gebiete/wbi/teaching/studienDiplomArbeiten/finished/2019/expose_fletcher.pdf) +with [deep learning](https://github.com/amr-amr/am-i-the-asshole). With a +dataset this rich, there’s much more to be investigated, including continuing to +refine these existing methods. And there’s almost certainly room to push the +state of the art in asshole detection! + +If you're interested in learning more about using Reddit data, check out +[pushshift.io](https://pushshift.io/), a database that contains basically all of +Reddit's content (so why make this dataset? I wanted to remove some of the +barriers to analyzing text from r/AmItheAsshole by providing an +already-processed and cleaned version of the data that can be downloaded with a +line of code; pushshift takes some work). You might use pushshift's API and/or +praw to augment this dataset in some way- perhaps to compare activity in this +subreddit with another, or broader patterns on Reddit. diff --git a/content/blog/2020-02-19-february-20-community-gems.md b/content/blog/2020-02-19-february-20-community-gems.md new file mode 100644 index 00000000000..abbc241846d --- /dev/null +++ b/content/blog/2020-02-19-february-20-community-gems.md @@ -0,0 +1,153 @@ +--- +title: February '20 Community Gems +date: 2020-02-19 +description: | + Great discussions and technical Q&A's from our users. +descriptionLong: | + Look here every month for great discussions and technical Q&A's from our users + and core development team. +picture: ../../static/uploads/images/2020-02-19/feb20_gems_header_gr.png +author: ../authors/elle_obrien.md +commentsUrl: https://discuss.dvc.org/t/feb-20-community-gems/330 +tags: + - Discord + - Google Drive + - Azure + - Gems + - Homebrew +--- + +## Discord gems + +Welcome to the Februrary roundup of useful, intriguing, and good-to-know +discussions going on with DVC users and developers. Let's dive right in with +some questions from our Discord channel. + +### Q: [If I have multiple outputs from a DVC pipeline and only want to checkout one, what command would I run?](https://discordapp.com/channels/485586884165107732/563406153334128681/670233820326264843) + +By defult, `dvc checkout` is written for a +[Git-like experience](https://dvc.org/doc/command-reference/checkout), meaning +that it will sync your local workspace with all the model files, dependencies, +and outputs specified by a project's `.dvc` files. If you only want to access +one artifact from the project, you can do this with +`dvc checkout
+ +Thanks, [@cyberomin](https://twitter.com/cyberomin)! + +Elsewhere on the internet, DVC made the cut in a much-shared blog, +[Five Interesting Data Engineering Projects](https://medium.com/@squarecog/five-interesting-data-engineering-projects-48ffb9c9c501) +by [Dmitry Ryaboy](https://twitter.com/squarecog) (VP of Engineering at biotech +startup Zymergen, and formerly Twitter). Dmitry wrote: + +> To be honest, I’m a bit of a skeptic on “git for data” and various automated +> data / workflow versioning schemes: various approaches I’ve seen in the past +> were either too partial to be useful, or required too drastic a change in how +> data scientists worked to get a realistic chance at adoption. So I ignored, or +> even explicitly avoided, checking DVC out as the buzz grew. I’ve finally +> checked it out and… it looks like maybe this has legs? Metrics tied to +> branches / versions are a great feature. Tying the idea of git-like braches to +> training multiple models makes the value prop clear. The implementation, using +> Git for code and datafile index storage, while leveraging scalable data stores +> for data, and trying to reduce overall storage cost by being clever about +> reuse, looks sane. A lot of what they have to say in +> https://dvc.org/doc/understanding-dvc rings true. + +Check out the full blog here: + +We give tools like Slack and Zoom a lot of credit for making remote work possible, and I think Git and every hosted DVC system should equally get the same credit. Imagine life for a second without version control. Think about that.
— Celestine (@cyberomin) February 1, 2020
+
+ ${
+ attrs.image
+ ? `${escape(attrs.title)}
+${escape(attrs.description)}
+ ${escape(attrs.link)}
+
+ })
+
`
+ : ``
+ }
+
+
+
+ {picture ? (
+
+ )
+}
+
+export const query = graphql`
+ fragment FeedPost on MarkdownRemark {
+ id
+ timeToRead
+ fields {
+ slug
+ }
+ frontmatter {
+ date(formatString: "MMM DD, YYYY")
+ title
+ description
+ descriptionLong
+ picture {
+ childImageSharp {
+ big: fluid(
+ maxWidth: 650
+ maxHeight: 450
+ cropFocus: CENTER
+ quality: 90
+ ) {
+ ...GatsbyImageSharpFluid_withWebp
+ }
+ small: fluid(
+ maxWidth: 300
+ maxHeight: 250
+ cropFocus: CENTER
+ quality: 90
+ ) {
+ ...GatsbyImageSharpFluid_withWebp
+ }
+ }
+ }
+ author {
+ childMarkdownRemark {
+ frontmatter {
+ name
+ avatar {
+ childImageSharp {
+ fixed(width: 40, height: 40, quality: 50, cropFocus: CENTER) {
+ ...GatsbyImageSharpFixed_withWebp
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+ }
+`
+
+export default BlogFeedItem
diff --git a/src/components/BlogFeed/Item/placeholder.svg b/src/components/BlogFeed/Item/placeholder.svg
new file mode 100644
index 00000000000..7b6992815b1
--- /dev/null
+++ b/src/components/BlogFeed/Item/placeholder.svg
@@ -0,0 +1,8 @@
+
diff --git a/src/components/BlogFeed/Item/styles.module.css b/src/components/BlogFeed/Item/styles.module.css
new file mode 100644
index 00000000000..5419843e35a
--- /dev/null
+++ b/src/components/BlogFeed/Item/styles.module.css
@@ -0,0 +1,187 @@
+.wrapper {
+ position: relative;
+ width: 300px;
+ height: 500px;
+ margin: 0 50px 50px 0;
+ background-color: var(--color-light-blue);
+
+ @media (--md-scr) {
+ &.big {
+ width: 650px;
+ height: auto;
+ }
+ }
+
+ @media (--sm-scr) {
+ width: auto;
+ height: auto;
+ margin: 0;
+
+ &.big {
+ width: auto;
+ }
+
+ & + & {
+ margin-top: 10px;
+ }
+ }
+
+ @media (--lg-scr) {
+ &.big {
+ width: 1005px;
+ height: 450px;
+ }
+ }
+}
+
+.pictureLink {
+ @mixin active;
+
+ display: block;
+
+ &:hover,
+ &:focus {
+ outline: none;
+ opacity: 0.7;
+ }
+}
+
+.picture {
+ display: block;
+ width: 300px;
+ height: 250px;
+ background-color: #dee8ed;
+
+ @media (--md-scr) {
+ .big & {
+ width: 650px;
+ height: 450px;
+ }
+ }
+
+ @media (--sm-scr) {
+ width: auto;
+ height: auto;
+
+ .big & {
+ width: auto;
+ height: auto;
+ }
+
+ .placeholder & {
+ display: none;
+ }
+ }
+
+ @media (--lg-scr) {
+ .big & {
+ float: left;
+ width: 650px;
+ height: 450px;
+ margin-right: 50px;
+ }
+ }
+}
+
+.body {
+ position: relative;
+ overflow: hidden;
+ padding: 10px 20px;
+ max-height: 166px;
+
+ &.overflown::after {
+ content: '';
+ position: absolute;
+ right: 0;
+ bottom: 0;
+ left: 0;
+ height: 30px;
+ background: linear-gradient(
+ 0deg,
+ rgba(238, 244, 248, 1) 0%,
+ rgba(238, 244, 248, 0) 100%
+ );
+ }
+
+ @media (--md-scr) {
+ .big & {
+ padding-top: 30px;
+ }
+ }
+
+ @media (--sm-scr) {
+ .placeholder & {
+ padding-top: 30px;
+ }
+ }
+
+ @media (--lg-scr) {
+ .big & {
+ max-height: 314px;
+ padding: 30px;
+ }
+ }
+}
+
+.title {
+ @mixin h3-desktop;
+ @mixin active;
+ @mixin focus;
+ @mixin hover;
+
+ display: block;
+ text-decoration: none;
+ color: var(--color-black);
+
+ @media (--sm-scr) {
+ @mixin h3-mobile;
+ }
+
+ @media (--lg-scr) {
+ .big & {
+ @mixin h2-desktop;
+ }
+ }
+
+ &:hover {
+ opacity: 0.7;
+ }
+}
+
+.description {
+ @mixin text-secondary;
+
+ margin-top: 10px;
+ color: var(--color-gray);
+
+ @media (--lg-scr) {
+ .big & {
+ margin-top: 40px;
+ }
+ }
+}
+
+.meta {
+ position: absolute;
+ right: 20px;
+ bottom: 15px;
+ left: 20px;
+
+ @media (--md-scr) {
+ .big & {
+ position: static;
+ padding: 0 30px 15px;
+ }
+ }
+
+ @media (--sm-scr) {
+ position: static;
+ padding: 0 30px 15px;
+ }
+
+ @media (--lg-scr) {
+ .big & {
+ left: 700px;
+ }
+ }
+}
diff --git a/src/components/BlogFeed/index.tsx b/src/components/BlogFeed/index.tsx
new file mode 100644
index 00000000000..7d01d08e7a7
--- /dev/null
+++ b/src/components/BlogFeed/index.tsx
@@ -0,0 +1,67 @@
+import { graphql } from 'gatsby'
+
+import React from 'react'
+
+import cn from 'classnames'
+
+import Paginator, { IPageInfo } from '../Paginator'
+import BlogFeedItem, { IBlogFeedPostData } from './Item'
+
+import styles from './styles.module.css'
+
+export interface IBlogFeedPostList {
+ edges: Array<{
+ node: IBlogFeedPostData
+ }>
+}
+
+interface IBlogFeedProps {
+ feedPostList: IBlogFeedPostList
+ bigFirst?: boolean
+ header: React.ReactNode
+ leadParagraph?: React.ReactNode
+ pageInfo: IPageInfo
+}
+
+const BlogFeed: React.SFC
+
+ {title}
+
+
+ {description}
+
+
+
+
+ )
+}
+
+export const query = graphql`
+ fragment FeedPostList on MarkdownRemarkConnection {
+ edges {
+ node {
+ ...FeedPost
+ }
+ }
+ }
+`
+
+export default BlogFeed
diff --git a/src/components/BlogFeed/styles.module.css b/src/components/BlogFeed/styles.module.css
new file mode 100644
index 00000000000..aef9be33022
--- /dev/null
+++ b/src/components/BlogFeed/styles.module.css
@@ -0,0 +1,63 @@
+.wrapper {
+ overflow: hidden;
+ max-width: 1005px;
+ margin: 0 auto;
+
+ @media (--md-scr) {
+ max-width: 650px;
+ }
+
+ @media (--sm-scr) {
+ padding: 0 15px;
+ }
+}
+
+.meta {
+ margin: 20px 0 40px;
+
+ @media (--sm-scr) {
+ max-width: auto;
+ margin: 30px;
+ }
+}
+
+.metaSlim {
+ max-width: 650px;
+}
+
+.header {
+ @media (--md-scr) {
+ @mixin h1-mobile;
+ }
+
+ @media (--lg-scr) {
+ @mixin h1-desktop;
+ }
+
+ margin: 0;
+}
+
+.lead {
+ @mixin text-secondary;
+
+ margin-top: 10px;
+ color: var(--color-gray);
+}
+
+.posts {
+ display: flex;
+ flex-wrap: wrap;
+
+ @media (--md-scr) {
+ width: 700px;
+ }
+
+ @media (--sm-scr) {
+ display: block;
+ width: auto;
+ }
+
+ @media (--lg-scr) {
+ width: 1050px;
+ }
+}
diff --git a/src/components/BlogFeedMeta/index.tsx b/src/components/BlogFeedMeta/index.tsx
new file mode 100644
index 00000000000..6faae1de2e5
--- /dev/null
+++ b/src/components/BlogFeedMeta/index.tsx
@@ -0,0 +1,54 @@
+import Image, { FixedObject } from 'gatsby-image'
+import React from 'react'
+
+import { pluralizeComments } from '../../utils/i18n'
+
+import styles from './styles.module.css'
+
+interface IBlogFeedMetaProps {
+ avatar: {
+ childImageSharp: {
+ fixed: FixedObject
+ }
+ }
+ commentsUrl?: string
+ commentsCount?: number
+ date: string
+ name: string
+ timeToRead: string
+}
+
+const BlogFeedMeta: React.SFC
+
+ {header}
+ {leadParagraph &&{leadParagraph}
}
+
+ {edges.map(({ node }, index) => (
+
+
+
+ )
+}
+
+export default BlogFeedMeta
diff --git a/src/components/BlogFeedMeta/styles.module.css b/src/components/BlogFeedMeta/styles.module.css
new file mode 100644
index 00000000000..e78afebeff3
--- /dev/null
+++ b/src/components/BlogFeedMeta/styles.module.css
@@ -0,0 +1,43 @@
+.wrapper {
+ position: relative;
+ display: flex;
+ flex-wrap: wrap;
+ align-items: center;
+ min-height: 44px;
+ padding-left: 40px;
+}
+
+.avatar {
+ position: absolute !important;
+ top: 50%;
+ left: 0;
+ margin-top: -20px;
+ border-radius: 20px;
+}
+
+.list {
+ overflow: hidden;
+ margin: 0 0 0 10px;
+ padding: 0;
+}
+
+.item {
+ @mixin text-secondary;
+
+ position: relative;
+ display: inline-block;
+ margin-right: 14px;
+ white-space: nowrap;
+ line-height: 20px;
+ color: var(--color-gray);
+
+ &::before {
+ content: '• ';
+ position: absolute;
+ right: 100%;
+ }
+}
+
+.link {
+ @mixin link;
+}
diff --git a/src/components/BlogHome/index.tsx b/src/components/BlogHome/index.tsx
new file mode 100644
index 00000000000..3d71b7087a2
--- /dev/null
+++ b/src/components/BlogHome/index.tsx
@@ -0,0 +1,35 @@
+import React from 'react'
+
+import { IPageInfo } from '../Paginator'
+import PageContent from '../PageContent'
+import BlogFeed, { IBlogFeedPostList } from '../BlogFeed'
+import Subscribe from '../Subscribe'
+
+interface IBlogHomeProps {
+ posts: IBlogFeedPostList
+ pageInfo: IPageInfo
+}
+
+const BlogHome: React.SFC-
+
- {name} +
- + {date} • {timeToRead} min read + + {commentsUrl && typeof commentsCount === 'number' && ( +
- + + {pluralizeComments(commentsCount)} + + + )} +
+
+ )
+}
+
+export default HeroPic
diff --git a/src/components/BlogPost/HeroPic/styles.module.css b/src/components/BlogPost/HeroPic/styles.module.css
new file mode 100644
index 00000000000..e7a1b3e855a
--- /dev/null
+++ b/src/components/BlogPost/HeroPic/styles.module.css
@@ -0,0 +1,70 @@
+.pictureWrapper {
+ display: flex;
+ position: relative;
+ flex-direction: column;
+
+ @media (--md-scr) {
+ margin: 10px 15px 0;
+ }
+
+ @media (--lg-scr) {
+ &::before {
+ content: '';
+ position: absolute;
+ top: 0;
+ right: 0;
+ bottom: 50%;
+ left: 0;
+ background-color: var(--color-light-blue);
+ }
+ }
+
+ @media (--lg-scr) {
+ align-items: center;
+ justify-content: center;
+ }
+}
+
+.picture {
+ width: 100%;
+
+ /* z-index: 1000; */
+ position: relative;
+ overflow: hidden;
+
+ @media (--lg-scr) {
+ max-width: 850px;
+ }
+}
+
+.pictureBackground {
+ position: absolute;
+ width: 110%;
+ height: 110%;
+ background-repeat: no-repeat;
+ background-size: cover;
+ background-position: center;
+ filter: blur(5px);
+ background-color: #fff;
+}
+
+.pictureComment {
+ @mixin text-secondary;
+
+ margin-top: 5px;
+ font-style: italic;
+ text-align: center;
+ color: var(--color-gray);
+
+ & p {
+ margin: 0;
+
+ + p {
+ margin-top: 5px;
+ }
+ }
+
+ & a {
+ color: var(--color-gray);
+ }
+}
diff --git a/src/components/BlogPost/Markdown/index.tsx b/src/components/BlogPost/Markdown/index.tsx
new file mode 100644
index 00000000000..683bd584072
--- /dev/null
+++ b/src/components/BlogPost/Markdown/index.tsx
@@ -0,0 +1,18 @@
+import React from 'react'
+
+import styles from './styles.module.css'
+
+interface IMarkdownProps {
+ html: string
+}
+
+function Markdown({ html }: IMarkdownProps) {
+ return (
+
+ )
+}
+
+export default Markdown
diff --git a/src/components/BlogPost/Markdown/styles.module.css b/src/components/BlogPost/Markdown/styles.module.css
new file mode 100644
index 00000000000..f305d57c84d
--- /dev/null
+++ b/src/components/BlogPost/Markdown/styles.module.css
@@ -0,0 +1,512 @@
+.wrapper {
+ @media (--md-scr) {
+ @mixin text-mobile;
+ }
+
+ @media (--lg-scr) {
+ @mixin text-desktop;
+ }
+
+ & *:first-child {
+ margin-top: 0;
+ }
+
+ & *:last-child {
+ margin-bottom: 0;
+ }
+
+ :global(.gatsby-highlight),
+ :global(.gatsby-resp-iframe-wrapper),
+ :global(.gatsby-resp-image-wrapper),
+ :global(.gatsby-resp-image-wrapper) + em,
+ hr,
+ pre,
+ table {
+ @media (--md-scr) {
+ margin: 10px 0;
+ }
+
+ @media (--md-scr) {
+ margin: 30px 0;
+ }
+
+ @media (--lg-scr) {
+ margin: 40px 0;
+ }
+ }
+
+ p {
+ margin: 25px 0;
+
+ img {
+ display: block;
+ width: 100%;
+ }
+ }
+
+ :global(.gatsby-resp-image-wrapper) + em {
+ @mixin text-secondary;
+
+ display: block;
+ text-align: center;
+ color: var(--color-gray);
+
+ @media (--md-scr) {
+ margin-top: -30px;
+ }
+
+ @media (--lg-scr) {
+ margin-top: -40px;
+ }
+ }
+
+ :global(.gatsby-resp-image-link) {
+ @mixin active;
+
+ &:hover,
+ &:focus {
+ outline: none;
+ opacity: 0.7;
+ }
+ }
+
+ h1 {
+ @media (--md-scr) {
+ @mixin h1-mobile;
+
+ margin: 45px 0 15px;
+ }
+
+ @media (--lg-scr) {
+ @mixin h1-desktop;
+
+ margin: 60px 0 25px;
+ }
+ }
+
+ h2 {
+ @media (--md-scr) {
+ @mixin h2-mobile;
+
+ margin: 30px 0 15px;
+ }
+
+ @media (--lg-scr) {
+ @mixin h2-desktop;
+
+ margin: 40px 0 25px;
+ }
+ }
+
+ h3 {
+ margin: 60px 0 15px;
+
+ @media (--md-scr) {
+ @mixin h3-mobile;
+ }
+
+ @media (--lg-scr) {
+ @mixin h3-desktop;
+ }
+ }
+
+ blockquote:not(:global(.twitter-tweet)) {
+ margin: 30px 0;
+ padding-left: 25px;
+ position: relative;
+ font-size: 0.92em;
+ font-style: italic;
+
+ em {
+ font-style: normal;
+ }
+
+ &::before {
+ content: '';
+ position: absolute;
+ top: 5px;
+ bottom: -3px;
+ left: 0;
+ width: 5px;
+ border-radius: 2.5px;
+ background: var(--color-black);
+ }
+ }
+
+ :not(pre) > code,
+ kbd,
+ samp {
+ padding: 0 5px;
+ border-radius: 4px;
+ font-style: inherit;
+ line-height: 1.6em;
+ font-size: 0.74em;
+ color: var(--color-black);
+ background: var(--color-light-blue);
+ }
+
+ h1,
+ h2,
+ h3 {
+ code {
+ line-height: 1.8em;
+ font-size: 0.84em;
+ }
+ }
+
+ ol,
+ ul {
+ margin: 30px 0;
+ padding: 0;
+ }
+
+ li {
+ margin: 15px 0;
+ padding-left: 30px;
+ }
+
+ ul > li {
+ position: relative;
+ list-style: none;
+
+ &::before {
+ content: '•';
+ color: var(--color-black);
+ position: absolute;
+ margin-left: -30px;
+ }
+ }
+
+ ol {
+ counter-reset: ol;
+ }
+
+ ol > li {
+ counter-increment: ol;
+ position: relative;
+ list-style: none;
+
+ &::before {
+ content: counter(ol) '.';
+ position: absolute;
+ margin-left: -30px;
+ color: var(--color-black);
+
+ @media (--md-scr) {
+ top: 1px;
+ font-size: 16px;
+ }
+
+ @media (--lg-scr) {
+ top: 2px;
+ font-size: 20px;
+ }
+ }
+ }
+
+ hr {
+ border: none;
+ text-align: center;
+ font-weight: 400;
+
+ &::before {
+ content: '...';
+ letter-spacing: 0.6em;
+ font-size: 28px;
+ }
+ }
+
+ abbr,
+ acronym {
+ border-bottom: 1px dotted var(--color-gray);
+ cursor: help;
+ }
+
+ abbr[title] {
+ text-decoration: none;
+ }
+
+ a {
+ @mixin link;
+
+ word-wrap: break-word;
+
+ code {
+ color: inherit;
+ }
+ }
+
+ pre {
+ box-sizing: border-box;
+ overflow: scroll;
+ max-width: 100%;
+ padding: 15px;
+ border-radius: 4px;
+ background: var(--color-light-blue);
+ }
+
+ table {
+ @mixin text-diminished;
+
+ border-collapse: collapse;
+ margin-left: auto;
+ margin-right: auto;
+ background: var(--color-light-blue);
+ border-radius: 4px;
+ }
+
+ thead {
+ text-align: left;
+ }
+
+ tr:last-child {
+ td {
+ border: none;
+ }
+ }
+
+ td {
+ border-bottom: 1px solid var(--color-light-gray);
+ font-feature-settings: tnum;
+ padding: 10px 15px 10px;
+ }
+
+ th {
+ font-feature-settings: tnum;
+ padding: 10px 15px 10px;
+ background: var(--color-lighter-blue);
+
+ &:first-child {
+ border-radius: 4px 0 0 0;
+ }
+
+ &:last-child {
+ border-radius: 0 4px 0 0;
+ }
+ }
+
+ th:first-child,
+ td:first-child {
+ padding-left: 25px;
+ }
+
+ th:last-child,
+ td:last-child {
+ padding-right: 25px;
+ }
+
+ /* prism theme */
+
+ /* based on coy theme */
+
+ pre[class*='language-'] {
+ background: #40354d;
+
+ /* this code fixes hilghlight with scroll */
+ code[class*='language-'] {
+ float: left;
+ min-width: 100%;
+ padding: 0;
+ color: #a0a0a0;
+ font-size: 0.74em;
+ line-height: 1.6em;
+ }
+
+ /* DVC's command outputs are a different color */
+ code[class*='language-dvc'] {
+ color: #ccc;
+ }
+
+ :global {
+ .gatsby-highlight-code-line {
+ display: block;
+ margin: 0 -15px;
+ padding: 0 15px 0 13px;
+ border-left: 2px solid #55436a;
+ background: #55436a;
+ }
+
+ .language-dvc {
+ font-weight: normal;
+
+ .token.line {
+ font-weight: bold;
+ color: #a0a0a0;
+ }
+
+ .token.comment {
+ font-weight: normal;
+ color: #a0a0a0;
+ }
+ }
+
+ .token.input {
+ user-select: none;
+ }
+
+ .token.comment,
+ .token.block-comment,
+ .token.prolog,
+ .token.doctype,
+ .token.cdata {
+ color: #999;
+ }
+
+ .token.url,
+ .token.constant,
+ .token.operator,
+ .token.punctuation {
+ color: #a0a0a0;
+ }
+
+ .token.property,
+ .token.tag,
+ .token.boolean,
+ .token.function-name,
+ .token.symbol,
+ .token.deleted {
+ color: #4badd2;
+ }
+
+ .token.function {
+ color: #ae41bb;
+ }
+
+ .token.number,
+ .token.attr-name,
+ .token.string,
+ .token.char,
+ .token.builtin,
+ .token.inserted {
+ color: #219161;
+ }
+
+ .token.entity,
+ .token.variable {
+ color: #a67f59;
+ }
+
+ .token.class-name {
+ color: #0086b3;
+ }
+
+ .token.dvc {
+ color: #56b1d0;
+ }
+
+ .token.git {
+ color: #e9836e;
+ }
+
+ .token.command,
+ .token.selector,
+ .token.atrule,
+ .token.attr-value,
+ .token.keyword {
+ color: #e4b872;
+ }
+
+ .token.regex,
+ .token.important {
+ color: #b68;
+ }
+
+ .token.parameter {
+ color: #a0a0a0;
+ }
+
+ .token.function-variable {
+ color: #7ece42;
+ }
+
+ .token.important {
+ font-weight: normal;
+ }
+
+ .token.bold {
+ font-weight: bold;
+ }
+
+ .token.italic {
+ font-style: italic;
+ }
+
+ .token.entity {
+ cursor: help;
+ }
+
+ .token.namespace {
+ opacity: 0.7;
+ }
+ }
+ }
+
+ :global {
+ .elp-content-holder {
+ display: block;
+ border: 1px solid var(--color-gray);
+
+ .external-link-preview {
+ color: var(--color-black);
+ display: flex;
+ align-items: center;
+ min-height: 160px;
+
+ @media (--xs-scr) {
+ flex-direction: column-reverse;
+ }
+ }
+
+ .elp-description-holder {
+ flex-grow: 1;
+ padding: 15px 20px;
+ align-self: stretch;
+ }
+
+ .elp-title {
+ font-size: 20px;
+ margin-bottom: 0.5rem;
+ line-height: 1.25;
+ }
+
+ .elp-description {
+ font-size: 18px;
+ line-height: 1.5;
+ }
+
+ .elp-link {
+ font-size: 16px;
+ color: var(--color-gray);
+ }
+
+ .elp-image-holder {
+ width: 160px;
+ flex-shrink: 0;
+ align-self: stretch;
+ position: relative;
+
+ @media (--xs-scr) {
+ width: 100%;
+ }
+ }
+
+ img {
+ position: absolute;
+ object-fit: cover;
+ display: block;
+ width: 100%;
+ height: 100%;
+ }
+ }
+ }
+
+ :global(.twitter-tweet) {
+ margin: 0 auto;
+ }
+
+ :global {
+ .gist td {
+ border-bottom: none;
+ }
+ }
+}
diff --git a/src/components/BlogPost/Share/icons/facebook.svg b/src/components/BlogPost/Share/icons/facebook.svg
new file mode 100644
index 00000000000..9c52237d23e
--- /dev/null
+++ b/src/components/BlogPost/Share/icons/facebook.svg
@@ -0,0 +1,4 @@
+
\ No newline at end of file
diff --git a/src/components/BlogPost/Share/icons/subscribe.svg b/src/components/BlogPost/Share/icons/subscribe.svg
new file mode 100644
index 00000000000..87f0adb4eee
--- /dev/null
+++ b/src/components/BlogPost/Share/icons/subscribe.svg
@@ -0,0 +1,4 @@
+
\ No newline at end of file
diff --git a/src/components/BlogPost/Share/icons/twitter.svg b/src/components/BlogPost/Share/icons/twitter.svg
new file mode 100644
index 00000000000..d07652127bb
--- /dev/null
+++ b/src/components/BlogPost/Share/icons/twitter.svg
@@ -0,0 +1,4 @@
+
\ No newline at end of file
diff --git a/src/components/BlogPost/Share/index.tsx b/src/components/BlogPost/Share/index.tsx
new file mode 100644
index 00000000000..de3ffabbd29
--- /dev/null
+++ b/src/components/BlogPost/Share/index.tsx
@@ -0,0 +1,93 @@
+import { graphql, useStaticQuery } from 'gatsby'
+import React, { useCallback } from 'react'
+
+import Tooltip from '../Tooltip'
+
+import { ReactComponent as Facebook } from './icons/facebook.svg'
+import { ReactComponent as Subscribe } from './icons/subscribe.svg'
+import { ReactComponent as Twitter } from './icons/twitter.svg'
+
+import styles from './styles.module.css'
+
+function openWindow(e: React.MouseEvent, href: string) {
+ e.preventDefault()
+
+ window.open(
+ href,
+ '',
+ 'menubar=no,toolbar=no,resizable=yes,scrollbars=yes,height=300,width=600'
+ )
+}
+
+interface IShareProps {
+ className?: string
+ text: string
+ slug: string
+}
+
+function Share({ className, text, slug }: IShareProps) {
+ const {
+ site: {
+ siteMetadata: { siteUrl }
+ }
+ } = useStaticQuery(
+ graphql`
+ query {
+ site {
+ siteMetadata {
+ siteUrl
+ }
+ }
+ }
+ `
+ )
+
+ const encodedURL = encodeURI(`${siteUrl}/${slug}`)
+
+ const facebookHref = `https://www.facebook.com/sharer/sharer.php?u=${encodedURL}`
+ const twitterHref = `https://twitter.com/share?url=${encodedURL}&via=DVCorg&text=${text}`
+
+ const onClickFacebook = useCallback(
+ (e: React.MouseEvent) => openWindow(e, facebookHref),
+ [facebookHref]
+ )
+
+ const onClickTwitter = useCallback(
+ (e: React.MouseEvent) => openWindow(e, twitterHref),
+ [twitterHref]
+ )
+
+ return (
+
+ )
+}
+
+export default Share
diff --git a/src/components/BlogPost/Share/styles.module.css b/src/components/BlogPost/Share/styles.module.css
new file mode 100644
index 00000000000..b93b58b5c57
--- /dev/null
+++ b/src/components/BlogPost/Share/styles.module.css
@@ -0,0 +1,18 @@
+.link {
+ @mixin active;
+
+ position: relative;
+ display: block;
+ width: 40px;
+ height: 40px;
+ margin: 5px;
+
+ &:hover {
+ opacity: 0.7;
+ }
+
+ &:focus {
+ outline: none;
+ opacity: 0.7;
+ }
+}
diff --git a/src/components/BlogPost/Tooltip/index.tsx b/src/components/BlogPost/Tooltip/index.tsx
new file mode 100644
index 00000000000..967ce55de1c
--- /dev/null
+++ b/src/components/BlogPost/Tooltip/index.tsx
@@ -0,0 +1,70 @@
+/* Used https://reacttraining.com/reach-ui/tooltip as the base */
+
+import Portal from '@reach/portal'
+import {
+ TooltipPopup,
+ TooltipProps,
+ useTooltip,
+ Position
+} from '@reach/tooltip'
+
+import React from 'react'
+
+import styles from './styles.module.css'
+
+const centered: Position = (triggerRect, tooltipRect) => {
+ if (!triggerRect || !tooltipRect) {
+ return { left: 0, top: 0 }
+ }
+
+ const triggerCenter = triggerRect.left + triggerRect.width / 2
+ const left = triggerCenter - tooltipRect.width / 2
+ const maxLeft = window.innerWidth - tooltipRect.width - 2
+
+ const result = {
+ left: Math.min(Math.max(2, left), maxLeft) + window.scrollX,
+ top: triggerRect.top - 7 - tooltipRect.height + window.scrollY
+ }
+
+ return result
+}
+
+const portalStyle = (triggerRect: DOMRect | null) => {
+ if (!triggerRect) {
+ return { left: 0, top: 0 }
+ }
+
+ return {
+ left: triggerRect
+ ? triggerRect.left - 4 + triggerRect.width / 2
+ : undefined,
+ top: triggerRect ? triggerRect.top + window.scrollY - 7 : undefined
+ }
+}
+
+function ModifiedTooltip({ children, label, ariaLabel }: TooltipProps) {
+ const [trigger, tooltip] = useTooltip()
+
+ const { isVisible, triggerRect } = tooltip
+
+ return (
+ <>
+ {React.cloneElement(children as React.ReactElement, trigger)}
+
+ {isVisible && (
+
+
+ {pictureComment && (
+
+ )}
+
+
+
+
+
+ {picture && (
+
+
+ {title}
+ {descriptionLong ? ( + + ) : ( +{description}
+ )}
+
+
+ {tags && (
+
+ {tags.map(tag => (
+
+ {tag}
+
+ ))}
+
+ )}
+ {commentsUrl && counterLoaded && (
+
+
+ Discuss this post
+
+
+ {pluralizeComments(commentsCount)}
+
+
+ )}
+