Computer vision model reuse / DataLabs

[metazool]

@cmtso referred a researcher to me who has compound issues working with computer vision models within DataLabs. We have an intro chat later to find out more about the use case, the data etc. I'm not sure what to expect but anticipate something like

• advice and support on "just enough python" to do ML workflows and reuse the output in R
• questions about parallel / GPU workloads in DataLabs which I don't feel equipped to answer
• potential feature enhancements (still think there is a lot of consolidation work to do on the platform first)

I'll update this with any notes, @albags is in the invite

See also https://github.com/NERC-CEH/datalab

[metazool]

We talked about a project that has set up a small camera network in an area that is also doing physical sampling of seeds and saplings. The aim is to observe "phenophases" of deposited material and complement lab-based sampling with visual sampling

Common ground with other projects: image preprocessing, managing cloud object storage, exploration of suitable computer vision techniques
Probable differences: stronger emphasis on timeseries change detection, issues with variable lighting conditions and discriminating image background which can't be disturbed

Researcher support needs: setting up python virtual environments, some pair programming to consolidate python skills needed to work with ML/CV toolkits, getting into a git-based workflow

Some high-level consultancy on ML approaches from University of Edinburgh via https://homepages.inf.ed.ac.uk/omacaod/

[mattjbr123]

The "Common ground" and "Probable differences" summaries are a really nice and helpful idea!

[mattfry-ceh]

Spotted this and thought I should mention we have a large dataset of "phenocam" images from COSMOS-UK and a processing pipeline (non ML) and a range of use cases for manual checking of these, and labelled datasets for typical issues in these use cases, some of which could probably be addressed by CV approaches. Examples are that the images are checked to see if they contain people before publishing them so they don't cause privacy issues, they look for cows or other livestock as these may affect readings or cause damage, they look for snow on the ground, and even just checking if the images are obscured by rain so they don't include in the processing (which effectively produces an RGB time series for a masked area of the field). All these require manual checking, so someone is spending half a day a week flicking through images. Labelled datasets might make it a useful exemplar. Technology will be used in FDRI as well. Examples here but full dataset available: https://cosmos.ceh.ac.uk/cosmos-uk-videos-and-animations

[metazool]

Thank you for this link and references @mattfry-ceh - it looks like another use case for vector search of embeddings from BioCLIP that would offer motivation to set up a reusable pipeline, outwith a notebook environment, for running experiments like this...

[metazool]

BioCLIP - A Vision Foundation Model for the Tree of Life came up in an email thread (via @mattfry-ceh ) and is worth linking here. Fined-tuned CLIP/ViT - "We curate and release TreeOfLife-10M (the largest and most diverse available dataset of biology images), train BioCLIP, rigorously benchmark our approach on diverse fine-grained biology classification tasks"

• Good potential candidate for an integration with DataLabs
• Possibly of direct interest to this project @albags , if it goes in a Segment Anything direction because you can take a SAM+CLIP approach that looks like this (computationally heavy, though)

[metazool]

See also NERC-CEH/plankton_ml#10 (BioCLIP for plankton embeddings) and this related really nice blog post about taking the same approach to aerial imagery with vector embeddings from off the shelf models https://blog.rtwilson.com/searching-an-aerial-photo-with-text-queries-a-demo-and-how-it-works/

[metazool]

https://github.com/agentmorris/MegaDetector - MILA / MegaDetector via @KatrionaGoldmann at Turing Inst

"...helping conservation biologists spend less time doing boring things with camera trap images." - object detection model that originally came out of AI for Earth / Planetary Computer. Could be a shoe-in for prototyping; would be really interested to explore how the group in Biodiversity are reusing it.

[metazool]

@albags and myself and the researcher who's coding on this project ran a 2 hour mob programming session yesterday, which was interesting. Some of this needs split out into other places (additions to the diagrams of requests coming to the RSE group and how we route them #19 ; and documentation PRs for problems we've helped solve more than once). A quick summary:

• The work's moved off DataLabs onto an internally provisioned VM, for one specific labelling application that won't run there
• A lot of the session was taken up with python environment setup and dependency management. This is normal in my experience, my previous team would typically dedicate at least one 2-hour session to mutual support for getting everyone on the same page.
• Much of the work is confidence-building, reassurance, tuning in to the interpretation of error messages in a new language, exchange of tips and tricks (like commandline tab-complete and Ctrl-R for autocomplete)
• We didn't get as far as source control and git workflow but I would not want to leave it longer, and again it's a matter of reassurance and tips (like git add -p), there's already a very good degree of basic competency
• On the image machine learning specifics, the instincts are good, the questions are about model choice and documentation levels. It took a code read to find the configuration parameters, and some domain experience to recognise the dataset format. This could be obviated by providing a few strong recommendations along with canned environments, possibly a dedicated project to keep those in, and I'd like to see that happen as a cross-organisational effort (cough Turing)

Followups here (for me anyway) are

• Walkthrough documentation for a recommended path for setting up an environment with VSCode Remote SSH and miniconda. I've ran through this with two people in as many weeks. Ideally it's documentation that we offer to the Scientific Computing group and they keep in their reference material.
• Reconstruct the writeup I used to have of mob/pair programming in a research setting, adaptations and recommendations from experience
• Find the right place in the process diagrams to add this service/experience, talk about how we time budget for future needs
• Set a time for another session on this project, leave schedules allowing, which definitely has a goal of work in source control coming out of it

[metazool]

A quick update note on how this is going!

@albags and I are holding mob programming sessions with @elirai about once every three weeks, in practise more code walkthrough and consultancy than much hands-on work.

The work's progressing really fast (almost completely without our involvement!) and currently at a stage of trialling NNI (new to me) for hyperparameter sweeping. It's all on JASMIN now, performance limitations on the available on-prem VM.

• Data and metadata management for image collections - ease of getting data into cloud storage and reusing it in experiments; ease of linking processes applied to intermediate forms of data ("provenance"). The object store API is intended to address the first part of this. DVC could be a shoe-in for the latter part - @matthewcoole has been evaluating it recently for LLM fine-tuning workflows. DVC has got s3 storage integration which could work with the s3 storage on JASMIN or the planned in-house one. The trick here is introducing this in a way that has the right kind of affordance, doesn't feel like too much overhead for a researcher in the early stages of an experiment
• Experiment tracking for ML model training. I've happily used MLFlow for this, both standalone and as it's integrated behind Databricks / Azure Machine Learning. DVC has its own moral equivalent, Weights and Biases could work well in a single-developer, local-install setting. There are too many options, this is a matter both of making a strong recommendation and producing example-driven documentation. We'd all benefit from a hands-on deep dive into DVC, post-RSECon
• Python packaging for re-use. There's a project repository with a naturally-emerging practise of breaking down experimental notebooks into sets of functions. Thanks @lewis-chambers for making the python-template repo used in the bigger Digital Research Infrastructure projects publically available!

[albags]

[heart] Alba Gomez Segura reacted to your message:

…

________________________________ From: Jo Walsh ***@***.***> Sent: Wednesday, August 28, 2024 5:02:40 AM To: NERC-CEH/rse_group ***@***.***> Cc: Alba Gomez Segura ***@***.***>; Mention ***@***.***> Subject: Re: [NERC-CEH/rse_group] Computer vision model reuse / DataLabs (Discussion #12) CAUTION: This email originated from outside of the organisation. Do not click links or open attachments unless you recognise the senders email address and know the content is safe. A quick update note on how this is going! @albags<https://github.com/albags> and I are holding mob programming sessions with @elirai<https://github.com/elirai> about once every three weeks, in practise more code walkthrough and consultancy than much hands-on work. The work's progressing really fast (almost completely without our involvement!) and currently at a stage of trialling NNI<https://nni.readthedocs.io/en/stable/> (new to me) for hyperparameter sweeping. It's all on JASMIN now, performance limitations on the available on-prem VM. * Data and metadata management for image collections - ease of getting data into cloud storage and reusing it in experiments; ease of linking processes applied to intermediate forms of data ("provenance"). The object store API<https://github.com/NERC-CEH/object_store_api/> is intended to address the first part of this. DVC<https://dvc.org/> could be a shoe-in for the latter part - @matthewcoole<https://github.com/matthewcoole> has been evaluating it recently for LLM fine-tuning workflows. DVC has got s3 storage integration<https://dvc.org/doc/user-guide/data-management/remote-storage/amazon-s3> which could work with the s3 storage on JASMIN<https://s3-portal.jasmin.ac.uk/login> or the planned in-house one. The trick here is introducing this in a way that has the right kind of affordance, doesn't feel like too much overhead for a researcher in the early stages of an experiment * Experiment tracking for ML model training. I've happily used MLFlow<https://mlflow.org/> for this, both standalone and as it's integrated behind Databricks / Azure Machine Learning. DVC has its own moral equivalent, Weights and Biases<https://wandb.ai/site> could work well in a single-developer, local-install setting. There are too many options, this is a matter both of making a strong recommendation and producing example-driven documentation. We'd all benefit from a hands-on deep dive into DVC, post-RSECon * Python packaging for re-use. There's a project repository with a naturally-emerging practise of breaking down experimental notebooks into sets of functions. Thanks @lewis-chambers<https://github.com/lewis-chambers> for making the python-template<https://github.com/NERC-CEH/python-template/> repo used in the bigger Digital Research Infrastructure projects publically available! * * * — Reply to this email directly, view it on GitHub<#12 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AFNXLR4HDGO4P4QHCV6QEW3ZTVKXBAVCNFSM6AAAAABJPXZYUWVHI2DSMVQWIX3LMV43URDJONRXK43TNFXW4Q3PNVWWK3TUHMYTANBXGAZTCMQ>. You are receiving this because you were mentioned.Message ID: ***@***.***> This email and any attachments are intended solely for the named recipients and are confidential. If you are not the intended recipient, please reply to the email to highlight the error and delete this email from your system; you must not use, disclose, copy, or distribute this email or any of its attachments. UK Centre for Ecology & Hydrology (UKCEH) has taken reasonable precautions to minimise risk of this email or any attachments containing viruses or malware, but the recipient should carry out its own virus and malware checks before opening the attachments. UKCEH does not accept any liability for any losses or damages which the recipient may sustain due to presence of any viruses. Opinions, conclusions or other information in this message and attachments that are not related directly to UKCEH business are solely those of the author and do not represent the views of UKCEH. We process your personal data in accordance with our Privacy Notice, available on the UKCEH website. https://www.ceh.ac.uk/privacy-notice Registered office address; Maclean Building Benson Lane, Crowmarsh Gifford, Wallingford, Oxfordshire, United Kingdom, OX10 8BB Companies Registered Name; UK Centre for Ecology & Hydrology Place of Registration; England Registered Company Number; 11314957

[metazool]

We spent about 1.5 hours today looking at how to fit DVC in this project to help with either data management, model training experiment tracking, or both via pipelines. Notes to catch you up @albags - and thanks for popping in to share notes on DVC @matthewcoole

It would be wonderful to get an automated workflow setup going with dvc stages. At the moment we've avoided this, because the training's being run on JASMIN/Orchid via Slurm. What we did today was add dvclive experiment tracking (logging metrics per epoch) into @elirai 's current code, and set off a test run.

The training data is included in the repository; the growing image collection for doing change detection and classification from is living in what looks like the filesystem of a JASMIN Group Workspace

• https://dvc.org/doc/user-guide/pipelines
• https://github.com/iterative/dvc
• https://dvc.org/doc/start/experiments/experiment-tracking

We touched on JASMIN's object storage and the way in which different projects use that. It's added complexity for an early stage project, reaction is useful for planning how to make it easier to get data to and fro, we still haven't picked out a pattern for doing this happily with DVC.

We also touched on off-the-shelf model choice for doing classification of areas picked out as containing changes by the initial model. This probably wants to be a chain or ensemble. It could be LeafNet for an initial prototype. This part of the project isn't ours to get into though, we're here for the wiring and tooling...

[metazool]

Asian Hornet Database - labelling annotation, good test case, gold-ish standard

Project may soon become unfunded