MagicCardZoom

MagicCardZoom is a browser extension to identify Magic: the Gathering™ cards in videos and streams.

Installation

The extension can be installed directly from the Chrome Web Store.

How does it work?

See this page for an explanation of how it works.

Contributing

Please submit any feedback, issues, or feature requests you have using GitHub issues or this quick feedback link, or by sending an email to [email protected].

Also, you are more than welcome to contribute to this repository! Please first open an issue (or send an email) so we can discuss the approach and I can help you get setup.

Setup

The following instructions guide you through setting up, building and loading the extension from source.

Overview

1. Prerequisites
2. Clone the repository
3. Install dependencies
4. Download images
5. Index images
6. Build the extension
7. Load the extension

Prerequisites

Make sure you have the following installed:

• NodeJS
• Yarn

Clone the repository

git clone [email protected]:arnaudh/magic-card-zoom.git
cd magic-card-zoom/

Install dependencies

yarn

Download images

From here onwards, the cards used to build and run the extension are taken from the formats specified in config.json, using the allowedFormats property. If left empty, all sets found on Scryfall will be used. To speed up development and testing, it is recommended to limit to a single format, e.g. allowedFormats: ["standard-kld"]

The following command downloads all images and necessary metadata from Scryfall (can take around 4 hours for all sets, or 3 minutes for a single Standard).

node src/js/util/download_all.js

This will create the assets/ folder at the root of the repository.

• assets/images/ contains all the card images
• assets/metadata/ contains information about cards and sets

The script can be interrupted, and when re-run it will continue to download images and metadata that it hasn't downloaded yet.

Index images

Indexing is the generation of feature descriptors, i.e. the "card signatures" that are later used to match the video pixels to a specific card. Currently the feature descriptor used is ORB, as provided by the jsfeat library.

node src/js/util/index_images.js

This will create the assets/indexes/ folder containing the indexes of all images, grouped by sets.

Build the extension

node src/js/build/build.js

Note: check there is no WARNING - unable to locate ..., as this means we'd miss out on some indexes.

This will create the build/ folder containing all code and assets required to load & run the extension in the browser.

Load the extension

In Chrome: Menu (⋮) > More Tools > Extensions > Load unpacked > select the build/ folder.

The extension should now be loaded and ready to use.

After making changes to the code, re-build and then click the refresh button on the extensions page. If you have modified the feature descriptors used to identify images, you will need to re-run the indexing before building again.

Adding a new Magic: the Gathering™ set

The following steps need to be done whenever a new set is released.

1. Check if there is a new set available on Scryfall: https://scryfall.com/sets

   Make sure that latest set has all its cards available on scryfall.

2. Edit download_standard_info.js to specify the latest set code:

   const latestMtgSet = 'znr';
   

3. (Optional) delete existing metadata and index files

   trash ./assets/metadata/cards/full/ ./assets/indexes/orb-maxkeypoints200-cardheight70/
   

   Doing this makes sure we download and index the latest list of cards for each set (there may have been new images uploaded on older sets since the last run).

   Can also see discrepancies between metadata and indexes:

   jq -r '.[] | {code,card_count} | join(" ")' assets/metadata/sets/sets.json | sort > metadata.csv
   for set in assets/indexes/orb-maxkeypoints200-cardheight70/*; do indexed_cards=$(grep -o keypoints $set | wc -l | tr -d ' ') && echo $(basename $set .json) $indexed_cards; done | sort > indexes.csv
   join indexes.csv metadata.csv | awk '$2<$3'
   

4. Re-run setup instructions starting from step 4 (Download images etc.)

   There may be some errors here that need to be dealt with manually. For example the parsing of the Standard formats from Wikipedia is prone to issues, so special cases may need to be added in download_standard_info.js.

   Check that the build/ folder loads and the extension works as expected on the new set(s).

5. Regenerate test files

   First set let regenerate = true in mtg_sets_test.js, then re-run unit tests: yarn test --recursive test/unit/. Check that the changes in test/unit/assets/expanded_sets.json make sense (should just add the new sets), then revert to let regenerate = false.

   Then commit the changes:

   git commit -am "Add <name_of_the_set> set"
   

6. Bump the version

   Bump the patch version in src/manifest.json and commit the changes:

   git commit -am "Set version to <X.X.X>"
   

7. Release the new version of the extension on the Webstore

   make package
   

   Confirm the zip file loads and runs as expected.

   Then manually upload the zip file to the Chrome Dev Console.

Test

Run unit tests

yarn test --recursive test/unit/

Run the integration test, which makes sure the extension can be built and loaded in the browser

yarn test test/integration/build_test.js

Benchmark

Identifying cards from pixels is a challenge and depends on many factors such as video resolution, lighting and card obstruction. In order to verify that changes to the algorithms improve the overall accuracy, I have annotated a dataset of YouTube screenshots with the ground truth (i.e. the true card ID). The dataset and benchmark script can be found under test/benchmark/.

Aknowledgements

• Scryfall: comprehensive search engine and API for Magic: the Gathering™ cards
• jsfeat: JavaScript computer vision library
• OpenCV: open source computer vision library with C++, Python and Java interfaces (and JavaScript binding)
• opencv4nodejs: OpenCV bindings for NodeJS
• emscripten: toolchain for compiling C/C++ to asm.js and WebAssembly
• ORB: fast rotation-invariant feature descriptor
• PyImageSearch: computer vision tutorials in Python using OpenCV

Disclaimer

MagicCardZoom is unofficial Fan Content permitted under the Fan Content Policy. Not approved/endorsed by Wizards. Portions of the materials used are property of Wizards of the Coast. ©Wizards of the Coast LLC.