Development (#197)

[johnhbenetech]

• File cluster proof of concept (File cluster proof of concept #162)

• Implement files data-access object

• Migrate to files-dao

• Update server tests

• Fix regression

• Make module naming more ideomatic

• Resolve linting issues

• Support expunging by session scope

• Draft matches extraction

• Filter matches by distance

• Test query matches with cycles

• Test match loading

• Emulate matches pagination

• Update MatchesDAO tests

• Migrate server.api to MatchesDAO

• Update api/matches tests

• Test multiple hops with cycles

• Update file matches page

• Update cluster page to consume new matches format

• Improve graph container responsiveness

• Fix linting issues

• Improve graph responsiveness

• Implement generic loading trigger

• Support matches filtering

• Implement dynamic matches loading

• Fix loading trigger message

• Fix match reducers

• Improve dynamic match loading

• Implement neighbor loading

• Adjust graph style

• Enable zooming

• Enable cluster navigation

• Fix tooltips

• Make edge opacity dynamic

• Make color scheme static

• Handle playback issues (Handle playback issues #165)

• Handle missing file in thumbnail endpoint

• Handle missing file in watch endpoint

• Display playback/loading errors

• Handle file missing in database case

• Use bundled flv.js

By default react-player tries to lazy-load playback SDK from CDN.
But the application must be able play video files when Internet
connection is not available. To solve that we bundle flv.js and
initialize global variable consumed by react-player's FilePlayer.

See https://www.npmjs.com/package/react-player#sdk-overrides
See cookpete/react-player#605 (comment)

• Suppress excessive flv.js error logs (Catch playback issues #149)

• coding style improvements and small refactors

• Fix matches loading (Investigate intermittent failure to load matches #166) (Fix matches loading (#166) #168)

• Support config tags (Support config tags #167)

• Support tags by PathReprStorage

• Support tags by SQLiteReprStorage

• Update extract_features.py to support tags

• Add missing dependency

• Make config tag opaque

• Update extract_features.py sript

• Delete obsolete code

• Update repr storage tests

• Update generate_matches.py script

• Update template_matching.py script

• Update general_tests

• Add missing unit-test dependency

• Optimize module dependencies

• Implement side-by-side match comparison view (Side-by-side match comparison view #155) (Implement side-by-side match comparison view (#155) #169)

• Refactor file cluster page

• Refacto VideoInformationPane

• Make video details elements collapsible

• Move distance element to common package

• Add comparable file component

• Implement reusable file summary

• Implement match file list

• Implement mother file view

• Fetch matched files scenes

• Setup comparison page routing

• Reset video player on file change

• Fix matched file header

• Improve distance style

• Hook up compare button

• Fix match duplication

• Fix linting issues

• Make compare button primary-colored

• added missing dependency

• removed unnecessary files

• additional refactor and linting fixes

• Improve cluster view (Improve cluster view #171)

• Refactor FileSummaryHeader

• Refactor linear list item

• Extract match file id from URL

• Navigate to comparison page on edge click

• Navigate to compare from single match preview

• Improve cluster tooltips

• Implement node highlighting

• Improve link popover

• Fix linting issues

• Add navigation from comparison page (Add 'back to matches' arrow on side by side view #172) (Improve comparison page navigation #177)

• Handle partial processing results (Handle partial processing results #178)

• Handle missing data in backend (Allow for resiliency of API/display if into isn't available #135)

• Test missing data processing

• Fix go back navigation (Fix go back navigation #179)

• Fix go-back navigation

• Preserve filters and search results

Previous file filters and fetched files should not be updated when
we navigate from video-details page to the collection page.

• Handle missing video length

• Fix match category selector

• Ensure initial file loading

• Add match file enumeration & sorting (Add match file enumeration & sorting #181)

• Add match enumeration (Add match enumeration to side by side view #175)

• Sort files by match score

• Handle uninitialized ref

• Add image version option (Add image version option #180)

• Ensure shell is bash

• Refactor setup script

• Select image version during setup (Add dev/prod option in make flow for prebuilt images #97)

• Add make-goal for forced setup update

• Improve scripts output

• Add docker update and purge goals (make rebuild doesn't seem to work as expected #114)

• Use arrow-select for pre-built option

• Add various UI improvements (make "hit box" for cluster viz linkages larger #173, maintain row/card view setting within session #174, Add visual indicator for active filter #176) (Add various UI improvements (#173, #174, #176) #182)

• Handle dense layout

• Add active filters indication (Add visual indicator for active filter #176)

• Preserve file view type during session (maintain row/card view setting within session #174)

• Enlarge links hitbox (make "hit box" for cluster viz linkages larger #173)

• Fix link click handler

• augmented dataset evaluation pipeline

• Modifications to support benchmarking script

• Add cluster API endpoint (Separate cluster and matches endpoints. #183) (Add cluster API endpoint (#183) #184)

• Separate cluster and matches enpoints (Separate cluster and matches endpoints. #183)

• Update server tests

• Test matches endpoint

• Update REST client

• Refacto API client

• Creat generic hook for entity loading

• Consume matches and cluster API separately

• Manage fields inclusion

• Update immediate match loading

• Update comparison page

• Remove trash

• Remove trash

• Refactor ui state management (Refactor frontend state management #185, Investigate issues with cached match loading after API refactor #189) (Refactor ui state management (#185, #189) #190)

• Move helpers to separate modules

• Move file cache to separate package

• Move file matches state to separate package

• Move cluster state to separate package

• Move file-list state to separate package

• Collect reusable prop-types in a public package

• Extract entity fetching logic

• Refactor file-cluster state

Use generic approach to manage file-cluster state.

• Refactor file-matches state

Use generic approach to manage file-matches state.

• Update server client

• Update matches params (Investigate issues with cached match loading after API refactor #189)

• Fix file cluster update

• Disable false-positive linting issue

• Create post_push

• Simplify docker-compose workflow (Simplify docker-compose workflow. #187, Add git revision information to the image metadata. #188) (Simplify docker-compose workflow (#187, #188) #191)

• Simplify docker-compose workflow

Use build and image simultaneously in compose-file. As a result
no shared configuration is needed and all the configuration is
managed by environment variables (located in the .env file).

To use prebuild images user will need to run make pull
To use local images (if production mode is disabled) user will
need to run make build. That's it.

• Commit missing script

• Add revision information to docker images (Add git revision information to the image metadata. #188)

• Benchmarking improvements

• Update and rename post_push to build

• Delete build

• Parse exif general encoded date (Parse exif General_Encoded_Date format #137) (Parse exif general encoded date (#137) #193)

• Update db schema

• Fix media-info output parsing

• Parse exif date-time

• Use date-time on backend

• Update tests

• Use numeric timestamps on frontend

• Handle NaN-representation of missing values

The exif metadata is pre-processed by the pandas.json_normalize
method which replaces any missing value with NaNs. As a result
the float value may appear in place of string.

• Handle pandas missing data representation

pandas.Series heuristically determines type of the underlying data
and tries to represent a missing values according to that data
time. In case of datetime the missing values are represented by
pandas.NaT which is not compatible with SQLAlchemy framework. To
fix that we have to explicitly transform NaTs to Nones.
See https://pandas.pydata.org/pandas-docs/stable/user_guide/missing_data.html#datetimes

• Fix deprecated import

• Fix database URI

• Handle more date columns

• Update video metadata model (Refactor server model / transform.js to line up with deprecation of metadata columns #139) (Update video metadata model (#139) #196)

• Update db schema

• Update db access logic

• Update frontend

• Update server tests

• Update storage tests

• Additional error handling for dealing with missing files

• Added Missing frame level features check

• Quick merge fix

Co-authored-by: Stepan Anokhin [email protected]
Co-authored-by: Felipe Batista [email protected]
Co-authored-by: Stepan Anokhin [email protected]