Skip to content

Commit

Permalink
fix(sourcerepo-sync): improve debugability (#2700)
Browse files Browse the repository at this point in the history
Problem:

```
+ poetry run python source_sync.py --kind SourceRepository --project oss-vdb-test --file ../../source_test.yaml --no-dry-run --verbose
Loaded 26 local source repositories
Validated 26 source repositories
Retrieved 26 source repositories from datastore
Traceback (most recent call last):
  File "/workspace/tools/sourcerepo-sync/source_sync.py", line 162, in <module>
    main()
  File "/workspace/tools/sourcerepo-sync/source_sync.py", line 134, in main
    validate_repository(ds_repos, False)
  File "/workspace/tools/sourcerepo-sync/source_sync.py", line 69, in validate_repository
    if 'link' in repo and repo['link'][-1] != '/':
                          ~~~~~~~~~~~~^^^^
TypeError: 'NoneType' object is not subscriptable
```

happening on #2699 was harder to
get to the bottom of than I'd have liked...

Capture some of the knowledge gained as part of debugging this in some
basic service documentation to help the next person
  • Loading branch information
andrewpollock authored Oct 4, 2024
1 parent fe16186 commit 649fd23
Show file tree
Hide file tree
Showing 2 changed files with 32 additions and 0 deletions.
30 changes: 30 additions & 0 deletions tools/sourcerepo-sync/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
# sourcerepo-sync

## What

Synchronise the contents of [`source.yaml`](../../source.yaml) (for Production) and
[`source_test.yaml`](../../source_test.yaml) (for Staging) with Cloud Datastore.

## Why

To reduce the need for unilateral editing of
[`SourceRepository`](https://github.com/google/osv.dev/blob/fe6155f7cfa0e5df0ae1ef20c7b16f5c20bebed1/osv/models.py#L814)
kind contents in Cloud Datastore (with attendant fat-fingering risks) and
generally best-practices around config-as-code and transparency around data
sources.

## How

See [`run_source_update.sh`](run_source_update.sh):

* Uses the `gcr.io/oss-vdb/ci` Docker image
* Validates both [the YAML contents of the pull
request](https://github.com/google/osv.dev/blob/fe6155f7cfa0e5df0ae1ef20c7b16f5c20bebed1/tools/sourcerepo-sync/source_sync.py#L125) *and* [the existing
content in Cloud
Datastore](https://github.com/google/osv.dev/blob/fe6155f7cfa0e5df0ae1ef20c7b16f5c20bebed1/tools/sourcerepo-sync/source_sync.py#L134)

## Operational matters

* Triggered by Cloud Build on pushes of `source.yaml` or `source_test.yaml` to
`master`, using [`source\_build.yaml`](source_build.yaml)
* Can fail non-prominently post-merge
2 changes: 2 additions & 0 deletions tools/sourcerepo-sync/source_sync.py
Original file line number Diff line number Diff line change
Expand Up @@ -60,6 +60,8 @@ def validate_repository(repository, local=True):
"""Check the attributes of the source repo."""
sourcerepo_names = []
for repo in repository:
if args.verbose:
print(f'Validating: {repo}')
if repo['name'] == '' or repo['name'] is None or repo['name'] == 'null':
raise ValueError(f'Empty sourcerepo name in {file}')
if repo['name'] in sourcerepo_names:
Expand Down

0 comments on commit 649fd23

Please sign in to comment.