-
-
Notifications
You must be signed in to change notification settings - Fork 3.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add GitLab repo sync and webhook support #1870
Changes from 10 commits
ef9c792
cf5c47c
2356d84
7113e94
5d7d6c1
f9df35e
e4adb98
da7471f
aec6941
9152bb6
299a2b5
8a10848
97a4fb1
dff4d7d
4fbe6bb
50285fe
e288bf3
6c8d1c3
0cb1ed5
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -9,7 +9,7 @@ worked with push knows, pushing a doc update to your repo and watching it get | |
updated within seconds is an awesome feeling. | ||
|
||
GitHub | ||
--------- | ||
------ | ||
|
||
If your project is hosted on GitHub, you can easily add a hook that will rebuild | ||
your docs whenever you push updates: | ||
|
@@ -27,7 +27,7 @@ If you ever need to manually set the webhook on GitHub, | |
you can point it at ``https://readthedocs.org/github``. | ||
|
||
Bitbucket | ||
----------- | ||
--------- | ||
|
||
If your project is hosted on Bitbucket, you can easily add a hook that will rebuild | ||
your docs whenever you push updates: | ||
|
@@ -40,6 +40,17 @@ your docs whenever you push updates: | |
If you ever need to manually set the webhook on Bitbucket, | ||
you can point it at ``https://readthedocs.org/bitbucket``. | ||
|
||
GitLab | ||
------ | ||
|
||
If your project is hosted on GitLab, you can manually set the webhook on Gitlab and | ||
point it at ``https://readthedocs.org/gitlab``: | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Should this webhook be updated to reflect the new webhook end points added in #2433 ? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So, without a webhook management page, these webhooks aren't completely deprecated yet. Once that feature exists, this page will need to be updated to instead reference that page and drop information on manual set up. Likewise, we should add a deprecation notice on GitHub Services for RTD as well. I'll open an issue outlining the work that will go into this one -- fine to point to this webhook endpoint for now. |
||
|
||
* Click the settings icon for your project | ||
* Select "Webhooks" | ||
* Enter the above URL, select "Push events" and "Enable SSL verification" | ||
* Click "Add Webhook" | ||
|
||
Others | ||
------ | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -10,4 +10,8 @@ | |
getattr(settings, 'OAUTH_BITBUCKET_SERVICE', | ||
'readthedocs.oauth.services.bitbucket.BitbucketService')) | ||
|
||
registry = [GitHubService, BitbucketService] | ||
GitLabService = import_by_path( | ||
getattr(settings, 'OAUTH_GITLAB_SERVICE', | ||
'readthedocs.oauth.services.gitlab.GitLabService')) | ||
|
||
registry = [GitHubService, BitbucketService, GitLabService] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm not sure this is required, @agjohnson can you please check. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is required, we use this pattern to override API actions on private resources on readthedocs.com. |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,235 @@ | ||
"""OAuth utility functions""" | ||
|
||
import logging | ||
import json | ||
import re | ||
|
||
from django.conf import settings | ||
from requests.exceptions import RequestException | ||
from allauth.socialaccount.models import SocialToken | ||
from allauth.socialaccount.providers.gitlab.views import GitLabOAuth2Adapter | ||
from urlparse import urljoin | ||
|
||
from readthedocs.restapi.client import api | ||
|
||
from ..models import RemoteOrganization, RemoteRepository | ||
from .base import Service, DEFAULT_PRIVACY_LEVEL | ||
|
||
|
||
log = logging.getLogger(__name__) | ||
|
||
|
||
class GitLabService(Service): | ||
"""Provider service for GitLab""" | ||
|
||
adapter = GitLabOAuth2Adapter | ||
url_pattern = re.compile(re.escape(adapter.provider_base_url)) | ||
default_avatar = { | ||
'repo': urljoin(settings.MEDIA_URL, 'images/fa-bookmark.svg'), | ||
'org': urljoin(settings.MEDIA_URL, 'images/fa-users.svg'), | ||
} | ||
|
||
def paginate(self, url, **kwargs): | ||
"""Combines return from GitLab pagination. GitLab uses | ||
LinkHeaders, see: http://www.w3.org/wiki/LinkHeader | ||
:param url: start url to get the data from. | ||
:param kwargs: optional parameters passed to .get() method | ||
See http://doc.gitlab.com/ce/api/README.html#pagination | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This URL should be updated to |
||
""" | ||
resp = self.get_session().get(url, data=kwargs) | ||
result = resp.json() | ||
next_url = resp.links.get('next', {}).get('url') | ||
if next_url: | ||
result.extend(self.paginate(next_url, **kwargs)) | ||
return result | ||
|
||
def sync(self): | ||
"""Sync repositories from GitLab API""" | ||
org = None | ||
repos = self.paginate( | ||
u'{url}/api/v3/projects'.format(url=self.adapter.provider_base_url), | ||
per_page=100, | ||
order_by='path', | ||
sort='asc' | ||
) | ||
for repo in repos: | ||
# Skip archived repositories | ||
if repo.get('archived', False): | ||
continue | ||
if not org or org.slug != repo['namespace']['id']: | ||
org = self.create_organization(repo['namespace']) | ||
|
||
self.create_repository(repo, organization=org) | ||
|
||
def create_repository(self, fields, privacy=DEFAULT_PRIVACY_LEVEL, | ||
organization=None): | ||
"""Update or create a repository from GitLab API response | ||
:param fields: dictionary of response data from API | ||
:param privacy: privacy level to support | ||
:param organization: remote organization to associate with | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Organizations are called Groups in GitLab, not sure if that matters or if that'd be confusing for users. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @connorshea, @destroyerofbuilds of course you're both right, but i had to adapt the given api and this api uses There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
|
||
:type organization: RemoteOrganization | ||
:rtype: RemoteRepository | ||
""" | ||
# See: http://doc.gitlab.com/ce/api/projects.html#projects | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. And this URL as well. |
||
repo_is_public = fields['visibility_level'] == 20 | ||
|
||
def is_owned_by(owner_id): | ||
return self.account.extra_data['id'] == owner_id | ||
|
||
if privacy == 'private' or (repo_is_public and privacy == 'public'): | ||
try: | ||
repo = RemoteRepository.objects.get( | ||
full_name=fields['name_with_namespace'], | ||
users=self.user, | ||
account=self.account, | ||
) | ||
except RemoteRepository.DoesNotExist: | ||
repo = RemoteRepository.objects.create( | ||
full_name=fields['name_with_namespace'], | ||
account=self.account, | ||
) | ||
repo.users.add(self.user) | ||
|
||
if repo.organization and repo.organization != organization: | ||
log.debug('Not importing %s because mismatched orgs' % | ||
fields['name']) | ||
return None | ||
else: | ||
repo.organization = organization | ||
repo.name = fields['name'] | ||
repo.full_name = fields['name_with_namespace'] | ||
repo.description = fields['description'] | ||
repo.ssh_url = fields['ssh_url_to_repo'] | ||
repo.html_url = fields['web_url'] | ||
repo.private = not fields['public'] | ||
repo.admin = not repo_is_public | ||
repo.clone_url = (repo.admin and repo.ssh_url or | ||
fields['http_url_to_repo']) | ||
if not repo.admin and 'owner' in fields: | ||
repo.admin = is_owned_by(fields['owner']['id']) | ||
repo.vcs = 'git' | ||
repo.account = self.account | ||
owner = fields.get('owner') or {} | ||
repo.avatar_url = (fields.get('avatar_url') or | ||
owner.get('avatar_url') or | ||
self.default_avatar['repo']) | ||
repo.json = json.dumps(fields) | ||
repo.save() | ||
return repo | ||
else: | ||
log.info( | ||
u'Not importing {0} because mismatched type: public={1}'.format( | ||
fields['name_with_namespace'], | ||
fields['public'], | ||
) | ||
) | ||
|
||
def create_organization(self, fields): | ||
"""Update or create remote organization from GitLab API response | ||
:param fields: dictionary response of data from API | ||
:rtype: RemoteOrganization | ||
""" | ||
try: | ||
organization = RemoteOrganization.objects.get( | ||
slug=fields.get('path'), | ||
users=self.user, | ||
account=self.account, | ||
) | ||
except RemoteOrganization.DoesNotExist: | ||
organization = RemoteOrganization.objects.create( | ||
slug=fields.get('path'), | ||
account=self.account, | ||
) | ||
organization.users.add(self.user) | ||
|
||
organization.name = fields.get('name') | ||
organization.account = self.account | ||
organization.url = u'{url}/{path}'.format( | ||
url=self.adapter.provider_base_url, path=fields.get('path') | ||
) | ||
avatar = fields.get('avatar') or {} | ||
if avatar.get('url'): | ||
organization.avatar_url = u'{url}/{avatar}'.format( | ||
url=self.adapter.provider_base_url, | ||
avatar=avatar.get('url'), | ||
) | ||
else: | ||
organization.avatar_url = self.default_avatar['org'] | ||
organization.json = json.dumps(fields) | ||
organization.save() | ||
return organization | ||
|
||
def setup_webhook(self, project): | ||
"""Set up GitLab project webhook for project | ||
:param project: project to set up webhook for | ||
:type project: Project | ||
:returns: boolean based on webhook set up success | ||
:rtype: bool | ||
""" | ||
session = self.get_session() | ||
|
||
# See: http://doc.gitlab.com/ce/api/projects.html#add-project-hook | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ditto for this one. |
||
data = json.dumps({ | ||
'id': 'readthedocs', | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. maybe we should use There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. indeed, i assume this would fail or at least overwrite the existing webhook if a user set up a webhook with readthedocs.org and readthedocs.com -- or a private instance. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. As stated in the documentation of GitLab, the ID of the project (or the NAMESPACE/PROJECT_NAME of a project) should be used. I've tested it: you can set up multiple webhooks with the same ID. |
||
'push_events': True, | ||
'issues_events': False, | ||
'merge_requests_events': False, | ||
'note_events': False, | ||
'tag_push_events': True, | ||
'url': u'https://{0}/gitlab'.format(settings.PRODUCTION_DOMAIN), | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. In #2433 these webhook endpoints were deprecated. A new handler for GitLab will be required and this should be updated. We can either address this in the PR, or I'm also +1 on filing a new PR to address this change. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Oh, uh, I already implemented this. Nothing to see here. |
||
}) | ||
resp = None | ||
try: | ||
repositories = RemoteRepository.objects.filter( | ||
clone_url=project.vcs_repo().repo_url | ||
) | ||
assert repositories | ||
repo_id = repositories[0].get_serialized()['id'] | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. To create the webhook we need |
||
resp = session.post( | ||
u'{url}/api/v3/projects/{repo_id}/hooks'.format( | ||
url=self.adapter.provider_base_url, | ||
repo_id=repo_id, | ||
), | ||
data=data, | ||
headers={'content-type': 'application/json'} | ||
) | ||
if resp.status_code == 201: | ||
log.info('GitLab webhook creation successful for project: %s', # noqa | ||
project) | ||
return (True, resp) | ||
except (AssertionError, RemoteRepository.DoesNotExist) as ex: | ||
log.error('GitLab remote repository not found', exc_info=ex) | ||
except RequestException as ex: | ||
pass | ||
else: | ||
ex = False | ||
|
||
log.error('GitLab webhook creation failed for project: %s', # noqa | ||
project, exc_info=ex) | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This try block could shortened, it seems like there is extraneous handling of |
||
return (False, resp) | ||
|
||
@classmethod | ||
def get_token_for_project(cls, project, force_local=False): | ||
"""Get access token for project by iterating over project users""" | ||
# TODO why does this only target GitHub? | ||
if not getattr(settings, 'ALLOW_PRIVATE_REPOS', False): | ||
return None | ||
token = None | ||
try: | ||
if getattr(settings, 'DONT_HIT_DB', True) and not force_local: | ||
token = api.project(project.pk).token().get()['token'] | ||
else: | ||
for user in project.users.all(): | ||
tokens = SocialToken.objects.filter( | ||
account__user=user, | ||
app__provider=cls.adapter.provider_id) | ||
if tokens.exists(): | ||
token = tokens[0].token | ||
except Exception: | ||
log.error('Failed to get token for user', exc_info=True) | ||
return token | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not entirely sure this is required, or it could be moved to the base class. https://github.com/rtfd/readthedocs.org/blob/e4958838a512b095d0bc8cdf8617c30cc9e489d4/readthedocs/restapi/views/model_views.py#L83-L90 was only targeting Github before refactoring oauth services. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This should be formatted as GitLab
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
what do you mean here?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
GitLab branding is to always call it
GitLab.com
(though I can't find their public documentation on that at this time).However it's usually referred to as
GitLab
.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
done.