Task to remove orphan symlinks

[humitos]

This is an initial implementation of this task to remove orpahn symlinks, which should be ran periodically every 1h aprox.

In the ticket, I also suggested that the Edit action doesn't make sense to me and should be reconsidered.

Finally, if we consider to remove the Edit action, this task should be run just once and should be enough.

Closes #3493

[humitos]

A different approach on the implementation, making just one query to database could be like this:

            valid_cnames = set(Domain.objects.all().values_list('domain', flat=True))
            orphan_cnames = set(os.listdir(domain_path)) - valid_cnames
            for cname in orphan_cnames:
                os.unlink(orphan_domain_path)

[agjohnson]

I like this approach, it saves a lot of queries.

[ericholscher]

The issue link here is wrong. Would be good to know more about the background here. I imagine there are places we should fixing this logic in the code too.

[humitos]

@ericholscher I fixed the issue link in the description. There are some context in the issue.

I imagine there are places we should fixing this logic in the code too.

I'm not very familiarized about how symlinks work, but this PR is half of the solution that it's described in the issue.

[agjohnson]

Tests look good! I do prefer the second query you posted, so I'd say 👍 on moving to that query.

[agjohnson]

I like this approach, it saves a lot of queries.

[humitos]

@agjohnson I changed the query and did a quick test locally: all the cnames that comes with the test data (djangokong.com for example) were removed but temporal.humitos.ar (the one that I have been using for .com) was kept there.

[09/Mar/2018 08:15:27] readthedocs.projects.tasks:795[6934]: INFO Unlinking orphan CNAME: /home/humitos/rtfd/code/readthedocs.org/private_cname_project/djangokong.com

Please, take a new a look at this and after merged I will create a new PR under -ops to configure the periodic task.

[agjohnson]

Any reason to not set the task up as a scheduled task in the application? I think we can keep this out of ops.

[humitos]

No strong reason, really. I though we were setup them only in the ops repo as I think we don't have any other scheduled task in this repository. If we have, can you point me where they are? Otherwise, can you give an example (from the web) where/how I should configure this?

[agjohnson]

We have scheduled tasks set up in our corporate repo. The ops tasks are just for tasks that only make sense in production. This is a task that would be useful for a development instance.

[humitos]

I define the task here under CELERYBEAT setting and opened a PR in the ops repo to extend the basic definition from here.

This should be ready to be merged.

[agjohnson]

Whoops! One thing i didn't catch here was that this isn't calling broadcast to all webs. Celery beat only runs one instance per cluster, not on each instance. The task will need to call broadcast instead.

The changes look okay at this point, but the call logic and tests will change with this.

[humitos]

I just pushed the changes requested.

I didn't find any example for celery beat and broadcast. So, I created a new task just to broadcast the one that I already had.

The broadcast function could be a decorator (maybe) in the future if we need to do this multiple times.

[agjohnson]

I resolved conflicts here, going to wait on tests to merge though

[agjohnson]

Same issue with linting. Merging!