Update robots.txt with new URL endpoints; #504

[Ninjaclasher]

Closes #504.

[codecov-commenter]

Codecov Report

Merging #1427 into master will increase coverage by 0.51%.
The diff coverage is n/a.

@@            Coverage Diff             @@
##           master    #1427      +/-   ##
==========================================
+ Coverage   37.92%   38.44%   +0.51%     
==========================================
  Files         198      201       +3     
  Lines       11337    11398      +61     
==========================================
+ Hits         4300     4382      +82     
+ Misses       7037     7016      -21     

Impacted Files	Coverage Δ
judge/admin/problem.py	57.23% <0.00%> (-2.21%)	⬇️
judge/models/comment.py	38.81% <0.00%> (-0.79%)	⬇️
judge/migrations/0108_bleach_problems.py	0.00% <0.00%> (ø)
judge/jinja2/markdown/bleach_whitelist.py	100.00% <0.00%> (ø)
judge/jinja2/markdown/test_markdown.py	100.00% <0.00%> (ø)
dmoj/settings.py	97.61% <0.00%> (+0.04%)	⬆️
judge/models/problem.py	52.11% <0.00%> (+0.15%)	⬆️
judge/jinja2/markdown/__init__.py	59.66% <0.00%> (+25.35%)	⬆️
judge/jinja2/markdown/math.py	74.46% <0.00%> (+36.17%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c6e7a38...bf04b1f. Read the comment docs.

[Xyene]

Minor nit: we should probably sort these so that checking if a URL is in this file is not O(n) for a human. sort < robots.txt should do that for you; you'd just need to move User-agent back up.

[Xyene]

Thanks!