Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore:add encode #40

Closed
wants to merge 54 commits into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
54 commits
Select commit Hold shift + click to select a range
c8d81c3
Update archive-pycontw.yml
josix Aug 2, 2021
aae3868
fix: Resolve wrong base url issue
josix Aug 4, 2021
38700f8
Merge pull request #11 from josix/main
josix Aug 4, 2021
3ca8b4c
local switch
mozixreality Aug 22, 2021
2de3448
base_url -> base url
mozixreality Aug 22, 2021
cade367
Update README.md
mozixreality Aug 22, 2021
0e2b9a8
Merge pull request #13 from pycontw/mozix
josix Aug 22, 2021
98e5588
fix(main.py): crawler all page in specific year
allen91wu Aug 26, 2021
0e4d1a4
fix(main.py): crawler sponsorship prospectus page
allen91wu Aug 26, 2021
a9cb450
fix(main.py): remove nav-lang-form method
allen91wu Aug 26, 2021
e8d333a
fix(main.py): locale switch
allen91wu Aug 26, 2021
ca927fe
fix(main.py): fix css-background url
allen91wu Aug 26, 2021
a04ebce
style(main.py): fix coding style
allen91wu Aug 27, 2021
539f3ee
Merge pull request #17 from allen91wu/fix-bug
josix Aug 27, 2021
1793860
Update archive-pycontw.yml
josix Aug 27, 2021
c5f1fbb
fix: Fix 2020/zh-hant/sponsor/sponsor/ page 404 error
josix Aug 27, 2021
0175d2e
feat: Update to scrape favicon
josix Aug 27, 2021
ce0cc14
feat: Remove redundant code in get_assets
josix Aug 28, 2021
d713916
Merge pull request #18 from josix/main
josix Aug 28, 2021
9d3d49a
fix(website_2020): fix crawler of warmup event page
allen91wu Oct 14, 2021
559ef46
style(black): reformat coding style via black
allen91wu Oct 14, 2021
50d281c
Merge pull request #19 from allen91wu/fix-crawler
allen91wu Oct 14, 2021
54cb81d
ci: Add Year action input and validation
josix Oct 22, 2021
5608e7e
ci: Update to use python3.9 and fix wrong working dir
josix Oct 22, 2021
5271427
ci: Avoid clean branch before deploy
josix Oct 22, 2021
54487c9
Merge pull request #20 from josix/update/github_crawling_action
josix Oct 23, 2021
b7aa412
build: Update to pin used packages
josix Oct 24, 2021
16073c5
chore: rm unused Dockerfile & compose yaml files
mattwang44 Oct 24, 2021
fd305df
doc: update README for repo description and script usage
mattwang44 Oct 24, 2021
a75e355
Merge pull request #23 from josix/main
josix Oct 24, 2021
b52b3e5
Delete .DS_Store
josix Oct 24, 2021
e54f8b5
Merge pull request #24 from pycontw/rm-docker
josix Oct 24, 2021
e4a7317
fix: Unify path format by inoking urlparser and prevent keep old base…
josix Oct 25, 2021
b6dba3c
chore: update pipenv serve command name
josix Oct 25, 2021
12d635d
docs: Fix worng introduction for running local server
josix Oct 25, 2021
d1ebba5
Merge pull request #26 from josix/fix/compability-with-sites-not-star…
josix Oct 26, 2021
e049fbe
refactor(cruft): add linter and formater via cruft
allen91wu Oct 27, 2021
c04865b
refactor(cruft): add project description
allen91wu Oct 27, 2021
cbc4383
Merge pull request #28 from pycontw/add-linter-and-formater
allen91wu Oct 27, 2021
9cc9e1a
ci(main-updated.yaml): remove publish-github-page action
allen91wu Oct 27, 2021
325807e
Merge pull request #29 from pycontw/fix-workflow
allen91wu Oct 27, 2021
0b2ec83
ci(main-updated.yaml): change access token
allen91wu Oct 28, 2021
f1e7849
Merge pull request #30 from pycontw/fix-token
josix Oct 28, 2021
6e506ff
bump: version 0.0.1 → 0.1.0
github-actions[bot] Oct 28, 2021
b6756de
refactor: Add basic common strucutre
Darkborderman Nov 1, 2021
7809aae
refactor: Refine dataio functions
Darkborderman Nov 1, 2021
11c277e
refactor: Add crawler website base structure
Darkborderman Nov 1, 2021
b90b544
refactor: Refine common modules
Darkborderman Nov 1, 2021
c6c2eeb
refactor: Refactor crawlers
Darkborderman Nov 2, 2021
3444bcc
refactor: Update main function
Darkborderman Nov 2, 2021
653e3d9
refactor: Refactor crawler classes
Darkborderman Nov 7, 2021
824cc09
Merge pull request #36 from Darkborderman/refactor/modulize
josix Nov 8, 2021
c3592da
bump: version 0.1.0 → 0.1.1
github-actions[bot] Nov 8, 2021
69e555a
chore:add encode
guizaida May 15, 2022
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Binary file removed .DS_Store
Binary file not shown.
3 changes: 3 additions & 0 deletions .bandit
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[bandit]
targets: pycon_archive_past_website
exclude: tests
26 changes: 26 additions & 0 deletions .cruft.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
{
"template": "https://github.com/Lee-W/cookiecutter-python-template",
"commit": "95b9897dc83fc93eee9eab0f11e23bc116172089",
"checkout": null,
"context": {
"cookiecutter": {
"project_name": "pycon_archive_past_website",
"project_slug": "pycon_archive_past_website",
"project_description": "Freeze and archive historical PyConTW official websites as static sites.",
"github_username": "pycontw",
"github_url": "https://github.com/pycontw/pycon_archive_past_website",
"author_name": "PyCon Taiwan",
"author_email": "[email protected]",
"python_version": "3.9",
"dependency_management_tool": "pipenv",
"default_branch": "main",
"use_strict_mypy_config": "n",
"build_pypi_package": "n",
"build_docker_image": "n",
"open_source_license": "MIT license",
"_template_version": "1.1.2",
"_template": "https://github.com/Lee-W/cookiecutter-python-template"
}
},
"directory": null
}
95 changes: 95 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
# Git
.git
.gitignore

# CI
.codeclimate.yml
.travis.yml
.taskcluster.yml

# Docker
docker-compose.yml
.docker

# Byte-compiled / optimized / DLL files
__pycache__/
*/__pycache__/
*/*/__pycache__/
*/*/*/__pycache__/
*.py[cod]
*/*.py[cod]
*/*/*.py[cod]
*/*/*/*.py[cod]

# C extensions
*.so

# Distribution / packaging
.Python
env/
build/
develop-eggs/
dist/
downloads/
eggs/
lib/
lib64/
parts/
sdist/
var/
*.egg-info/
.installed.cfg
*.egg

# PyInstaller
# Usually these files are written by a python script from a template
# before PyInstaller builds the exe, so as to inject date/other infos into it.
*.manifest
*.spec

# Installer logs
pip-log.txt
pip-delete-this-directory.txt

# Unit test / coverage reports
htmlcov/
.tox/
.coverage
.cache
nosetests.xml
coverage.xml

# Translations
*.mo
*.pot

# Django stuff:
*.log

# Sphinx documentation
docs/_build/

# PyBuilder
target/

# Virtual environment
.env/
.venv/
venv/

# PyCharm
.idea

# Python mode for VIM
.ropeproject
*/.ropeproject
*/*/.ropeproject
*/*/*/.ropeproject

# Vim swap files
*.swp
*/*.swp
*/*/*.swp
*/*/*/*.swp

# End of https://github.com/themattrix/python-pypi-template/blob/master/.dockerignore
17 changes: 17 additions & 0 deletions .editorconfig
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
# http://editorconfig.org

root = true

[*]
indent_style = space
indent_size = 4
trim_trailing_whitespace = true
insert_final_newline = true
charset = utf-8
end_of_line = lf

[*.md]
trim_trailing_whitespace = false

[Makefile]
indent_style = tab
39 changes: 18 additions & 21 deletions .github/ISSUE_TEMPLATE/bug_report.md
Original file line number Diff line number Diff line change
@@ -1,38 +1,35 @@
---
name: Bug report
name: 🛠 Bug report
about: Create a report to help us improve
title: ''
labels: ''
assignees: ''

title: "[Bug Report] Good bug title tells us about precise symptom, not about the root cause."
labels: "bug"
assignees: ""
---

**Describe the bug**
A clear and concise description of what the bug is.
## Description
<!-- A clear and concise description of what the bug is. -->

## Python Project version
<!-- x.y.z or commit hash -->

**To Reproduce**
## Steps to Reproduce
<!--
Steps to reproduce the behavior:
1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error
-->

**Expected behavior**
## Expected Behavior
<!--
A clear and concise description of what you expected to happen.

**Screenshots**
If applicable, add screenshots to help explain your problem.
-->

**Desktop (please complete the following information):**
- OS: [e.g. iOS]
- Browser [e.g. chrome, safari]
- Version [e.g. 22]

**Smartphone (please complete the following information):**
- Device: [e.g. iPhone6]
- OS: [e.g. iOS8.1]
- Browser [e.g. stock browser, safari]
- Version [e.g. 22]
## Actual Behavior
<!-- What happens actually so you think this is a bug. -->

**Additional context**
Add any other context about the problem here.
## More Information
18 changes: 18 additions & 0 deletions .github/ISSUE_TEMPLATE/documentation.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
---
name: 📖 Documentation
about: Suggest an improvement for the documentation of this project
title: "[Documentation] Content to be added or fixed"
labels: "documentation"
assignees: ""
---

## Type
* [ ] Content inaccurate
* [ ] Content missing
* [ ] Typo

## URL
<!-- URL to the code we did not clearly describe or the document page where the content is inaccurate -->

## Description
<!-- A clear and concise description of what content should be added or fixed -->
25 changes: 12 additions & 13 deletions .github/ISSUE_TEMPLATE/feature_request.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,19 @@
---
name: Feature request
name: 🚀 Feature request
about: Suggest an idea for this project
title: ''
labels: ''
assignees: ''

title: "[Feature Request] <One feature request per issue>"
labels: ""
assignees: ""
---

**Is your feature request related to a problem? Please describe.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
## Description
<!--A clear and concise description for us to know your idea.-->

**Describe the solution you'd like**
A clear and concise description of what you want to happen.
## Possible Solution
<!--A clear and concise description of what you want to happen.-->

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
## Additional context
<!--Add any other context or screenshots about the feature request here.-->

**Additional context**
Add any other context or screenshots about the feature request here.
## Related Issue
<!--If applicable, add link to existing issue also help us know better.-->
38 changes: 38 additions & 0 deletions .github/pull_request_template.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
<!--(Thanks for sending a pull request! Please fill in the following content to let us know better about this change.)-->

## Types of changes
<!--Please remove the types that does not apply to this change-->

- **Bugfix**
- **New feature**
- **Refactoring**
- **Breaking change** (any change that would cause existing functionality to not work as expected)
- **Documentation Update**
- **Other (please describe)**

## Description
<!--Describe what the change is**-->

## Checklist:
- [ ] Add test cases to all the changes you introduce
- [ ] Run `inv style` locally to ensure all linter checks pass
- [ ] Run `inv test` locally to ensure all test cases pass
- [ ] Run `inv secure` locally to ensure no major vulnerability is introduced
- [ ] Update the documentation if necessary

## Steps to Test This Pull Request
<!--
Steps to reproduce the behavior:
1. ...
2. ...
3. ...
-->

## Expected behavior
<!--A clear and concise description of what you expected to happen-->

## Related Issue
<!--If applicable, reference to the issue related to this pull request.-->

## Additional context
<!--Add any other context or screenshots about the pull request here.-->
27 changes: 14 additions & 13 deletions .github/workflows/archive-pycontw.yml
Original file line number Diff line number Diff line change
Expand Up @@ -3,37 +3,38 @@ name: Deploy PyConTW sites to branch
on:
workflow_dispatch:
inputs:
re-scrape:
description: 'Re-scrape all archived sites'
year:
description: The year of official website that haven't been archived e.g. "2021"
required: true
base_url:
description: 'Dynamic base url for different domain e.g. "/pycon_archive_past_website"'
required: true

jobs:
deploy-pycontw-sites:
deploy-dev-pycontw-sites:
if:
${{github.event.inputs.year > 2020 }}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v2
with:
token: '${{ secrets.ACCESS_TOKEN }}'
token: '${{ secrets.GITHUB_TOKEN }}'
fetch-depth: 0
- name: Set up Python
uses: actions/setup-python@v1
with:
python-version: '3.x'
python-version: '3.9'
- name: Install dependencies
run: |
python -m pip install -U pip pipenv
pipenv install
- name: Crawl Past PyConTW Websites
run: |
pipenv run python3 main.py -y 2016 &
pipenv run python3 main.py -y 2017 &
pipenv run python3 main.py -y 2018 &
pipenv run python3 main.py -y 2019 &
pipenv run python3 main.py -y 2020 &
wait
mkdir build; mv 2016/ 2017/ 2018/ 2019/ 2020/ build/
pipenv run python3 main.py -y ${{github.event.inputs.year}} --base ${{github.event.inputs.base_url}}
mkdir build; mv ${{github.event.inputs.year}}/ build/
- name: Deploy to GitHub Pages
uses: JamesIves/[email protected]
with:
branch: gh-pages
folder: build
folder: build
clean: false
23 changes: 23 additions & 0 deletions .github/workflows/main-updated.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
name: main branch updated

on:
push:
branches:
- main

jobs:
bump-version:
if: "!startsWith(github.event.head_commit.message, 'bump:')"
runs-on: ubuntu-latest
name: "Bump version and create changelog with commitizen"
steps:
- name: Check out
uses: actions/checkout@v2
with:
token: ${{ secrets.GITHUB_TOKEN }}
fetch-depth: 0
- name: Create bump and changelog
uses: commitizen-tools/commitizen-action@master
with:
github_token: ${{ secrets.GITHUB_TOKEN }}
branch: main
Loading