Modularize main script for scraping different years websites

[josix]

The script main.py is used for scraping PyConTW 2016-2020 websites. Currently this script contains lots of global variables and shared functions which will increase the difficulty to keep flexible for scraping the official website in the future. It is hard to maintain/develop. It will be great if we could make this script more structural like separating different year parsing detail into different handler or module, keep the same crawling processing in a base class, etc. Any ideas about this enhancement are welcome.

[Darkborderman]

Hello @josix I have some suggestion about this issue.

• Select different year scrapper using dict and/or split each year scrap into function

Pseudo code:

try:
    SCRAP_YEAR[year]()
except KeyError:
    logger.error("Specified year does not exist!")

Pseudo structure:

/common/    <- sharing modules between scrappers
   /scrap.py  <- used to store web scrap function like `getcssimg` 
   /dataio.py <- used to store file IO functions like `writefile`
/websites/
   /year2018.py <- Scrap function for 2018
   /year2019.py <- Scrap function for 2019
   /utilities.py <- Scrap handler or base class, etc. 
/main.py <- Main entrypoint

• Encapsulate some check method (like path[6:8] == "zh" to get_lang(str))

• Using typing library and add function documentation

About global variables like PYCON_URL, maybe pass it via parameter or base class construction.

[Darkborderman]

I'd like to pick up the issue if you're willing to assign it to me. Thanks!

[josix]

It looks GREAT! I would be appreciated if you could help on this.

[josix]

I guess we also need to reformat the code or add some linter to make the script conform PEP8 in the project. Introducing some reformat tools like black, isortor linter like pylint to this project will be helpful.

[Darkborderman]

I guess we also need to reformat the code or add some linter to make the script conform PEP8 in the project. Introducing some reformat tools like black, isortor linter like pylint to this project will be helpful.

Sure. I'll add those packages and resolve this issue with #22 .

[Lee-W]

If you're interested in following the convention from mail-handler, maybe you can give https://github.com/Lee-W/cookiecutter-python-template a try. It comes with all the tools you mention.

[Darkborderman]

If you're interested in following the convention from mail-handler, maybe you can give https://github.com/Lee-W/cookiecutter-python-template a try. It comes with all the tools you mention.

Thanks! I'll adopt some Coding style & testing packages from it.

[Lee-W]

In addition to that, please try cruft instead of using cookie-cutter directly. It's a tool that can help us get updates from the template easier.

[josix]

I think we could leave this issue simpler just for handling the modularity of the codebase. I'll create another two issues including improving coding style by introducing linter/reformatter and adding more tests for checking the reliability of the code.

[Darkborderman]

I think we could leave this issue simpler just for handling the modularity of the codebase. I'll create another two issues including improving coding style by introducing linter/reformatter and adding more tests for checking the reliability of the code.

+1 for this, this issue can be split into smaller issues.