Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

IE 11 is not supported. For an optimal experience visit our site on another browser #208

Open
thetesttoy opened this issue Oct 27, 2024 · 1 comment
Assignees
Labels
bug Something isn't working question Further information is requested

Comments

@thetesttoy
Copy link

When scraping the ranking of movies on Douban, the message "IE 11 is not supported. For an optimal experience, visit our site on another browser" appears. I also encountered the same problem when scraping data from other websites. Could you please tell me how to solve this?this is my code

from crawl4ai.extraction_strategy import JsonCssExtractionStrategy
import json
from crawl4ai.chunking_strategy import RegexChunking
from crawl4ai import AsyncWebCrawler
import asyncio

async def main():
    # 定义提取模式
    schema = {
        "name": "Douban Movies",
        "baseSelector": "div.item",  # 每个电影项的基础选择器
        "fields": [
            {
                "name": "title",
                "selector": "div.info div.hd a span.title:first-child",
                "type": "text",
            },
            {
                "name": "rating",
                "selector": "div.star span.rating_num",
                "type": "text",
            },
            {
                "name": "quote",
                "selector": "div.info div.bd p.quote span.inq",
                "type": "text",
            },
            {
                "name": "info",
                "selector": "div.info div.bd p:first-child",
                "type": "text",
            },
            {
                "name": "image",
                "selector": "div.pic img",
                "type": "attribute",
                "attribute": "src"
            }
        ],
    }

    # 设置请求头
    headers = {
        "user-agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/129.0.0.0 Safari/537.36",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
        "Accept-Language": "zh-CN,zh;q=0.9,en;q=0.8"
    }

    # 创建爬虫实例并执行
    async with AsyncWebCrawler(verbose=True) as crawler:
        result = await crawler.arun(
            url="https://movie.douban.com/top250",
            extraction_strategy=JsonCssExtractionStrategy(schema, verbose=True)
        )
        
        # 解析结果
        extracted_data = json.loads(result.extracted_content)
        print(f"提取到 {len(extracted_data)} 部电影")
        print(json.dumps(extracted_data, ensure_ascii=False, indent=2))

if __name__ == "__main__":
    asyncio.run(main())
@unclecode
Copy link
Owner

unclecode commented Nov 4, 2024

@thetesttoy I ran your code on my Mac and it worked fine. Could you provide more details about your specs and platform? There might be an issue on your end, ensure it's installed and running properly, so I can confirm everything works as expected on my side.

image

@unclecode unclecode self-assigned this Nov 4, 2024
@unclecode unclecode added bug Something isn't working question Further information is requested labels Nov 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working question Further information is requested
Projects
None yet
Development

No branches or pull requests

2 participants