Improved SEO Capabilities

[dazoulay]

This enhancement aims to improve Composable UI's Search Engine Optimization (SEO) by optimizing the robots.txt file. The robots.txt file plays a crucial role in guiding search engine crawlers and determining which pages should be indexed and crawled. By enhancing the robots.txt file, we can ensure that search engines can efficiently crawl and index the most relevant and valuable content on the platform, boosting organic visibility and search rankings. The implementation will involve fine-tuning the robots.txt directives to exclude irrelevant pages, allowing search engines to focus on essential content, and including appropriate directives to allow for better indexing of important pages.