# Crawler Setup User-agent: * # Directories Disallow: /app/ Disallow: /bin/ Disallow: /dev/ Disallow: /lib/ Disallow: /phpserver/ Disallow: /pkginfo/ Disallow: /report/ Disallow: /setup/ Disallow: /update/ Disallow: /var/ Disallow: /vendor/ # Paths (clean URLs) Disallow: */index.php/ Disallow: */review/ Disallow: */catalogsearch/ Disallow: */customer/account/ Disallow: */catalog/product_compare/ #Disallow: */catalog/category/view/ #Disallow: */catalog/product/view/ Disallow: */sendfriend/ Disallow: */tag/ Disallow: */sociallogin/ Disallow: */productalert/ #Parameters not added as all have noindex # Website Sitemap Sitemap: https://www.christmastreeworld.co.uk/media/sitemap.xml Sitemap: https://www.christmastreeworld.co.uk/media/sitemap.xml