frontend/static/robots.txt (view raw)
1 2 3 4
User-Agent: * Disallow: /man/ Disallow: /*?*page=0 Crawl-Delay: 2