User-agent: * Disallow: /admin/ Disallow: /static/ Disallow: /media/ # Crawl-delay can be adjusted or removed depending on your crawler policy # Crawl-delay: 10 # Sitemap - replace with your real domain (include scheme http/https) Sitemap: https://westernlabellr.com/sitemap.xml # Optional: specify host (used by some crawlers) # Host: https://westernlabellr.com # Note: To serve this from Django, either place this file in your static root # and configure your webserver to serve it at /robots.txt, or add a URL # pattern that returns this text directly.