About the Silotree crawler

If you saw SiloTree/1.0 (+https://silotree.io/bot) in your logs, someone asked Silotree to check the internal links on your site — usually you, or someone who works on it.

How it behaves

• It reads public pages only, one request at a time with a pause between each.
• It follows robots.txt, including Crawl-delay, which it will only ever make slower.
• It never submits forms, signs in, or changes anything on your site.

How to limit or block it

Add a group for it to your robots.txt:

User-agent: SiloTree
Crawl-delay: 10     # slower
# or
Disallow: /         # not at all

Questions or a problem with a crawl? Email hello@silotree.io.