btw i dont get why website owners would block crawling.
crawling is done for two reasons:
- your search sucks ass, so someone needs to crawl your content to find anything
- you site is being indexed for search engine. this will bring you more visitors
(- your search cannot be used because you return 403 forbidden when its used)