JobsiteCreativeBot

What it is

A small crawler run by Jobsite Creative. Every request it makes carries this user agent:

JobsiteCreativeBot/1.0 (+https://jobsitecreative.com/bot; hello@jobsitecreative.com)

What it reads

It has three jobs. For all of them, it reads only public pages of a company’s own website. It does not log in, does not fill in forms and does not run scripts.

• Finding a business address. On the website of a contractor, trade or real estate business it reads the home page and a few pages such as /contact or /about, and looks for the email address the company publishes. Addresses on the company’s own domain come first. A free-mail address, such as Gmail, is used only as a fallback, and only because the company lists it on its own website. The crawler keeps the sentence the address was found in, together with the page and the date, so we can always show where an address came from.
• Checking a request. When someone asks for a free ad at jobsitecreative.com/start and enters a company website, the crawler reads that website to check that the request really comes from that company: the home page, up to two contact or about pages linked from it, and /contact — at most four pages. It compares the company name, phone number and email address with the request, and stops early once it has found an email address and a phone number.
• Re-checking an address before we write. If an address we found is older than 25 days when we are about to write to it, the crawler may read the one page it was found on again, to confirm the address is still published there. It first asks for the page headers only, and reads the page only if it is HTML and at most 500 KB. If the address is gone, we do not write.

How it behaves

• Before it reads anything else on a site, it fetches /robots.txt and follows it as described in RFC 9309. It obeys the group for JobsiteCreativeBot, or the group for all crawlers (User-agent: *) when there is none.
• If robots.txt cannot be read — the server answers 401, 403 or a 5xx error, does not answer in time, or cannot be reached — the crawler treats the whole site as off limits and reads nothing. A missing robots.txt (404 or another 4xx answer) means no restrictions.
• It requests one page at a time, with a one-second pause between requests.
• It follows at most three redirects per page (five for robots.txt), and only while they stay on the same site. It gives up on a page after 8 seconds, reads at most 500 KB of it, and reads HTML pages only.

How to stop it

Add this to your robots.txt:

User-agent: JobsiteCreativeBot
Disallow: /

That stops all three jobs on your site. Or write to hello@jobsitecreative.com with your domain and we remove it for good. If we have already written to you, the unsubscribe link in that email works with one click.