About our crawler

How Join a Startup collects job listings, and how to ask us to stop or to remove something.

Join a Startup lists jobs that employers publish on their own careers pages and hiring systems. Our crawler is called JoinAStartupBot, and its requests carry that name. When we read an employer's own web pages, we check its robots.txt file and avoid paths it asks crawlers to stay away from. We also read publicly accessible job feeds from hiring systems. If a site returns an error or asks us to slow down, we wait before trying again.