About Our Bot - Companies.now

What is CompaniesNowBot?

CompaniesNowBot is the web crawler used by Companies.now to collect publicly available information about companies. When a company is submitted to our platform, our bot visits the company's website to gather basic information such as the company description, logo, and social media links.

Technical Details

User Agent

CompaniesNowBot/1.0 (+https://companies.now/bot)

Hostnames

webcrawler.1.companies.now
webcrawler.2.companies.now
webcrawler.N.companies.now

The number may vary as we use multiple crawler instances.

Crawl Behavior

Visits only submitted company domains
Typically fetches the homepage only
Does not follow internal links extensively
Crawls at a polite rate with delays between requests

Robots.txt Compliance

CompaniesNowBot fully respects robots.txt directives. If your robots.txt file disallows our bot, we will not crawl your website.

To block CompaniesNowBot, add this to your robots.txt:

User-agent: CompaniesNowBot
Disallow: /

Edit or Remove Your Company

If you find incorrect information about your company on our platform, you have several options:

Edit the profile
Anyone can suggest edits to company profiles. Visit the company page and click "Edit" to propose changes.
Flag the profile
If you believe the profile contains inappropriate content or should be removed, use the "Report" button on the company page.
Contact us directly
For urgent matters or removal requests, email us at hello@companies.now

Contact

If you have questions about our crawler or need to report an issue, please contact us at: hello@companies.now