KlarenseBot

Who we are

KlarenseBot is the crawler of Klarense. Klarense checks how visitors, search engines and AI assistants see a site and writes a report with advice.

We do not crawl the web on our own. Every check is started by a person on klarense.com: the site owner, their contractor or a visitor. An owner can subscribe to repeat checks of their own site.

What it does

• Opens public pages only: the home page, robots.txt, the sitemap, up to 30 pages of the site and the links on the home page.
• Makes no more than 2 requests a second to one site.
• Does not submit forms, does not log in and does not buy anything.
• Opens the home page in a browser, like a visitor on a phone and on a computer. The browser loads the page's images and styles, as it would for a person.
• Introduces itself as: KlarenseBot/1.0 (+https://klarense.com/bot)

How to block the check

Add this to the robots.txt file at the root of your site:

User-agent: KlarenseBot
Disallow: /

We will then open no page of the site, and the person who asked for the check will see that the site owner has blocked it. We read the file again on every check, so the block works at once.

How to close only part of the site

For example, the /private/ section:

User-agent: KlarenseBot
Disallow: /private/

What if robots.txt blocks all robots?

We follow the «User-agent: *» rule when we go through the site's pages. But we still open the page a person asked us to check: otherwise we could not tell the owner that their site is closed to search engines. To stop that too, name us, as shown above.

Contact us

If KlarenseBot causes trouble for your site, write to hello@klarense.com.