Yeksa
Sign inTry Yeksa free
For website owners

About YeksaBot

YeksaBot is the web crawler behind Yeksa's AI knowledge base. It visits a website only when one of our customers adds that site to their AI agent's knowledge, so the agent can answer questions with accurate, current information.

How to recognize it

YeksaBot identifies itself with this user-agent string:

YeksaBot/1.0 (+https://yeksa.ai/bot)

What it does

  • It reads public pages only. It never logs in, fills in forms or gets past paywalls.
  • It stays on the website it was asked to read and stops at a fixed page limit per crawl.
  • It finds pages through your sitemap and through links, up to four levels deep.
  • It retries a few times on errors and then backs off.
  • On paid plans, it may revisit a site from time to time to keep the knowledge current.

Control it with robots.txt

YeksaBot follows robots.txt. It uses the group addressed to YeksaBot, or the * group if there isn't one, and it respects Crawl-delay. If a site has no robots.txt, YeksaBot treats it as open to crawling.

Block YeksaBot from your whole site:

User-agent: YeksaBot
Disallow: /

Keep it out of one section:

User-agent: YeksaBot
Disallow: /private/

Ask it to slow down:

User-agent: YeksaBot
Crawl-delay: 10

Questions or problems?

If you think YeksaBot is misbehaving on your site, email us with your domain and the time it happened, and we'll look into it. info@yeksa.ai