Who we are
SerpiosBot is the robot of Serpios, a service that tracks the positions of websites in search engines (operated by 4INTERNET, s.r.o.). For our users it checks whether the pages they gave us still link to their website, and on their request it analyses the page that ranks in search results for a keyword they track (and a few pages above it for comparison). If you found it in your logs, someone at Serpios watches a link from your page or compares their page with yours.
How it identifies itself
Mozilla/5.0 (compatible; SerpiosBot/1.0; +https://serpios.com/bot)
We use the same User-Agent to download a website's icon (/favicon.ico), which we show our users next to their projects.
What it does
- It downloads only the exact pages a user entered. It does not follow other links or crawl the whole website.
- It visits each page with a link once every 7 days. When something changed, it confirms it with a second visit about 60 minutes later. A user can also ask for an immediate check.
- For an analysis it downloads the page once per request of a user, who pays for it, not regularly.
- It sends at most one request to a website at a time and pauses between requests, so it does not load your server.
- Before visiting a website it reads its
robots.txt(at most once a day) and does not visit the pages it disallows. - It reads at most 2 MB of HTML, follows at most 5 redirects and does not run JavaScript, submit forms or log in.
- It does not bypass protections against robots. When a website answers with a challenge (e.g. Cloudflare), error 401, 403 or 429, we mark the page as unverified and make no attempt to get around the protection.
What it stores
Only what we need to answer "is the link there?": the page address, the HTTP status code, the address and text of the link found to the user's website, its rel attribute and whether the page has noindex or nofollow in its meta robots. We do not store, index or pass on the content of the page.
For a page analysis we store its basic facts (title, meta description, headings, number of words, structured data and the like) and the review, shown only to the user who asked for it. The text of the page is used only to write the review (our artificial intelligence supplier processes it too) and is not stored.
How to block it
Add to the robots.txt of your website:
User-agent: SerpiosBot
Disallow: /
The robot sees the change within a day. It also follows the rules for User-agent: * when you have none for SerpiosBot, and a ban for a part of the website only (Disallow: /section/).
Contact
If SerpiosBot causes a problem on your website or you have a question, write to us through the Contact page. Please include the website address and ideally the time of the requests from your logs.