MarlyBot

MarlyBot is a simple reader that MARLY uses when it audits someone's online marketing. It is not a search engine, it doesn't index anything and it doesn't collect websites in advance.

MarlyBot opens a website only when a logged-in MARLY user enters that website's address for their own audit or asks for a campaign draft. It first reads the robots.txt file, then the public HTML of the homepage and, if the homepage links to one, the "About" page. During an audit it also reads the sitemap and builds a list of up to 25 subpages of the same website; for at most 8 of them it reads the title and description.

It doesn't read anything behind a login, doesn't submit forms, doesn't run JavaScript and doesn't access private or internal networks. Only the text and basic details (title, description, headings) of each page are used.

MarlyBot doesn't try to get around a website's protection. If your site rejects a request whose User-Agent contains "MarlyBot" (for example with a 403 response) or shows an "are you a robot" check, MARLY treats it as a refusal and doesn't read the site. Blocking by User-Agent on your server or firewall is therefore the most reliable way.

MarlyBot respects robots.txt: it follows the Disallow (and Allow) rules for "User-agent: MarlyBot" as well as for "User-agent: *". A "User-agent: MarlyBot" record with "Disallow: /" stops it from reading the website at all; pages that are disallowed only in part are not read. Sitemap lines are used as a source of subpages.

If you have a question or want us to exclude your website, write to social@marlyagent.cz.