About the Rugela link fetcher

When someone posts a link to Rugela, our server looks up a short description of that link so we can show a preview alongside the post. This page explains those automated requests, since you may see them in your server logs.

What it does

• It asks for the page that was linked and reads the OpenGraph tags and standard HTML metadata in the page <head>. That is one HEAD request to find out whether the page is HTML, then one GET for the page itself.
• For YouTube watch, Shorts and youtu.be links, we do not request the page at all. We ask a link-metadata service, Iframely, to describe the video. If Iframely has no answer, we ask the metadata endpoint YouTube publishes for this purpose. Each is one small GET that names only the video's address. Every other link on YouTube and everywhere else is read from its page as described above. If neither source has anything useful to say, we stop there rather than falling back to the page.
• If your site refuses our request, for example with a bot challenge, we may ask Iframely to describe the page instead. That request comes from Iframely, not from us. We never do this for a site whose robots.txt opts out of Rugela, as described below.
• If what we read names a preview image, we download that image once and keep our own resized copy. We never point our readers' browsers at your copy, so your server does not hear from them.
• It does not crawl. It does not follow links on the page, submit forms, or run JavaScript, and it does not come back on a schedule.
• A link is looked up when it is first posted, and we reuse what we found for about a week, so the same link posted repeatedly does not mean repeated requests. After that a later post may look it up again, and a lookup that failed may be tried again by a later post.
• It identifies itself with a User-Agent of the form Rugela/<version> (+https://rugela.com/p/bot/), which is how you arrived here.
• Its requests are signed with Web Bot Auth. To make sure a request really came from us, check its signature against our public key at https://rugela.com/.well-known/http-message-signatures-directory.
• Before fetching a linked page, it requests that site's robots.txt through the same network-safety checks and caches the result for 24 hours. The same check applies to a metadata endpoint, against that endpoint's own address.

Opting out

To stop Rugela from creating previews for all or part of your site, add a section that explicitly names Rugela:

User-agent: Rugela
Disallow: /

Rugela does not apply rules under User-agent: * to these individual, member-requested preview fetches. It does not crawl sites or follow links.

Contact

If you have questions about the link fetcher or its robots policy, email us at bots@rugela.com.