Linkifying robots.txt

by Peter Jaric

I find it quite fun to look for vulnerabilities in websites and then report them (to hopefully gain a bug bounty or at least a place on a Hall of Fame list). One place to find interesting stuff is in the robots.txt file present on many websites.

It is quite annoying though, that the relative URLs found in the file are not clickable. So I made a bookmarklet (JavaScript code run from a bookmark in your browser) to make this possible. I had to make two versions, one for Firefox and on for Chrome, working slightly different.

Installation

Go to this bookmarklet editor.
Paste the code below (depending on browser) into the text field.
Enter a name of your choice for the bookmarklet (e.g. “Linkify robots.txt”).
Click “Compress”.
Drag the bookmarklet link found at the bottom to your bookmarks.

FireFox

javascript:
(function(){
  var base=location.protocol+"//"+location.hostname+(location.port&&":"+location.port);
  var html=document.body.textContent.replace(/: (/.*)/g,': <a href="'+base+'$1">$1</a>');
  html=html.replace(/n/g,'<br>');
  document.open();
  document.write('<body>'+html+'</body>');
  document.close();
}
)()

Chrome

javascript:
(function(){
  var w=window.open();
  var base=location.protocol+"//"+location.hostname+(location.port&&":"+location.port);
  var html=document.body.textContent.replace(/: (/.*)/g,': <a href="'+base+'$1">$1</a>').replace(/n/g,'<br>');
  w.document.write('<body>'+html+'</body>');
  w.document.close()
}
)();

Usage

Go to a robots.txt file somewhere on the web and click the bookmarklet.