This was a really fun little script I wrote that I thought I would share with everyone. I basically wrote a Perl script using the popular LWP Perl module to screen scrape and obtain the top 25 sites and the web servers they use. Once I screen scrape to get the relevant website, I am able to send a specific request to the server to obtain further information.
Now, I am not saying I ran this script, and, I am not responsible in any way for how you may use this script. Further, you should be aware that some sites may not allow you to use automated processes to capture and use their information; this could be a possible violation of copyright or intellectual property laws. So now that you have been warned, below I display the code I wrote and also what the output would look like if you were to run this script. You can download the full source code from the download section.
Tags: alexa, alexa.com, bot, bots, download webpage, hack, hacking, HTTP, HTTP header, HTTP request, HTTP response, Library for WWW in Perl, libwww-perl, LWP, LWP::Simple, LWP::UserAgent, perl, screen scraping, script, scripts, server, Simple, top 25 websites, UserAgent, web server. web servers. automate