FireSpider is a an extension for Firebug that can help you find broken or misleading urls and optimize your web page for search engines.
Extension enables you start and the stop the spider. After pressing the start button it fetch the current url and parses the content for new urls. Every unique url is fetch only once. It follows only the current domain urls and detects its content type. Not html content types are reported. Besides fetching, firebug panel gives you information about page title, page where url was got from, its link name and loading time. Reported are also not found urls and ones went in time out after 10 seconds. Requests are send one after another, so some sites may find it to aggressive or may detect you as a dangerous script. Be careful.
This is the first version (hope not the last) and first extension i ever made. Some things may not be very well done, but the basic functionalities are working, i think. Try it and tell me what you think. Hope you find it useful.
I just tried it on a phpBB topic/post with 2 pages and it really took off ! I guess it finds all the links on all linked pages ??? (I'm just learning how to use it). It took quite a while, but it's a huge phpBBS.
Is there anyway to export the results besides SELECT ALL and COPY ?
Anyway to restrict it to only the current page in the browser tab/window ?
Anyway it looks like a nice add-on. I'll have to keep working with it.
yes it finds links from the whole page. no for now is not possible to export the results, may be in future. what kind of report would you need and why? yes i'm planing to add more features, like current page only, internal && external urls, ... i just need some free time.
For exporting it'd be nice just either to list the urls it finds on the page as plain text, or with the data you provide as CSV type information. Be nice to copy the export to the clipboard, since Firebug can copy HTML to the clipboard. Just ideas at this point.
I can grab URLs from web pages and websites using various tools, like wGET, CURL, XMLStarlet and automating MSIE via the MSHTML object model; also tools like WinHTTRack can spider and download web sites.
The hard part is downloading all the pages in a phpBBS or a vBulletin topic. Ever been to a bbs and the topic you want is 20+ pages !!! ?
So I've written scripts that will download all the pages in that one topic and then rewrite the navigation links so that the local pages can now use them.
Things like that.
Anyway, I don't want to heap lots of work on you; just ideas at this point. However the ability to examine all the links on the tab or page I'm viewing and export those would be very nice ! Sure beats having to write a script each time !
Very nice add-on and I will hopefully learn a lot from the hard work you've put into it.
Nice plugin. It's exactly what I'm looking for testing my application.
But there are some features for improvement:
1) Options for specifiyng ignorable "GET" parameters.
2) Support of pages in UTF8 (where characters encripted as &#XXX;)
If you would like, I can help you with development of this plugin.
hi there! :) I am glad that you like it and you are welcome to help developing it. Since the first release, I've been busy with other stuff, so I didn't do anything more, but I like to. feel free to contact me. greetings.