cooking, programming and everyday life xrado

Monday, October 27, 2008

FireSpider extension for Firebug

FireSpider is a an extension for Firebug that can help you find broken or misleading urls and optimize your web page for search engines.

Extension enables you start and the stop the spider. After pressing the start button it fetch the current url and parses the content for new urls. Every unique url is fetch only once. It follows only the current domain urls and detects its content type. Not html content types are reported. Besides fetching, firebug panel gives you information about page title, page where url was got from, its link name and loading time. Reported are also not found urls and ones went in time out after 10 seconds. Requests are send one after another, so some sites may find it to aggressive or may detect you as a dangerous script. Be careful.

image

This is the first version (hope not the last) and first extension i ever made. Some things may not be very well done, but the basic functionalities are working, i think. Try it and tell me what you think. Hope you find it useful.

firespider-0.1.xpi ( download / install )

FireSpider on GitHub
Great script ... very useful ...

Awesome extension... tnx dude!

16:43, Monday, November 10, 2008 Paul Davis, from info@pauldavis.org.uk
Thanks!

Very Nice.

I just tried it on a phpBB topic/post with 2 pages and it really took off ! I guess it finds all the links on all linked pages ??? (I'm just learning how to use it). It took quite a while, but it's a huge phpBBS.

Is there anyway to export the results besides SELECT ALL and COPY ?

Anyway to restrict it to only the current page in the browser tab/window ?

Anyway it looks like a nice add-on. I'll have to keep working with it.

Thanks!
yes it finds links from the whole page. no for now is not possible to export the results, may be in future. what kind of report would you need and why? yes i'm planing to add more features, like current page only, internal && external urls, ... i just need some free time.
greetings

Hey xrado!

I was just looking at the .js files you included in the .xpi, I'm new to Javascript so I'm looking at your work to see how to do XMLHTTP requests and to wait for the pages to load, etc, etc.

For exporting it'd be nice just either to list the urls it finds on the page as plain text, or with the data you provide as CSV type information. Be nice to copy the export to the clipboard, since Firebug can copy HTML to the clipboard. Just ideas at this point.

I can grab URLs from web pages and websites using various tools, like wGET, CURL, XMLStarlet and automating MSIE via the MSHTML object model; also tools like WinHTTRack can spider and download web sites.

The hard part is downloading all the pages in a phpBBS or a vBulletin topic. Ever been to a bbs and the topic you want is 20+ pages !!! ?

So I've written scripts that will download all the pages in that one topic and then rewrite the navigation links so that the local pages can now use them.

Things like that.

Anyway, I don't want to heap lots of work on you; just ideas at this point. However the ability to examine all the links on the tab or page I'm viewing and export those would be very nice ! Sure beats having to write a script each time !

Very nice add-on and I will hopefully learn a lot from the hard work you've put into it.

Thanks again !

20:58, Wednesday, November 12, 2008 Mr. T, from outerspace
Great tool and congratulation with the .1 release. I got it through the Firebug Google Group.

Exporting the result to a csv or similar format would be great.
I guess your target audience would be expanded( like bosses and similar who likes to put things in a presentation etc.).

+1 for limiting the run to links on a single page.




Hi,

Nice plugin. It's exactly what I'm looking for testing my application.
But there are some features for improvement:
1) Options for specifiyng ignorable "GET" parameters.
2) Support of pages in UTF8 (where characters encripted as &#XXX;)

If you would like, I can help you with development of this plugin.

hi there! :) I am glad that you like it and you are welcome to help developing it. Since the first release, I've been busy with other stuff, so I didn't do anything more, but I like to. feel free to contact me. greetings.

14:39, Friday, March 06, 2009 Vadim P., from vperetokin@gmail.com
Quite useful thing, thanks. I hope you'll get time to improve it :)

00:29, Thursday, March 19, 2009 sic_people, from sic_people@hotmail.com
Hello i was trying the plug in and found a bug, when you try to access to an URL via other port besides the 80 port, firespider try to access all the URL through the 80 port.

thx for info!

nice plugin, but won't work with frames.

Firespider does not respect the Base-Tag which specifies the default prefix for all (internal) links on the page, e.g.


not compatible with FF 8.0.1. kindly update. thanks

Not compatible with FF 10.0.2

20:15, Friday, August 31, 2012 root, from Moscow
Not compatible with FF 15

19:14, Saturday, May 04, 2013 Daniel, from Lagos
Out of date. Please upgrade. Thanks

I think, you are merged with uncle G? developers.google.com/speed/pagespeed/

07:36, Monday, June 23, 2014 Azee, from tauseef.azhar@technocrab.in
This addon is not compatible with latest version of firefox. Will you please make it compatible with the latest version of ff.

What do you think?
name from url
   can't read the code? (reload)