SpiderFoot | Automated Intelligence OSINT Information Gathering Tool

SpiderFoot is an open source tool for online intelligence gathering and footprinting. It automates the task footprinting a given target like the details of IP address, domain name, hostname or network subnet etc.

SpiderFoot is an open source intelligence (OSINT) automation tool. Its goal is to automate the process of gathering intelligence about a given target, which may be an IP address, domain name, hostname, network subnet, ASN, e-mail address or person’s name.

SpiderFoot can be used offensively, i.e. as part of a black-box penetration test to gather information about the target or defensively to identify what information your organisation is freely providing for attackers to use against you.

Read more at the project website: http://www.spiderfoot.net

Key uses of SpiderFoot – An Open Source Intelligence Automation Tool (OSINT)

1)For pen-testers- automating the entire reconnaissance stage.

2)Gathering threat intelligence -about suspicious IPs or site names caught in logs or SIEM tools.

3)For corporate security teams -what sensitive information is openly exposed to the outside world.

Features Of SpiderFoot

SpiderFoot has plenty of features, including the following:

  • Utilizes a lot of different data sources, some of them being SHODAN, RIPE, Whois, PasteBin, Google, SANS and more.
  • Designed for intelligent and maximum data extraction through various modules.
  • Cross platform support- Runs on Linux and Windows and it is open source.
  • User-friendly visualizations -Built-in JavaScript-based visualizations or export to GEXF/CSV for use in other tools with easy to use, easy to navigate web-based UI.
  • Highly configurable modular design with modules written in python so the level of intrusiveness and functionality can be defined.
  • SQLite back-end support - scan results are stored in a local SQLite database so you can play with your data to your heart’s content.
  • Simultaneous scans-Each footprint scan runs as its own thread so you can perform footprinting of many different targets simultaneously.

Getting Started with SpiderFoot

Before installation make sure the following Pre-Requisites are met.

Linux/BSD/Solaris

SpiderFoot is written in Python (2.7), so to run on Linux/Solaris/FreeBSD/etc. you need Python 2.7 installed, in addition to the lxml, netaddr, M2Crypto, CherryPy and Mako modules.

To install the dependencies using PIP , run the following:

~$ pip install lxml netaddr M2Crypto cherrypy mako

Other modules such as MetaPDF, SOCKS and more are included in the SpiderFoot package, so you don’t need to install them separately.

Windows

SpiderFoot for Windows is a compiled executable file, and so all dependencies are packaged with it.

No third party tools/libraries need to be installed, not even Python.

Installation

Installing SpiderFoot is literally as simple as unpacking the distribution tar.gz/zip file.

Linux/BSD/Solaris

To install SpiderFoot on Linux/Solaris/FreeBSD/etc. you only need to un-targz the package, as follows:

~$ tar zxvf spiderfoot-X.X.X-src.tar.gz

~$ cd spiderfoot-X.X.X

~/spiderfoot-X.X.X$

Windows

Unzip the distribution ZIP file and run it.

Starting SpiderFoot

Linux/BSD/Solaris

To run SpiderFoot, simply execute sf.py from the directory you extracted SpiderFoot into:

~/spiderfoot-X.X.X$ python ./sf

Once executed, a web server will be started, which by default will listen on 127.0.0.1:5001. You can then use the web-browser of your choice by browsing to http://127.0.0.1:5001.

If you wish to make SpiderFoot accessible from another system, for example running it on a server and controlling it remotely, then you can specify an external IP for SpiderFoot to bind to, or use 0.0.0.0 so that it binds to all addresses, including 127.0.0.1:

~/spiderfoot-X.X.X$ python ./sf 0.0.0.0:5001

If port 5001 is used by another application on your system, you can change the port:

~/spiderfoot-X.X.X$ python ./sf 127.0.0.1:9999

Windows

SpiderFoot for Windows comes as a pre-packaged executable, with no need to install any dependencies.

For now, there is no installer wizard, so all that’s needed is to unzip the package into a directory (e.g. C:\SpiderFoot) and run sf.exe:

C:\SpiderFoot>sf.exe

As with Linux, you can also specify the IP and port to bind to:

C:\SpiderFoot>sf.exe 0.0.0.0:9999

It can be downloaded in Kali Linux using the following command:

git clone https://github.com/smicallef/spiderfoot.git

download

Once the download completes, make sure you have python installed as it is a pre-requisite for running it. You need to navigate to its downloaded directory and make sure you have all the recommended python packages installed by using the command given below;

python -m pip install requirements.txt

Running Spiderfoot

We will start the Spiderfoot GUI for starting its web server;

python sf.py

web-start

When the webserver gets started, open your preferred browser and go to the http://localhost:5001. It will open the Spiderfoot dashboard as shown in the below screenshot;

Now go to the New Scan tab and enter the scan you wish to utilise, then enter your preferred domain name or IP address you want to enumerate and click Run Scan.

Once the scan gets started, you ca view the results in GUI with a chart;

The data can be viewed logically too;

And also with a graph;

Spiderfoot is a sharp and handy penetration testing tool that can help in providing a great picture of a target within your red teaming scope. It is loaded with several modules, configuration, and OSINT sources to assist in the information gathering process, which gives an in-depth result within a GUI format.

Detailed step by instructions for installing, configuring and running a scan using SpiderFoot can be found on the documentation page http://www.spiderfoot.net/documentation/

9 Likes