Installing MASTIFF

So we have talked about using SecShoggoth 's MASTIFF (here, here, and here), but haven't really gone through the installation. Here goes...

I am assuming a base installation of Ubuntu 12.10 (because its easy and you can run it free on AWS). The first thing that we should do is update the base OS and install ssh for remote management.

sudo apt-get update; sudo apt-get upgrade -y; sudo apt-get dist-upgrade -y; sudo apt-get autoremove -y; sudo apt-get install openssh-server -y; sudo shutdown -r now later

Once that is back online we will begin installing the necessary packages for MASTIFF. I am running through the dependencies as they are introduced in the documentation. First lets get the python dependencies out of the way, as well as an editor (nano is fine...albeit evil)

sudo apt-get install python python-dev python-magic python-sqlite python-setuptools python-pip build-essential vim -y 

Install yapsy from pip

sudo pip install yapsy

Now, I tend to pull the majority of my software to /opt out of habit. You do not need to do the same but if you change the location be sure to update to commands below. First I will ensure that the user and group that I am currently using have access to /opt so I can write to that directory.

sudo chown -R `whoami`:`groups | awk '{print $1}'` /opt

Install TrID, download and run the TrID database updater.

cd /opt

mkdir /opt/trid

cd /opt/trid

wget wget http://mark0.net/download/trid_linux.zip

unzip trid_linux.zip

rm -f unzip trid_linux.zip

chmod +x trid

wget http://goo.gl/RQXV8

unzip RQXV8

rm -f RQXV8

chmod +x tridupdate.py

python tridupdate.py

*Note* if you are running this on a 64 bit machine you will need to install ia32-libs

sudo aptitude install ia32-libs

Next we will pull the dependencies down for ssdeep and pyssdeep and then install those packages

sudo apt-get install subversion libpcre3 libpcre3-dev libpcre++-dev -y

cd /opt/

wget http://sourceforge.net/projects/ssdeep/files/ssdeep-2.9/ssdeep-2.9.tar.gz

tar -xvzf ssdeep-2.9.tar.gz

rm -f ssdeep-2.9.tar.gz

mv ssdeep-2.9 ssdeep

cd /opt/ssdeep

./configure

make

sudo make install

sudo ldconfig

svn checkout http://pyssdeep.googlecode.com/svn/trunk/ pyssdeep

cd /opt/ssdeep/pyssdeep

python setup.py build

sudo python setup.py install

Next up is automake and yara:

sudo apt-get install automake -y

cd /opt

svn checkout http://yara-project.googlecode.com/svn/trunk/ yara

cd /opt/yara

sudo ln -s /usr/bin/aclocal-1.11 /usr/bin/aclocal-1.12

./configure

make

sudo make install

cd yara-python

python setup.py build

sudo python setup.py install

Now install simplejson from soure (NOT the APT repo)

sudo apt-get install git -y

cd /opt

git clone https://github.com/simplejson/simplejson simplejson

cd /opt/simplejson

python setup.py build

sudo python setup.py build install

Pull down Didier Stevens awesome pdf tools

mkdir /opt/pdftools

cd /opt/pdftools

wget http://didierstevens.com/files/software/pdf-parser_V0_3_9.zip

unzip pdf-parser_V0_3_9.zip

rm -f pdf-parser_V0_3_9.zip

chmod +x pdf-parser.py

wget http://didierstevens.com/files/software/pdfid_v0_0_12.zip

unzip pdfid_v0_0_12.zip

rm -f pdfid_v0_0_12.zip

chmod +x pdfid.py

Exiftool

cd /opt

wget http://www.sno.phy.queensu.ca/~phil/exiftool/Image-ExifTool-9.22.tar.gz

tar -xvzf Image-ExifTool-9.22.tar.gz

rm Image-ExifTool-9.22.tar.gz

mv Image-ExifTool-9.22 exiftool

PE-File (again NOT from the apt repo)

cd /opt

svn checkout http://pefile.googlecode.com/svn/trunk/ pefile

cd /opt/pefile

python setup.py build

sudo python setup.py build install

Disitool

mkdir /opt/disitool

cd /opt/disitool

wget http://www.didierstevens.com/files/software/disitool_v0_3.zip

unzip disitool_v0_3.zip

rm disitool_v0_3.zip

Openssl

sudo apt-get install openssl -y

pyOLEScanner

mkdir /opt/pyOLEScanner

cd /opt/pyOLEScanner

wget https://github.com/Evilcry/PythonScripts/raw/master/pyOLEScanner.zip

unzip pyOLEScanner.zip

rm pyOLEScanner.zip

chmod +x pyOLEScanner.py

Distorm

cd /opt

svn checkout http://distorm.googlecode.com/svn/trunk/ distorm

cd /opt/distorm

python setup.py build

sudo python setup.py build install

And finally MASTIFF itself

cd /opt

wget http://downloads.sourceforge.net/project/mastiff/mastiff/0.5.0/mastiff-0.5.0.tar.gz

tar -xvzf mastiff-0.5.0.tar.gz

rm mastiff-0.5.0.tar.gz

mv mastiff-0.5.0/ mastiff

cd /opt/mastiff

sudo make install

Now that MASTIFF is good to go we will want to ensure that the config file is created / edited properly. Ensure that you read through the config file as you will want to add the appropriate VirusTotal API key. Also if you installed the dependencies to different locations now is the time to correct those paths. 

mkdir /etc/mastiff

cd /etc/mastiff

 cat > /opt/mastiff/mastiff.conf.TEST <<EOF

# This is the configuration file for mastiff.

#

# Comments are preceded by a # or ;

#

[Dir]

# log_dir is the base directory where the logs generated will

# be placed in.

#log_dir = /usr/local/mastiff/log

log_dir = ./work/log

# plugin_dir is a list of directories plugins may be present in.

# should be comma-separated.

plugin_dir = ./plugins, /etc/mastiff

[Misc]

# verbose = [on|off]

verbose = off

[Sqlite]

# Sqlite database options

# db_file = Name of the database file

db_file = mastiff.db

[File ID]

# trid is the location of the TrID binary

# trid_db is the location of the TrID database

#trid = /usr/local/bin/trid

trid = /opt/trid/trid

#trid_db = /usr/local/etc/triddefs.trd

trid_db = /opt/trid/triddefs.trd

[Embedded Strings Plugin]

# Options for the Embedded Strings Plugin.

# strcmd is the path to the strings command

strcmd = /usr/bin/strings

[VirusTotal]

# Options for the VirusTotal Submission Plug-in.

# api_key is your API key from virustotal.com

#   - Leave this empty if you wish to disable this plug-in

api_key = GET_YOUR_OWN

# submit [on|off] - submit binary to VirusTotal

submit = off

[pdfid]

# Options to run Didier Stevens pdfid.py script

# pdfid_cmd = Path to the pdfid.py script

#   - Leave blank if you want the script disabled.

# pdfid_opts = Options for program.

#   - Do not put multiple options in quotes.

# Note: pdfid.py has bugs that may cause errors when examining

#       malformed PDFs when using the -e option.

pdfid_cmd = /opt/pdftools/pdfid.py

#pdfid_opts = -e

pdfid_opts =

[pdf-parser]

# Options to run Didier Stevens pdf-parser.py script

# pdf_cmd = Path to pdf-parser.py.

pdf_cmd = /opt/pdftools/pdf-parser.py

[PDF Metadata]

# Options for PDF Metadata script

# exiftool = path to exitfool

exiftool = /opt/exiftool/exiftool

[yara]

# Options for the Yara signature plug-in

# yara_sigs = Base path to Yara signatures. This path will be recursed

#             to find additional signatures.

#             Leave blank to disable the plug-in.

yara_sigs = /opt/yara

[Digital Signatures]

# Options to extract the digital signatures

#

# disitool - path to disitool.py script.

# openssl - path to openssl binary

disitool = /opt/disitool/disitool.py

openssl = /usr/bin/openssl

[Office Metadata]

# Options for Office Metadata script

# exiftool = path to exitfool

exiftool = /opt/exiftool/exiftool

[Single-Byte Strings]

# options for single-byte string extraction plug-in

# length - Minimum length to extract

length = 3

# raw - print raw characters instead of formatted ones (e.g. \\n vs. \n)

raw = False

[ZipExtract]

# options for Zip archive file extraction plug-in

# enabled: [on|off] - Extract files or not

# password: Password to use for zip file. OK to leave blank.

enabled = on

password = infected

[Office pyOLEScanner]

# olecmd = Path to pyOLEScanner.py

olecmd=/opt/pyOLEScanner/pyOLEScanner.py

EOF

Now testing MASTIFF out is as simple as:

cd /opt/mastiff

zwned@malwr:/opt/mastiff$ python mas.py /opt/mastiff/tests/test.exe

[2013-03-10 15:11:47,324] [INFO] [Mastiff] : Starting analysis on /opt/mastiff/tests/test.exe

[2013-03-10 15:11:47,326] [INFO] [Mastiff.Init_File] : Analyzing /opt/mastiff/tests/test.exe.

[2013-03-10 15:11:47,326] [INFO] [Mastiff.Init_File] : Log Directory: ./work/log/c69ffb3057b2077fcaecc99b9f16c7c8

[2013-03-10 15:11:47,417] [INFO] [Mastiff.DB.Insert] : Adding ['Generic', 'EXE']

[2013-03-10 15:11:47,506] [INFO] [Mastiff.Analysis] : File categories are ['Generic', 'EXE'].

[2013-03-10 15:11:47,507] [INFO] [Mastiff.Plugins.Embedded Strings Plugin] : Starting execution.

[2013-03-10 15:11:47,521] [INFO] [Mastiff.Plugins.File Information] : Starting execution.

[2013-03-10 15:11:47,602] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Starting execution.

[2013-03-10 15:11:47,602] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Generating fuzzy hash.

[2013-03-10 15:11:47,681] [INFO] [Mastiff.Plugins.Fuzzy Hashing.compare] : Comparing fuzzy hashes.

[2013-03-10 15:11:47,681] [INFO] [Mastiff.Plugins.VirusTotal] : Starting execution.

[2013-03-10 15:11:48,717] [INFO] [Mastiff.Plugins.VirusTotal.submit] : Submission disabled. Not sending file.

[2013-03-10 15:11:48,717] [INFO] [Mastiff.Plugins.yara] : Starting execution.

[2013-03-10 15:11:48,722] [INFO] [Mastiff.Plugins.Resources] : Starting execution.

[2013-03-10 15:11:48,774] [INFO] [Mastiff.Plugins.Single-Byte Strings] : Starting execution.

[2013-03-10 15:11:48,813] [INFO] [Mastiff.Plugins.PE Info] : Starting execution.

[2013-03-10 15:11:48,926] [INFO] [Mastiff.Plugins.Digital Signatures] : Starting execution.

[2013-03-10 15:11:48,975] [INFO] [Mastiff.Plugins.Digital Signatures] : No signature on the file.

[2013-03-10 15:11:48,976] [INFO] [Mastiff.Analysis] : Finished analysis for /opt/mastiff/tests/test.exe.

zwned@malwr:/opt/mastiff$ python mas.py /opt/mastiff/tests/test.pdf

[2013-03-10 15:12:36,299] [INFO] [Mastiff] : Starting analysis on /opt/mastiff/tests/test.pdf

[2013-03-10 15:12:36,299] [INFO] [Mastiff.Init_File] : Analyzing /opt/mastiff/tests/test.pdf.

[2013-03-10 15:12:36,300] [INFO] [Mastiff.Init_File] : Log Directory: ./work/log/3f53a4bf0097f9075ff641b03bb176f5

[2013-03-10 15:12:36,381] [INFO] [Mastiff.DB.Insert] : Adding ['PDF', 'Generic']

[2013-03-10 15:12:36,468] [INFO] [Mastiff.Analysis] : File categories are ['PDF', 'Generic'].

[2013-03-10 15:12:36,469] [INFO] [Mastiff.Plugins.pdf-parser] : Starting execution.

[2013-03-10 15:12:36,470] [INFO] [Mastiff.Plugins.pdf-parser.uncompress] : Uncompressing PDF.

[2013-03-10 15:12:36,563] [INFO] [Mastiff.Plugins.pdf-parser.get_objects] : Extracting interesting objects.

[2013-03-10 15:12:37,532] [INFO] [Mastiff.Plugins.PDF Metadata] : Starting execution.

[2013-03-10 15:12:37,643] [INFO] [Mastiff.Plugins.pdfid] : Starting execution.

[2013-03-10 15:12:37,729] [INFO] [Mastiff.Plugins.Embedded Strings Plugin] : Starting execution.

[2013-03-10 15:12:37,741] [INFO] [Mastiff.Plugins.File Information] : Starting execution.

[2013-03-10 15:12:37,819] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Starting execution.

[2013-03-10 15:12:37,820] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Generating fuzzy hash.

[2013-03-10 15:12:37,909] [INFO] [Mastiff.Plugins.Fuzzy Hashing.compare] : Comparing fuzzy hashes.

[2013-03-10 15:12:37,910] [INFO] [Mastiff.Plugins.VirusTotal] : Starting execution.

[2013-03-10 15:12:38,386] [INFO] [Mastiff.Plugins.VirusTotal.submit] : Submission disabled. Not sending file.

[2013-03-10 15:12:38,386] [INFO] [Mastiff.Plugins.yara] : Starting execution.

[2013-03-10 15:12:38,392] [INFO] [Mastiff.Analysis] : Finished analysis for /opt/mastiff/tests/test.pdf.

zwned@malwr:/opt/mastiff$ python mas.py /opt/mastiff/tests/test.doc

[2013-03-10 15:12:53,882] [INFO] [Mastiff] : Starting analysis on /opt/mastiff/tests/test.doc

[2013-03-10 15:12:53,883] [INFO] [Mastiff.Init_File] : Analyzing /opt/mastiff/tests/test.doc.

[2013-03-10 15:12:53,883] [INFO] [Mastiff.Init_File] : Log Directory: ./work/log/759f7e53f54df03f2ae06fcec25e8ac3

[2013-03-10 15:12:53,973] [INFO] [Mastiff.DB.Insert] : Adding ['Generic', 'Office', 'ZIP']

[2013-03-10 15:12:54,076] [INFO] [Mastiff.Analysis] : File categories are ['Generic', 'Office', 'ZIP'].

[2013-03-10 15:12:54,078] [INFO] [Mastiff.Plugins.Embedded Strings Plugin] : Starting execution.

[2013-03-10 15:12:54,088] [INFO] [Mastiff.Plugins.File Information] : Starting execution.

[2013-03-10 15:12:54,167] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Starting execution.

[2013-03-10 15:12:54,167] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Generating fuzzy hash.

[2013-03-10 15:12:54,234] [INFO] [Mastiff.Plugins.Fuzzy Hashing.compare] : Comparing fuzzy hashes.

[2013-03-10 15:12:54,234] [INFO] [Mastiff.Plugins.VirusTotal] : Starting execution.

[2013-03-10 15:12:55,239] [INFO] [Mastiff.Plugins.yara] : Starting execution.

[2013-03-10 15:12:55,244] [INFO] [Mastiff.Plugins.Office pyOLEScanner] : Starting execution.

[2013-03-10 15:12:57,497] [INFO] [Mastiff.Plugins.Office Metadata] : Starting execution.

[2013-03-10 15:12:57,681] [INFO] [Mastiff.Plugins.ZipInfo] : Starting execution.

[2013-03-10 15:12:57,682] [INFO] [Mastiff.Plugins.ZipExtract] : Starting execution.

[2013-03-10 15:12:57,683] [INFO] [Mastiff.Plugins.ZipExtract] : Password "infected" will be used for this zip.

[2013-03-10 15:12:57,683] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting [Content_Types].xml.

[2013-03-10 15:12:57,683] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting _rels/.rels.

[2013-03-10 15:12:57,684] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting theme/theme/themeManager.xml.

[2013-03-10 15:12:57,684] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting theme/theme/theme1.xml.

[2013-03-10 15:12:57,685] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting theme/theme/_rels/themeManager.xml.rels.

[2013-03-10 15:12:57,685] [INFO] [Mastiff.Analysis] : Finished analysis for /opt/mastiff/tests/test.doc.

zwned@malwr:/opt/mastiff$ python mas.py /opt/mastiff/tests/test.

test.doc  test.exe  test.pdf  test.zip

zwned@malwr:/opt/mastiff$ python mas.py /opt/mastiff/tests/test.zip

[2013-03-10 15:13:22,856] [INFO] [Mastiff] : Starting analysis on /opt/mastiff/tests/test.zip

[2013-03-10 15:13:22,870] [INFO] [Mastiff.Init_File] : Analyzing /opt/mastiff/tests/test.zip.

[2013-03-10 15:13:22,871] [INFO] [Mastiff.Init_File] : Log Directory: ./work/log/033d488bbe65e8aececb2c55bdfbc2fd

[2013-03-10 15:13:23,035] [INFO] [Mastiff.DB.Insert] : Adding ['Generic', 'ZIP']

[2013-03-10 15:13:23,106] [INFO] [Mastiff.Analysis] : File categories are ['Generic', 'ZIP'].

[2013-03-10 15:13:23,107] [INFO] [Mastiff.Plugins.Embedded Strings Plugin] : Starting execution.

[2013-03-10 15:13:23,115] [INFO] [Mastiff.Plugins.File Information] : Starting execution.

[2013-03-10 15:13:23,178] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Starting execution.

[2013-03-10 15:13:23,178] [INFO] [Mastiff.Plugins.Fuzzy Hashing] : Generating fuzzy hash.

[2013-03-10 15:13:23,238] [INFO] [Mastiff.Plugins.Fuzzy Hashing.compare] : Comparing fuzzy hashes.

[2013-03-10 15:13:23,238] [INFO] [Mastiff.Plugins.VirusTotal] : Starting execution.

[2013-03-10 15:13:23,440] [INFO] [Mastiff.Plugins.VirusTotal.submit] : Submission disabled. Not sending file.

[2013-03-10 15:13:23,440] [INFO] [Mastiff.Plugins.yara] : Starting execution.

[2013-03-10 15:13:23,445] [INFO] [Mastiff.Plugins.ZipInfo] : Starting execution.

[2013-03-10 15:13:23,446] [INFO] [Mastiff.Plugins.ZipExtract] : Starting execution.

[2013-03-10 15:13:23,447] [INFO] [Mastiff.Plugins.ZipExtract] : Password "infected" will be used for this zip.

[2013-03-10 15:13:23,447] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting test.bin.

[2013-03-10 15:13:23,447] [INFO] [Mastiff.Plugins.ZipExtract] : Extracting test.txt.

[2013-03-10 15:13:23,448] [INFO] [Mastiff.Analysis] : Finished analysis for /opt/mastiff/tests/test.zip.

If you did not receive any errors / warnings you should be good to go. Now that MASTIFF is up and running we can download MASTIFF2HTML on Github.

wget https://raw.github.com/1aN0rmus/TekDefense/master/MASTIFF2HTML.py

chmod +x MASTIFF2HTML.py

python MASTIFF2HTML.py -f /opt/mastiff/work/log/ -d mastiff.db

cd /opt/mastiff/work/log/www/

python -m SimpleHTTPServer

Now if you browse to 127.0.0.1:8000/mastiff.html you should be seeing the results for your analysis.

 If you run into any issues... please leave a comment so we can address / update as necessary.

Cheers