Thursday, January 19, 2017

Visual link analysis with Splunk (or SQL) and Maltego using the MDS

We're finally ready to release a public beta of the Maltego Data Server (MDS). The MDS is a server that allows you to trivially easy visualize data kept in SQL databases or indexes (such as Splunk) in Maltego - as a graph.

In the most simplest form you only need to write a query (SQL/Splunk) and a tell the MDS how to map the resultant data back to nodes on the graph.


In the most complex form you can write Python code around the query, mapping and nodes as well as use (global) replacement variables anywhere within the items above. With this we mean to say that the MDS can be as easy or as complex as you'd like it to become. The system can grow with your abilities and is very flexible.

With the very basic knowledge of SQL/Splunk and Maltego you can almost immediately get massive insight into the most mundane of logs. With two (basic AF) Splunk-based transforms and three of the standard OSINT transforms that ship with Maltego we can spot fake Googlebots almost instantly in our web server logs:


Keep in mind that the power of the existing Threat Intelligence transforms available in the Transform Hub is at your fingertips - making it possible to enrich your internal data to the max.


If you are interested to test drive the MDS *today* you can simply email us at mds-beta@paterva.com and we'll send you the server as an OVA to experiment with. You can read the comprehensive documentation for the MDS [here] right now. 

We love to get your feedback on our new project.

RT

PS: the commercial people just told us we should include that we're going to be selling this in future. Don't know why that's important...but ye.

Tuesday, January 17, 2017

Making Buzzfeed's TrumpWorld tables into a Maltego graph

Maltego 4.0.15 is on it's way, and with it a brand new interface for importing data into Maltego. With Buzzfeed's recent data dump of "TrumpWorld" we thought we would have some fun mapping out the data, whilst doing a walk-through of the new Tabular importer.

TL;DR
-----

With just a few easy clicks you can map out hundreds of links and entities. We can see the complex layout of Trump business empire, as well as how his social and business circles overlap.

Maltego provides a wide array of transforms to dig deeper into the information we have here. We'll leave that as an exercise for the reader ;)

Person - Company mapping



Person - Person mapping


Company - Company mapping


Just in case anyone was worried that we were getting too political (we're neutral, like Switzerland), here's a graph of Hillary Clinton's email infrastructure. What's the SSLVPN box by the way? ;)

Try It For Yourself

Here are all the Maltego graphs - feel free to open them in any version of Maltego as long as it starts with a 4.  (including the free (4) CE version!).

Download Graph Files


Maltego 4.0.15's new tabular import (aka how we did it)

Start by clicking "Import Graph from Table" under the "Import|Export" section of the ribbon bar.


Click "Next" and select an Excel or csv file.


In this case we will be using "TrumpWorld Data — Public - Person-Org". Once you have selected your file click "Next".


The Hint at the bottom of the next dialogue explains the different connectivity options. We're going to pick "Sequential" because it's really a A->B mapping, but the other defaults are useful in other situations.


We have to tell Maltego which column represents which type of data. We have chosen to map column 1 to a "Company" entity (we've imported it using the CaseFile entity pack in the Transform Hub) and column 2 to a "Person" entity.

The information in the other two columns we won't be using to make entities, so we set them to "Unmapped".


Under the "Map Columns to Links" tab we can choose to use column 3 as the label for the connection between column 1 and column 2.



We can now see a visual representation of how each row will be imported by going to "Connectivity Graph". We see that a link will be made from the Person to the Company.


The final step is to check that all the settings are correct and click "Next" to import the data into Maltego.


You will then see a summary of what was imported.





Tuesday, January 3, 2017

Short term (Q1 '17) plans for Maltego

Welcome to 2017. It's only the 3rd of January and we're all back at work. I thought I share some of the exciting things happening with Maltego in the short term.

Awesome documentation

Documentation was never our strong suit and so this year we're setting it right and putting a lot of effort into documenting Maltego. We started with the user guide - it's brand new and shiny and available [here].

We're redoing the transform guide on a wiki - so that other transform writers can also document their stuff a little - so far it's looking grand and useful. We're also doing a lot of maintenance on the developer portal to get that up to date. Let it never be said again that our documentation suck!

Maltego Data Server (MDS)

We're almost done with the MDS. It's currently (almost) in beta. If you want to play or get a copy of the user's manual [drop us a line]. Some time ago we've made a [sneak peek video] of the MDS:


The MDS is going to be 'the next big thing'.

Maltego GUI

In the spirit of making Maltego easier to work with your own data we have a two prong attack. We're doing a lot of work in terms of the tabular data import function for local data files. The partial screenshot below should give you a taste of what's coming:



Import speed has been optimized and we now load 200k records in a mere 11 seconds!

For work with big(ish) data in SQL databases and (Splunk/ELK) indexes we have the MDS (see above).

We are also planning to have a unified Maltego installer and lifelong license keys - meaning you can easily upgrade from CE to Classic to XL. It solves a lot of issues from us building new Maltego releases as well as solving a lot of licensing headaches (think renewal, different license keys every year etc. etc.)

Exciting times and more as it happens,
RT

Friday, December 23, 2016

Christmas Special & State of the Nation. It's a thing!

Here we are – at the end of 2016. For some 2016 was a great year. For others... not so much. It was indeed a year where we saw many changes around the world. And you know what they say about change. No no - not the holiday thing, that funny thing about not wearing ski pants in the desert. 

Was 2016 a bad year for Maltego? Hmm - no. We released a major version (M4) this year. We fixed a lot of bugs in it and we’re now up to the 12th update for the 4.0 version. We finally had the courage to split it into two flavors (Classic and XL). We built the MDS this year. A few more tests and it’s ready for production in early 2017. Every time I play with the MDS I smile. It’s truly a thing of beauty. Maltego 4 + MDS is going to turn out to be VERY powerful.

The company has grown too – not just with clients and sales – but with people too. Yesterday another recruit started with the company. His name is Andrew. How is it that we have five people working in one office and two of them are called Andrew?  Don’t ask – it’s still 2016. We now have proper offices in Pretoria(here and here) having moved from Cape Town. Yes – we moved there and then moved back. I said don’t ask. 

After almost 10 years around people tend to think we need to “grow up”. Become a “proper company”.  Focus more on “making the numbers” and thus less on making cool and useful tech anymore. Cut the intros for our videos and use "sensible fonts". Find a receptionist and a PABX and a stall at *that* conference and a real company letterhead. Well |=|_|[|< all that.  We fight against that on a daily basis.  We resist to conform, to become a “me too!”. And after every Maltego design meeting we sit down again and question our decisions and ask if it’s REALLY the direction we want to take.

I know you’re actually only here for the Christmas special coupon code. It’s a tradition we’ve kept going for the last few years. The plan here is that you can buy Maltego as a gift to your partner/wife/husband/girlfriend/boyfriend/dog/parakeet/goldfish/cactus/athlete's foot at a real bargain and that you don’t need to buy them socks - again. 

The coupon is all lower case. It contains no spaces. It’s the new server we’re releasing in 2017 – append to that the name of person that just joined the company. You can of course Tweet this coupon to the world. Do that and we'll give your Twitter alias to a South African traditional healer which in turn will give you a penis enlargement. Even if you’re not male. And keep in mind they’re not always super reliable so stuff might go wrong. It’s up to you.

The coupon gives you 40% off Maltego Classic and XL (discount do not apply to renewal free). The coupon is valid from now to the 27th of Dec (00h00, GMT+2).

Baby seals,
RT



Thursday, September 29, 2016

Maltego 4 CE / Kali Linux release is ready for download!

Hi there,

We're happy to announce that Maltego 4 is now (finally) ready for the masses! We're releasing the community (free) edition today and the Kali distros have been updated by the kind people from Offensive Security (thanks Dookie/Muts!).  In other words - we're ready to roll on a major upgrade of your favorite information visualization tool.


(click on the image above to see our very grown-up/proper promotional video of Sandra the 15 year old Dachshund and Maltego/Kali Linux. !(We plan to screen this at our booth at a major conference.))

Our decision to make CaseFile free with the release of Maltego 4 had some interesting side-effects. In CaseFile importing data from CSV/XLS was enabled. So too printing. And reporting. So when we made CaseFile free it did not make sense to limit the Kali/CE releases - you'd simply open CaseFile, import the data and save the graph - then open in CE.

So - bottom line - reporting/printing/CSV import is now enabled in the free release!

The major changes from 3.6 to 4.0 is the ability to render and use large graphs, the use of collection nodes and a brand new interface. To see a more complete overview of the improvements in Maltego 4 you might want to view our release video [HERE].

For the CE version (OSX/Windows/Linux/SNES/ZX81/C64) click [HERE], download and install.

For Kali Linux - if you're running 2016.2 (recommended) you can simply type:

# apt-get update && apt-get install maltegoce

If you're using Kali Linux 2016.1 it's a bit of a bigger mission but you can open a terminal and type:

# apt-get update && apt-get dist-upgrade

This will upgrade your Kali to the latest - and it's good thing(tm) anyhow.
Once you're good to go start Maltego like you normally do.



We hope you have endless fun using Maltego 4 and that you find it super useful in your explorations.

RT

Monday, May 16, 2016

Panama Papers in Maltego

By now everyone knows about the Panama Papers and the Offshore Leaks. If you don't you should read about it [here]. We've downloaded the CSV files from them, imported into a SQL database, then wrote some transforms for Maltego. That's the context.


Disclaimers. You should really really read this!

First off - some disclaimers. I know nobody ever reads disclaimers but these are pretty important so you really need to read them.

Disclaimer 1: Not everyone in the database is 'bad'. Having an offshore account is not a crime. There are good reasons to have one. Like they say on the their site: "There are legitimate uses for offshore companies and trusts. We do not intend to suggest or imply that any persons, companies or other entities included in the ICIJ Offshore Leaks Database have broken the law or otherwise acted improperly."

Disclaimer 2: People have the same names. Who would have thought?! You find someone in the data and go 'oooh! Het jou katvis!' - but remember that it could be someone else with that same name. Manually verify results - always!

Disclaimer 3: The data is not very clean. There could be four entries for the same person and in Maltego these nodes will not merge (different node_IDs). You'll need to manually merge them if you feel like it. Of course, see 2 - e.g. they could be four different people. The same goes for addresses - the data was clearly captured by hand, so people write the same address in many different ways. Best thing here is to take the most significant part of the address and search for that - then manually verify.

Disclaimer 4: The transforms might break. I am not even a proper coder. It should be OK, but when a query does not return or stuff falls apart then remember this disclaimer. If we get a LOT of interest on this then we might rewrite the transforms properly. Also - there's a lot of improvements that can be made on the transforms. Display info etc. etc. Don't tell us - we know this.

This was hacked together on a Friday afternoon and a Saturday night and by the end of the day it seemed very useful and that's why we're releasing it now.

With that out the way, let's first see how to get the transforms and entities into Maltego. We thought about adding this into the Transform Hub but decided against it. It's cool, but it's not THAT cool. That means you need to install the transforms by hand. Luckily, it's pretty easy.

How to install

In the transform hub, click on the [+] sign. Fill in the fields as you wish. The only part that needs to be the same as our example is the seed URL. The seed URL is [https://bark.paterva.com:8081/iTDSRunner/runner/showseed/PanamaPapers]


 

Once you filled it in hit OK. You'll now see the item appears in the transforms hub:

Hover over it and click on 'Install'. It should look something like this when you're done (this is Maltego 4, but the other versions should look similar):

Woot! Now you're ready to start using the transforms.

How to use 

Before we start we want to quickly discuss the data. There are 4 tables. Officers (people), Entities (companies, trusts or other legal entities), Addresses (duh - addresses), Intermediaries (think agents or companies or people doing the work on behalf of the officers). Then there's a table that links all of these together. 

There are 4 entities in Maltego - Officers, Entities, Intermediaries, Addresses and Country. The transforms implement an almost fully meshed grid between these with a couple of spaces where it's not really applicable.

The starting point for all transforms is a Phrase. As the data is mostly linked by node IDs you cannot start with any of the 'PanamaP' entities as you don't know what the node ID is. You always start with a Phrase and search from there.

Let's see how this works. Let's assume we're looking for an officer called 'Hillary Clinton'. We suggest looking for just the word 'Clinton'. We drag a Phrase entity (in the Personal section) onto the graph, double click on the text and change it to 'Clinton'. Then we right click on the entity to bring up the context menu, navigate all the way to the top (right click on the menu) and select the Panama Papers transforms:
In that group we select the 'PP Search officer' transform:
This results in:
Let's assume we're interested in one of the nodes and want to see what entities and addresses are connected to that officer. We select one of the nodes, right click and run the 'PP Get details' transform:
We can do the same on the Entity that's returned from here:

And so the story goes on...

Another interesting way to look at the data is to start looking for the Addresses. This is sometimes useful to identify Officers from certain locations. For broader searches you can start from a country...

Let's see which officers stays in Beverly Hills. We start with a phrase 'Beverly Hills' and run the 'PP Search addresses':
We get 47 addresses in Beverly Hills that's in the database. Let's see what's going on there. We select all the nodes and run the transform 'PP To officers or entities here' transform:

...but wait...

Does 'Beverly Hills' exist in other countries too? Yes. In Australia. In Hong Kong. Probably in other countries too. So we need to remove them. Control F, type in 'Hong'. Hit find. Control shift down arrow (select children). Delete. Rinse and repeat for others. Hmmm.. perhaps Beverly Hills was a bad choice. There's even a Beverly Hills in Balito, South Africa. Really? REALLY?

Anyhow. Rinse. Repeat. And then:

Pretty please read the disclaimers at the start of this post. You probably scrolled to the end right away. But please read them.

And this time, for realsies -- use responsibly!
RT

Tuesday, May 3, 2016

Maltego 4 - it's finally time...

TL;DR:

Maltego 4 is finally ready...click on the picture below to view the release video:


Download the software [here]

...but if you want to know more...

The Maltego 4 story

In March of 2015 myself, Chris and Andrew sat in a room in Cape Town to decide which feature to build next. It's one of the hardest challenges managing Maltego - deciding what to do next. There's always at least five major features competing for our attention. Be that geospatial view, temporal view, feeders or a browser plugin - there's always the next big thing waiting. We argued the entire day, everyone having their own favorite. At around 7 o clock we were tired, hungry and irate. I asked Paul (at the time still pretty green and struggling to keep up with all the intricacies of a new design) "if you could have any feature in Maltego - what would it be?". He didn't have to think long and answered "handling big graphs". Then he casually put his headphones back on and ignored us.

It wasn't what I wanted to hear. We kept on ignoring the issue to the point that we almost believed it wasn't a problem anymore. We didn't want to fix it. It was hard to fix. It meant ripping the guts out of our product. We all knew that it would mean many months of nothing but rebuilding things we already had. No new features, no new flashy bits. Just hard work - rebuilding Maltego from the ground up. But Paul was right. It wasn't the popular answer, but probably the right answer.

For the months to follow we had no new features coming out. We issued a couple of patches for Maltego Chlorine (3.6) and kept supporting the old version. I asked Chris and Sonja if they had a rough idea on when we'll be done. The first date we tried for was Black Hat Las Vegas 2015. August. By June we all knew it was way too early and we pushed it back to Christmas 2015. In early December 2015 they sent me a barely working version. It included lots of disclaimers on which parts I could play with - but it could handle 30 000 nodes with ease. It was exciting, so exciting that I had to make a video about it. We decided we needed a new website too. Paul was to run with that - it had to be ready to go with the release of the new version.

Putting back all the pieces took longer than we anticipated and we hadn't even started on collection nodes - the secret weapon in the fight against large graphs. Collection nodes were not a new concept. We tried it back in 2009 and never released it - it failed miserably, partly because the product (and perhaps we ourselves) were simply not mature enough. The trick then was usability and the usability of collection nodes was a major struggle now. We decided to completely redo the interface. The version I had in my hands looked really bad. The user experience was bad. It was riddled with bugs, things that simply didn't work. I pulled the video. It said we'll have it before 2016. There was simply no way we'd have it done. Christmas came and went and we had nothing.

During January 2016 I felt like the new version was never going to happen and that, even if we did get it right, users would hate it.  I didn't even want Andrew and Paul to try this version because it would leave a bad taste in their mouths. But we kept slogging and gradually things started to get better.

The turning point was early March 2016. After many usability / look /feel meetings we were slowly getting there. Things started to fall into place. It was looking the part and after several iterations the interface was starting to behave the way you expected it to. Preparing for a conference in April I exclusively used the new version. Using it in anger for the first time it was clear that this was something really special. All of the hard work was starting to pay off. Things that only lived in our imagination for a year were now right there on the interface, and it was working exactly the way we envisioned it. It was fast - terribly fast. And slick. And it handled almost anything I could throw at it. There would be no going back to Chlorine ever. It was time to set a date for the final release.

The date was set to be the first of May 2016. But that was a Sunday so we went with May 2. This was a public holiday in South Africa (and in many other countries) so we went with Tuesday May 3. Now we had to tie up all loose ends (memory leaks, branding, testing/fixing/testing/fixing). We contemplated calling the new version Plutonium, but this release was so different to anything we've had in the past that we decided it would be easier to just go with 'Maltego 4'. We sent out betas to a select group of trusted users. The feedback was phenomenal. They loved it.

We made a 'camera-ready' release  on the 26 of April and I flew to Cape Town to go make the release video. We shot an afternoon, an evening and the next morning and I flew back to Gauteng to edit. After some hiccups the final edit was ready on the Sunday before the release.

Today is Monday. Tomorrow we release. A brand new website, a brand new product. The release is not perfect. There are always things we want to improve and there are most likely a few minor bugs that we'll squash over time. With a system as complex as Maltego it's almost impossible to achieve perfection and I have to constantly remind myself that nobody cares about Maltego as much as we do. It's a child we all raised together as parents,siblings and a crazy uncle.

Some other stuff we probably need to say

Maltego 4 comes in two commercial flavors. Classic (the standard version) and XL (the pro version). The *only* difference between the two is that Classic is capped at 10 000 nodes. Oh wait - and the price - Classic is still $760 and XL is $1800. We had lots and lots of discussions about the price. We haven't raised the price on Maltego for a long time and we didn't want to raise the price for the new version. So we decided to split it into two products (we've been wanting to do this for a while now). We then had to decide what's in the XL version and what's not. An easy out would have been to exclude collection nodes from the Classic version. But collection nodes are super useful - even when working with small graphs as they quickly show you where you need to (probably) look - NOT at the collections. So collection nodes stayed. Then it was crippling Classic in some way...but that just felt wrong and so we didn't. Every time we thought about taking things out of Classic we cringed. Finally we decided capping the total number of nodes in a graph. But where to cap it? We decided on 10K nodes because of two reasons - the first being that in the past, working with 10K nodes would be painfully slow - so - we weren't taking functionality away from anyone...as they never had it. Secondly the slider was always maxed out of 10k - it didn't make sense to have it at a lower number. 10K it was.

Still more stuff

Maltego Chlorine users will be able to simply download Maltego 4 Classic and activate it with their license key. No upgrade fee to Maltego 4. Users that wish to upgrade to XL should just pop us an email.

Then there's the question of the community edition. Ye - we're no longer supporting it and we'll be removing it from our site. Hehe.. no. Give it a bit of time. We'll create Maltego 4 CE and Maltego 4 Kali soon. No really. We will. Currently the CE versions are still using the old tech.

And finally..

One last thing. CaseFile. The one we always leave behind at the bus stop. There's good news. With Maltego 4 being so totally amazing we're making CaseFile completely free. No registration. No nothing. Just download and use. And in time we'll upgrade CaseFile to goodness of collection nodes, large graphs and a face lift.


Right, that's about it. We're super excited to see what you think about all our new tech. It's been a long journey and we're really pleased with our progress. We hope you're too!

RT and the rest of the (tired) team.