making a Wordlist (for bruteforcing)

[hypn]

I'm tired of downloading crappy wordlists, and having to merge them, and remove duplicates, only to end up not getting the password, so I've decided to make my own wordlist.

I'm starting with Wikipedia... taking a recent dump, stripping all the HTML code out, and separating on spaces - I figure that's a pretty good starting point (I'll have every word, name, surname, country, city, band name, etc that is currently in Wikipedia)... I'll also have a LOT of (pretty much useless) numbers. I'll obviously filter out all dupes.

My plan is to convert everything to lower case, and then start manipulating it, so I'd like to know if you have any suggestions (not just "things I can do" to the words, but what will make the list more effective). So far I'm thinking:

1. make letters uppercase 1 letter at a time, incrementally (so each word will have every possible combination it could be typed using upper or lower case letters)

2. convert words to "l33t" speak (swapping in 1s, 3s, etc) incrementally (so once again, it's every possible combination, using numbers in places of letters, for the word)

3. add 3 character repetitive numbers (eg: "111", "222", "333", etc) and letters ("aaa", "bbb", "xxx", etc) to the beginning and ending of each word

4. add a number (from 1-1000) to the beginning, and end, of each word

5. incrementally add a punctuation characters (.,!@#$%"') to each word (I'll probably strip all punctuation out of my starting list, so they don't get in the way).

All of the above steps would leave the original word in the list, and would create all the possible variations of that process.

Once that's done (and any other suggestions you guys might have), I'll filter out any duplicates that might've snuck in (adding "aaa" before "aa", and adding "aaa" after "aa" would produce a duplicate word), then start joining words - so every word will be put before/after every word in the list. I'm trying to decide how many words I want to join (obviously this makes my list exponentially longer). Because of the puncation I'll be adding, words will be joined with spaces, underscores, etc.

So apart of "strong" passwords (consisting of random characters and letters), it should appear in my wordlist.


Because the words "george", "w", and "bush" would appear in my starting list, at some point it would end up creating "George W. Bush" (upper casing, punctuation, and joining - assuming I was joining 3 words). It should also do a pretty good job providing the Most Common Passwords.

If you don't know what a "wordlist" or "brute forcing" is, please don't reply to this thread.

[MasterOfChaos]

you can't include all combinations of 1 to 5, because that are roughly several millions to a billion per word. If we go with the low estimates one million variations of a word and 100k words, your list would contain 10^11 entries. Which are several hundred GB. Even if you do the variations on the fly it will take very long for any good protection.

[hypn]

Someone made a wordlist from Wikipedia back in 2005 (Generating a word list from Wikipedia - Programming stuff) and it came to 1,035,166 (1 million+) words... which isn't even 10mb when extracted (his wordlist is available for download from that link).

You're right though, doing steps 1-5 would definitely generate a wordlist WAY too big... I should've done the maths. So then the question is... which are the best steps to do. Just pairing the words in the list, with the words in the list, 3 times (to make "catcatcat") would turn a 20mb wordlist into a 156gb wordlist. Sigh.

Would be cool if someone had a wordlist a couple of terrabytes big, and some serious processing power, that you could rent. If I were rich, I'd set something like that

[Dyndrilliac]

Umm, this has nothing to do with programming?

[Shin666]

Instead of going the route of physical, i prefer psyche...Social Engineering ftw

[hypn]

Dyndrilliac: well it sorta does, more so than "game hacking" anyway, cos I'm going to be making a program to manipulate my wordlist? :|

Shin666: yes.... I'm going to socially engineer ZIP files, and encrypted strings...

[Raistlin]

This is going to sound like I'm being a dick but it's an honest thought. Why not try to get in without brute-forcing? Random example, a couple of years ago I was into an rpg and I ran into a guy who had an item I wanted. Instead of spending time trying to brute force my way into his account I befriended him. After knowing him for a few weeks I said I was thinking of buying the same item and asked if I could try it out. At which point I stole the **** out of it. Yea I know, I'm a creep. So what.

Of course if you have grand schemes for what you are going to do with this, that won't work. But if that is the case, I can also suggest that you pursue what you want legitimately instead of taking what seems like the easy road. If it costs money, use your brain to figure out a way to make that money. It may be easier in the long run.

[hypn]

Raistlin: nice nick (I think we've had that talk quite some time back), but like I said to Shin666 dealing with people isn't really an option (such as cracking ZIPs that you don't know who made, breaking encryption, etc), plus I figured I'd make the world's best wordlist and save everyone countless hours... or something.

LOL @ you befriending the guy and stealing his stuff though ;)

[Dyndrilliac]

Originally Posted by hypn

Dyndrilliac: well it sorta does

No, it doesn't...

Originally Posted by hypn

more so than "game hacking" anyway

How so? I don't see any discussion regarding the program to manipulate the list, just the list itself. At least with gamehacking, you have to find a why to implement ytour own code within the game, which requires a programmatic means.

[Shin666]

Originally Posted by hypn

Dyndrilliac: well it sorta does, more so than "game hacking" anyway, cos I'm going to be making a program to manipulate my wordlist? :|

Shin666: yes.... I'm going to socially engineer ZIP files, and encrypted strings...

you could ask the maker of the zip files for the password?

[Raistlin]

but like I said to Shin666 dealing with people isn't really an option (such as cracking ZIPs that you don't know who made, breaking encryption, etc),

Yea, I read the thread before you posted that and was writing a reply already.

[Jakor]

Go ahead, throw the rules at me for this one....

As you found out, making a word list as big as you first described in order to brute-force would have been worse than brute-forcing no words. However I have a few things I would suggest, and I have a project that I am going to use some of this on.

Let's keep in mind that we are "hopefully" not going to run through the whole list and will end up with the correct password at somepoint. First, order the words from small to big, smaller words will most likely makeup the password than giant words. We can then also pass a minimum/maximum password length to our engine so we know when to stop in our list. Add all the extra crap on the fly after searching for the basic words, they don't need to be in the wordlist. Offer adding the extra crap for smaller words that don't meet the minimum size requirements. Offer combining words on the fly so long as you don't go over the maximum size. Offer filling in with extra even on the combined words. If you do this all on the fly, you can set the order you check things in easier and try to get a match sooner than going through the entire wordlist. Make sure the list is either only lowercase or only uppercase (make two copies of the wordlist if you want). If you need all the different possibilities of capitalization, do it on the fly.

Always assume that the password is going to be the simplest first (lowercase/oneword). The fact that you get to use a wordlist in the beginning means that there is a lack of security in choosing the password.

[hypn]

Thanks Jakor, some good points in there.

I guess most cracking programs these days support word manipulation (numbers/caps/combining) so a smaller list will do.

Can't wait for the day when CPU's will be so powerful that we'll be able to brute force up to 100 characters in a single day... :P heh.

[HARD_ON]

How about grid network processing? I use Xgrid, Qmaster, 3Delight and things of that nature for rendering graphic clips over my internal home network, as well as over the Internet to a few friends with dedicated CPU's.

[hypn]

hmmm.... I suppose with something like Hamachi you could easily create quite a nice cluster :D