Cewl
From charlesreid1
Some notes on CeWL.
What It Is
CeWL is a wordlist generator. If you're not sure where to begin a password search, or if you're looking for more targeted wordlists but can't find what you're looking for, this is the perfect tool.
How To Use It
Be patient.
CeWL didn't work right out of the box on Kali. But it was actually my impatience - cewl was (and is) very slow and finnicky on my machine. I also picked some text-heavy sites, which I think can slow down cewl.
I scraped a single Wikipdia page (http://en.wikipedia.org/wiki/Seattle) and set --depth=1. My CeWL command looked like this:
$ cewl -v en.wikipedia.org/wiki/Seattle -d 1 -w seattle.txt
This process took about 5 minutes (not so great) and I got a 6,300 word file resulting (great).
Input Arguments
A word on the input flags - sometimes if you get the input flags wrong, or put them in the wrong order, or specify something incorrectly, CeWL might not tell you explicitly what's wrong. If your output doesn't make sense, make sure you double check those input arguments.