I'm 5'10, and 130 pounds. At the moment I ride an 1100 pound Ultra-Glide
Like they say, size does matter..
You're not going to carry this bike, you're going to BALANCE it. Don't let the engine size scare you off. 250's are fun little dirt bikes, but worthless on the road. Dual sports in the 350+ range are a good compromise, and street bikes less than 650 should be outlawed.
I wont bring brand into the equation, but you need to go test drive some bikes. Find the biggest bike you feel comfortable on, but don't buy that one. When you find that (250, 500, whatever) bike, start looking at the next size larger. Even if you don't feel quite comfortable on it, you wiil grow (and quickly outgrow) into it soon. Take it from a guy who has lost his ass on a ton of wrong sized enduro bikes.
Oh, and just so you don't think I'm some sort of freak, my 105 pound wife rides her own Harley too. And no, it's not one of those girly Sportsters..