Well, I know everyone's bodies are built differently, but you are pretty tall. Do you really honestly think that you could lose 20 lbs? Just because you may have lost 20 lbs before doesn't mean that you still can do it. At your height, I'd say that you are pretty darn close to your goal. Have you had a doctor do a body fat analysis on you? It is possible that you have built up muscle mass since the last time you were at your ideal weight, and now to get back down you'd have to lose muscle mass.
I have a friend who is 6' 1" and not very broad shouldered or anything, he's got an average amount of muscle mass, and weighs 190 lbs, according to his BMI he's technically overweight, but he also has a low body fat percentage, so his doctor has said he's probably just got muscle mass that isn't visually obvious.
So I would assess with a doctor whether or not you actually have as much to lose as you say you do. Remember weight is just a number, at my ideal BMI goal I'd weigh 155 lbs and I guarantee I'd look anorexic. So just because you may have seen somewhere that you should only weigh 160 or whatever, doesn't mean that you actually should.