Should I cut out the dairy?

I have been drinking some 0 fat/low cal yogurts and skim milk but I was wondering if I should cut them out my diet. I have 20 pounds left to lose and I wanna lose them quickly but I am a sucker for exercise!

What do you think?