If you are trying to lose any weight, remember that coffee causes your body to store fat (esp. in belly area) in order to protect itself from its highly acidic content. It can also cause you to crave more fatty foods.
Coffee (all cafeine for that matter) also messes with your blood sugar (leading to food cravings), and can even lead to a level of insensitivity to insulin, which we all know can cause another buttload of problems.
IMHO, the effects of caffeine/coffee aren't adverse enough to completely abstain from it, but if it's more than an occasional treat, it might be beneficial for you to kick it to the curb... The fewer addictive foods in your diet, the cleaner and healthier your body will be!