On the other hand, sugar is much more likely to be the culprit for your breakouts—and all milk products have sugar, even the plainest and most fat-free. And what happens is when brands remove fat from yogurt, they often will add other flavorings and toppings to make up for it. These are usually loaded with sugar or artificial flavoring.