Nervous about what you should and shouldn’t eat and drink now that you’re nursing? You shouldn’t be. There are just a few tricky foods you need to be cautious about.

You may have heard a rumor that you need to avoid certain strongly flavored foods while breastfeeding, such as garlic or spicy foods, but there really aren’t hard and fast rules about the tastes you can enjoy. Remember: Somewhere in the world, a breastfeeding mom is eating that garlicky or spicy dish, and her baby is just fine. So don’t worry about flavors, and instead focus on the few, rare foods you do need to be cautious about. The good news? You may be able to enjoy them in small amounts.

High-mercury fish

Virtually all fish contain some mercury, a common pollutant that’s a known neurotoxin. (That means it can affect baby’s brain — yikes!). But most of the time, the health benefits of eating fish — high protein, low fat — outweigh the risk. Most fish, after all, only contains trace amounts of mercury. The exceptions (and the ones you need to avoid) are shark, swordfish, tilefish and king mackerel. These older, top-predator fish contain the highest levels of mercury, so eliminate those foods from your diet completely. Instead, feel free to eat about to 12 ounces (that’s about two meals) per week of fish that have lower mercury levels, such as shrimp, canned light tuna, salmon, pollock and catfish.

Alcohol

After taking nine months off of drinking, you’d probably like to have an occasional beer or glass of wine, and that’s totally fine. It’s even okay to nurse your baby after you’ve had one glass of wine or beer (but, really just one). “Think of the DMV guidelines for drinking and driving,” says Jennifer Ritchie, IBCLC and author of I Make Milk…What’s Your Superpower? “If you can safely drive a car, you can breastfeed.” If you’ve had even a bit too much, wait until the alcohol has cleared your body before breastfeeding. If your breasts get uncomfortably full, you can pump and dump the milk. Otherwise, there’s no need to pump and dump; the alcohol won’t accumulate in the milk in your breasts (contrary to popular belief). By the time your blood is free of alcohol, your milk will be too.

Caffeine

Good news if you need a jolt after all those late nights you’ve been spending with baby. Caffeine is okay too, in moderate doses. So go ahead and enjoy your morning coffee, an afternoon tea or even the occasional soda. As long as you limit your caffeine intake to no more than two or three cups of a caffeinated beverage a day, you and baby will be just fine. Know that caffeine passes into your bloodstream, which means it can get into your breast milk, so having more than those couple cups of coffee could make baby jittery and more prone to wakefulness, so don’t go crazy with the java.

Peppermint

You probably don’t eat a ton of peppermint anyway, but a word to the wise: High doses of peppermint have been known to inhibit milk production. In fact, peppermint is an herb that’s sometimes used by moms who are ready to stop breastfeeding and want to dry up their milk supply. But don’t worry about the occasional cup of peppermint tea; you could probably even eat your fill of candy canes before you’d notice any adverse effects. But if you notice your supply dropping off after you’ve ingested a bunch of peppermint, it’s best to lay off for a while.

Sage

Think of sage the same way you do peppermint. This herb can decrease milk supply, but usually only when ingested in large doses. So you don’t have to tell Grandma to leave the sage out of the Thanksgiving stuffing, but don’t have a heaping helping of the spice.