Simmer the milk, heavy cream, lemon peel, orange peel, cinnamon stick, and vanilla extract over low heat for 30 minutes to infuse the flavors.

Cut the brioche into thick slices.

Place the brioche slices in a dish and pour the infused milk mixture over them. Refrigerate.

Sprinkle ground cinnamon to taste and serve with ice cream if desired.

Heat a skillet with butter and cook the soaked brioche slices over low heat, sprinkling brown sugar on each side.