Preheat oven to 400° F (200° C). Place drumsticks in a baking dish.

Whisk water, ketchup, vinegar, brown sugar, butter, salt, Worcestershire sauce, mustard, and chili powder together in a bowl; pour mixture over drumsticks. Cover with aluminum foil.

Bake in preheated oven until no longer pink at the bone and the juices run clear, about 1 hour, turning chicken about halfway through. An instant-read thermometer inserted near the bone should read 165° F (74° C).