Put Graham crackers, brown sugar, granulated sugar, butter and milk into a saucepan and bring it to a boil. Boil for 5 minutes.

Put a layer of crackers in a 9 x 13 - inch pan and pour 1/2 of the mixture over it. Put another layer of crackers and pour remaining mixture. Add one last row of crackers.

Melt, over low heat, the butterscotch chips, chocolate chips and peanut butter. When melted, spread evenly over top. When cool, cut into desired bars or squares.