Prepare the Peach Topping. In a saucepan, melt the butter and brown sugar over medium heat, stirring until smooth and caramel-like. Pour the mixture into the bottom of a greased loaf pan, spreading it evenly. Arrange the sliced peaches in a single layer over the brown sugar mixture, covering the entire bottom of the pan.
Make the Cake Batter. In a medium bowl, whisk together flour, baking powder, and a pinch of salt. In a separate bowl, cream the sugar and butter together until light and fluffy. Add the eggs one at a time, beating well after each addition. Stir in the vanilla extract. Gradually add the dry ingredients to the wet ingredients, alternating with milk, beginning and ending with the dry mixture. Mix until just combined.
Assemble and Bake. Pour the cake batter over the peach topping in the prepared pan, spreading it out evenly with a spatula. Bake the cake in a preheated oven at 350°F for 55-60 minutes, or until a toothpick inserted into the center comes out clean.
Invert and Serve. Once baked, allow the cake to cool for 10 minutes in the pan. Carefully run a knife around the edges of the cake to loosen it. Place a serving plate upside down over the pan, then invert the cake onto the plate. Let the cake rest for a few minutes to allow the topping to settle. Slice and serve warm!