In the bowl of a stand mixer fitted with the paddle attachment, cream together the peanut butter, butter, brown sugar, and granulated sugar until light and fluffy. Beat in the egg and vanilla until smooth.
Use a fork to press the criss cross shape on the cookies, pinch the middle to ensure the peanut shape stays. Repeat until all dough has been made into peanut shapes, making sure each cookie is at least 2 inches apart.
Add half the powdered sugar and mix until smooth, add 2 tablespoons of milk and mix well. Add the remaining powdered sugar and beat until fully mixed. Mix in the vanilla extract, and add additional milk a teaspoon at a time until desired consistency is reached.
To assemble, flip half the baked cookies so that the bottom is facing up and pipe or spread a generous amount of peanut butter frosting over the surface. Place a second cookie on top and gently press down to create cookie sandwiches.