Preheat oven to 400°F and line a baking sheet with parchment.
Roll out puff pastry and cut into rectangles about 4½” x 4½”. Make sure you have an even number for tops and bottoms.
On each bottom rectangle, spoon ~2 tablespoons of filling, leaving a 1-inch border.
Brush edges with egg wash, then place matching top piece over filling.
Use a fork to crimp edges and seal. Poke 2–3 small holes in the tops with a fork.
Place on baking sheet and bake 15–18 minutes, until golden brown.
Let cool. Mix powdered sugar, milk, and vanilla until smooth, then drizzle over pop tarts.
Once cooled, store them in the freezer and simply pop one in the toaster in the morning!