Preheat the oven to 350°F and grease an 8x8 baking dish.
Combine flour, baking powder, baking soda, and salt in a medium bowl.
In a stand mixer with the paddle attachment (or using an electric mixer), cream butter, oil, and sugar until fluffy, about 3 minutes at medium speed.
Add egg yolks and egg one by one, fully mixing each before adding the next. Stir in vanilla and almond extract.
Add half of the flour mixture, mix, then add half of the buttermilk and stir. Repeat with the rest, scraping the bowl's sides as needed.
Spread batter in the prepared pan and bake for 26-28 minutes until golden brown, toothpick clean. Cool in the pan on a wire rack.
For the filling, whisk cornstarch, salt, and egg yolks in a medium bowl.
Combine milk and sugar in a saucepot, heat until simmering, then slowly add to the egg yolk mixture, whisking constantly.
Pour the mixture back into the pot, bring to a boil while stirring.
Remove from heat, stir in butter, vanilla, and almond extract.
Strain the filling into a bowl, pressing to remove lumps. Chill for 2 hours.
For the almonds, bake in a single layer on a lined sheet tray at 18 minutes, stirring every 3 minutes.
Combine nuts with water and sugar, bake for 12-15 minutes, stirring every 3 minutes.
For the frosting, whip butter until smooth, gradually add powdered sugar.
Add salt, vanilla, almond extract, and milk, whip for 3 minutes until light and fluffy.
Once the cake and filling are cooled, cut the cake in half lengthwise and place one layer on a cake plate.
Spread the filling, leaving a ½-inch border, and place the other cake layer on top.
Apply a thin layer of frosting for a crumb coat and refrigerate for 1 hour.
Add the remaining frosting and press almonds onto the cake. Serve immediately.