Just picked mine up from Best Buy and have tested them side by side with the APP.
ANC - it's very apparent that it is better than the original AirPods Pro. No question that there's improvement there. I did the bathroom test by peeing twice with the same volume setting while listening to Whitney Houston's How Will I know.
Original APP, I can still hear my pee hit the water along with the toilet flushing. APP 2022, can't hear a thing.
Transparency Mode - there is noticeable improvement. I can hear my wife yell at me a lot better when the baby is crying while I am playing around and comparing the headphones a lot more clearly. With the original APP, Transparency is a bit muffled while listening to Usher's "Let it Burn" at mid volumes
Overall Sound Quality - APP2 has punchier bass it was the first thing I noticed right when I fired up No You Girls from Franz Ferdinand. The first AirPods Pro already sounds very good for a pair of earbuds but the APP2 sounds better on music in general I think they are just tuned better.
Now whether or not you want to spend $249 if you have existing AirPods is up to you, but it was definitely a much worthier upgrade than going from the iPhone 13 Pro to the iPhone 14 Pro.