"The stories range from running rich to losing a lot of HP to small gains. The car was dyno run after sitting at about 1.5 to 2 hours before each pull.
I think that if we had run the car with the stock ecu and immediately replaced it with the Mopar ecu the Mopar ecu would have yielded lower numbers due to some heat soak - that's why we waited between the pulls."
It is my belief that this is an inaccurate way of measuring- Heat soak can be an issue, you are correct. But, to see what the real results are, you should do a pull at operating temp (coolant, intake, and oil temps), swap ECUs, another pull and swap back and one more pull. This is the only definitive way to see what the real gains are. Most people don't start their cars, then immediately beat on them. Pulls at operating temp are the only way to see real differences. An LS1 vette, for example may test way better with a hypertech programmer when cold. Now, make that pull at operating temp vs. a stock pull at operating temp and you'll very commonly lose power due to the added timing- which turns into knock retard, thus loosing power. Oil temp can skew your readings also as cold oil (room temp) is heavier and takes more power push through. The best readings you can get is from a cool intake and warm oil (dyno tricks...). I always do back to back testing with products- it's real results. Whenever we tune on the dyno, the cars are ALWAYS tuned at a consistent, normal operating temp. If the customer chooses, then we make a "cold" pull at the end.
Just my $.02.