I went from B5 S4 to RS4 and don't regret it at all. But there is no comparison. The RS4 does not have the same low end that the S4 had but it has an amazing engine. Better than my ferrari (apart from the sound). That said the RS4 is faster than both the BMW and the S4. The handling is also completely different from the S4, much easier to rotate around a bend.
Last, but certainly not least it has that Audi interior which is probably better than any other car. I got the Aluminum trim, I think the CF is a little over the top but either way real class inside.
There are great deals on the RS4, $10K under list but that has got to hit the residuals. Check further down, 07 RS4s are trading for $48-50k right now. Having driven, but not owned, the 335i I would not hesitate to say the RS4 is way better.

2008 Mugello Blue RS4 that replaced a
Silver 02 S4 that replaced a
Silver 98.5 2.8 A4Q.
titanium 01 360 F1
Blue 00 MX5

"I spent a lot of money on booze women and fast cars, the rest I just squandered" George Best. R.I.P.