overall it might be less stiff but when going downhills from 3rd to 2nd sometimes is not possible as it requires huge tug. Sometimes it is smooth so it is not consistent. The old gear oil that was drained from the tranny was very dirty and full of metal pieces. Do I need a rebuild? was it worth changing the fluid at this point?

Its possible that you have a syncro going bad. Old thick oil will sort of act like a buffer and a syncro kinda of gluing the gear and the syncro together making the syncro spin before the gear engages it. by changing fluid you removed the "glue" and it works like its suppost to making the worn syncro show up more.