If your smartphone still gets system and security updates after 3- 4 years and having no above-mentioned issues, no need to go for a new one I guess.
I'm using my phone for the past three years and have found no issues so far.
Battery plays a major role in any electronic device, the times when we charge our battery from 0-100 at one go, these are the times when the battery element gets continuous heat and result in swelling in some cases if it is old and the same process is repeated ample times. And considering battery as a heart of the smartphone, if battery gets some issues then the smartphone will eventually end up functioning slow, sudden shutdowns and many more strange incidents.
Software Updates: It is well said the average life span of a smartphone is 2.5 years as after that not all smartphones are capable to cop up with the current updates and in some cases even the smartphone brands don't care of their old customers or old smartphone series to send a update and this will also result it some bad incidents coming up with your smartphone. As no matter if the smartphones sends an update for their OS or not, but the applications on PlayStore or AppStore will be updated with their latest updates and at one point of time your smartphone will not be capable of handling those applications smoothly.
However, this is noticeable in today's materialistic world that people change their smartphones considering the TREND which should not be the priority, and the answer to your question in my words would be, as far as your smartphone is capable of doing your regular tasks comfortably you should not update to a new one.
I find that the harder I work, the more luck I seem to have.