Is it beneficial to grab a lighter weight right after a set and push for 4 or 5 extra reps, or is that counterproductive?
That’s called a drop set. You should do it on the final set to maximize your effort.
It’s a great technique for building muscle, known as drop sets.