I'm enjoying my rewatch but I have to say it completely goes off the rails in S5 and S6. Like it's kinda not good anymore.

Yeah season 5 and 6 were my least favourite overall. I give season 4 some slack due to the writer’s strike, particularly the non-event of Michael’s return, but it was much better.