According to J.J. Abrams, Director of Star Wars – The Force Awakens:
We knew we had to do something fucking bold… The only reason why Kylo Ren has any hope of being a worthy successor is because we lose one of the most beloved characters.
I absolutely agree with this statement. It was the boldest thing done in the movie, and probably one of the only ways the franchise could move forward instead of back. From a plot perspective, this scene was absolutely necessary as it sets up a lot of different character motivations for the new characters introduced, and we’ll see a much darker Episode VIII with a worthy resolution in Episode IX