“…why Disney would pay $4 billion for the Star Wars franchise only to destroy not just its central character, but its central mythos, in the most callous way possible?”
I think this question perfectly reflects the massive disconnect between fans and critics of TLJ. The fans who enjoyed the film saw it as a intriguing twist in a much longer story arc that can’t rely on the Skywalkers to carry it any further. I don’t agree with them, but I do get where they’re coming from. The unexpected “twist” in Luke’s story arc certainly did manage to take the focus off of him, and that had to happen so that Rey, Finn and the rest of the gang could have the stage all to themselves in Episode IX. On the other hand, it didn’t have to be executed in such a divisive way. There were other options on the table they could have gone with.
And to be fair, I find it hard to believe that Disney meant to alienate such a substantial portion of the audience. I don’t think that was their intention. I think the more likely explanation is that Rian Johnson and the writers were more interested in leaving their collective mark on the franchise than they were with honoring and preserving the original mythos (hence the whole Canto Bight portion of the film, which was obviously included for purely political purposes, as it added nothing substantial to the story itself). If they wanted to throw a curveball at the audience, they could have done that without completely emasculating Luke and giving him such an underwhelming exit from the series. Granted, Disney had to approve of the plot, but I don’t think that would have happened had they foresaw the controversy that followed the film’s release.
That said, I wouldn’t give up on the (cinematic) franchise as a whole just yet. I agree with a lot of what you’ve written here, but there’s still a chance that J.J. Abrams and his team could find a way to smooth things over with TLJ’s critics while still moving the story in the fresh new direction that TLJ’s fans seem to be excited about.