“The main thing holding me back from agreeing to the divorce is how serious it is in halacha and it’s so bad that it causes the “Beis HaMikdash to cry”, whatever that means.”
That’s not a halacha. If halacha is what you’re worried about,
you can put that out of your mind.
“But that’s pretty serious. I also have this image of Hashem asking me why I gave up, when I die, and showing me why I should have stayed and fixed it.
I’ve been thinking about it and maybe divorce is the only solution.”
That’s the important question. If divorce is the only thing to do
in your situation, Hashem will not take you to task for it.
“I will speak to a rabbi as well but want some feedback on here.”
If you want to know more about the halachos of getting divorced, such as what rights you have before accepting a divorce, and what rights you have afterward, you should definitely speak to a rabbi.
(A rabbi who is familiar with you and your situation as a couple may also be helpful in deciding whether or not to divorce. A rabbi who doesn’t know you might also possibly be helpful, if he is experienced in these matters.)