Two quotes from chazal come to mind
1) you will be influenced by your neighbors
2) better the tail of the lion then head of the fox
Leave and leave quickly. Pick some place where you aspire to be like the guys there, that is a way to help ensure growth.
I recently heard a story from Rav Reisman. He was discussing a bochur that left Yeshiva and surprisingly (considering he hadn’t done much in Yeshiva) he was learning a lot. When he asked the young man what was the cause of his growth he responded, “in my neighborhood, if you don’t learn, you are nothing”.