I am in a position that is not quite the same but very similar, in that I watch family and friends having more and more children while its been a number of years and so far I only have one. I spoke to my Rosh Kollel about it once in passing, He also only has two children. He told me that the most important thing is not to obsess about it.
I try to think: I have what I am supposed to have and nothing can change that. Instead I try to focus on what I have and give my extra time and effort that would have been divided to what I do have.