I’ve always wanted children. But as a gay man, I long ago realized fatherhood might not ever happen for me.

Let me explain: I’ve wanted kids since I was no older than 5. One of my first memories is of telling my mother about my future wife and all of the children I planned to have with her. I would have at least 10 children, I told her, of all races, ages, and ethnicities. I wanted a Brad-and-Angelina-type family before Brangelina was ever even a thing.

Of course, not having a full understanding of the “birds and bees” at that age, I didn’t realize that to have my own biological children of all races, I would have to be a pretty promiscuous husband. Not sure how my make-believe wife would have felt about that.

Fortunately, I never had to find out; among the other things that 5-year-old me didn’t know at the time was that I would grow up to be gay.

When I came out to my mum at 17, she still had a 5-year-old version of me in her mind. How could her son, who had always dreamed of having a wife and children, be gay? If she’d really been paying attention she might have noticed that I started dropping the “wife” part out of my fantasy a long time ago.

My mother made it quite clear (in not the most amicable terms) that my sexuality would make it impossible for me to become the family man I’d always dreamed of being. For better or worse, her message stuck. As gay men, many of us learn this lesson all too young: Families are not in our future. Eventually, many of us find peace with the idea and move on.

In my early 20s, I put the idea of becoming a dad out of my mind, and instead began immersing myself in the gay community. It was rare to even hear mention of the word “children.” My friends from childhood, meanwhile, began to get married and have kids of their own. Before I knew it, it seemed everyone I’d grown up with had settled down and started a family.

Yet here I was a single gay man, spending more time in bars and clubs than is socially acceptable. I realized I was jealous of my childhood friends. Because of my sexuality, I thought, I would never get the same opportunities.

It wasn’t until my late 20s when my friendship circle began to mature that I started noticing gay couples with children. I was 24 when I began playing for the Sydney Convicts in 2012, which is an inclusive rugby team. During games, I started seeing dads and their small children on the sidelines. I began noticing gay families enjoying the weekend together in Sydney’s suburbs. These moments gave me hope — perhaps the idea of a gay man having children wasn’t so crazy after all.

At the same time, we began seeing amazing progress in the LGBT community. Though Australia, my home country, has unfortunately yet to legalize same-sex marriage, many Western countries have done so. This global progress has brought validity to our relationships and families. We definitely have a long way to go when it comes to the ability for gay men to have children. But I’ve been humbled watching our community grow and strengthen in the short time since I’ve begun living my life as an out gay man.

Today, my childhood dream of becoming a father seems like much less of a fantasy than it once did. Of course it’s possible for gay men to become dads; once you scratch the surface, there’s a multitude of options available. It’s simply a matter of deciding which path to parenthood is right for you.

But once instilled, the idea that gay men can’t be fathers doesn’t just go away; I long ago accepted the fact that I might never have children. Despite all the recent progress, it’s something I’ve already come to terms with. I’ll have opportunities to do other things with my life. Maybe I’ll travel the world or buy a Ferrari or Harley Davidson during a midlife crisis. I’m already an uncle. Maybe that’ll be enough.

Then again, you never know what life will bring.

So if you happen to pass a 50-year-old me playing rugby with some little ones in the park, you’ll know my childhood goal was achieved. But should you see me childless and sailing around the French Riviera? Don’t feel bad for me.

I’ve already made peace with either outcome.