When is it ok to 'move on'

Here's an interesting question:when is it ok/expected to realize that you don't get to raise your children in a full-time family situation, and start living for yourself?

Five years ago, after my divorce, I chose to keep the house my boys had lived in for four years under the premise that I'd keep some familiarity for them after they were taken to another state.
Now, most would say that this was unnecessisary, but for whatever reason, it's what I did.

For the following five years, I chose not to date, not to associate with new people, and not pursue a few options that arose. All of which was influenced by what had occured with my now ex-wife.

The part that makes sense, I assume, is that I gave up a lot of personal wants in an effort to save all the money I could so when I did get my kids, I'd be able to spend it on them, i.e. going out to eat once in a while, going to movies, taking trips. Even though I wasn't able to save a lot, which was not because I was splurging on myself, but from the lack of money I was bringing home, I was able to do somethings with/for my children.

Now it's five years later, I've passed on a chance to get married and move to Southern Wisconsin, only fulfilling one vice, and basically stayed where I am now that my children are back in the same town. But did I make the right choice?

Now that I'm pushing 40, I find myself wondering a LOT about where the last five, hell TEN years have gone. I wanted to do so much.
I wanted a family, (kids AND a wife), I wanted to be comfortable income wise, hell I wanted a Harley by the time I was 35 and I probably could have done it, but I chose to live for my children, even though I only got them part time, instead of doing for myself.

I was reminded of my monetary situation the other day when I went shopping with my boys and my eldest wanted a toy. Cool toy too. But, because of a shortage of funds, which is pretty much the norm for me, I had to say "no". And while he understood why, he still teared up. But I couldn't be certain if he was upset because I couldn't get it for him (which by the way, he pointed out that it was for both he AND his brother...........he's such a good kid!), or because he was upset about the lack of money. As I said, he's a good kid BUT also a smart kid. He understands about child support and how I pay it, and his mother spends it...............not always on them (the boys).

So when do I realize that I have a part-time family and begin to attempt to live for myself and 'not' so much for my children?

It's like I have to realize that my life is not, nor will be that of normal people due to the fact that I love my children TOO much combined with my allowing my ex to always get/keep me down.

*I fell like this has become a incoherant rambling of thoughts so if you comment, be gentle! lol