There is not a right age to let go of your kids. I met a woman whose kid is 18 and moving out, and it is very hard for her to let go of her "baby". Even when our kids would be married, we do become the insufferable in-laws and "inquire" if they are being treated right, are they eating well, etc., etc.
While some fathers refer to it as "being a control freak over your kids", I think it is just a natural drive that mothers have, that is to watch over their kids, that is a bond that is hard to break.
It seems unnatural for a parent to have to let go of their kids, specially at a young age. That does affect the children and parents at one point.
As I said before, I feel impotent when my child leaves my house (every weekday for 2 hours and every other weekend overnight) - I know there are some things I do with him, it is not assimilated on the other household (and why should they be, right?)
I hate the tears he brings home, the separation anxiety that has developed from it, just not knowing what is going on with him during that time.
This interchange begun when my son was born, my mistake was to allow the daily visitation because the dad and I lived very close to each other, so the commuting was quick, and felt sorry because the dad wanted to spend time with his son.
When the battle for custody started (dad wanted more time - 50/50) I was bounded with the schedule, because it would be very hard for a judge to give less time than what there was established already (my advice to you, don't ever jeopardize the well being of your child because you feel sorry for someone, IT IS NOT WORTH IT, YOUR CHILD IS). So, I had to learn to live with it. I had to, was my toddler learning to live with it?
It has been 2 1/2 years, and the separation anxiety continues (court pays no attention to it, unless it is extreme) from my son's part, the crying still happening, and that brings the sense of impotence in me.
What to do?
Trust and draw base from what you know. I know my son's dad loves him - we have VERY different styles of parenting - but the love is there.
Your child is developing fine despise the interchange. I am a firm believer in God and His plans for our lives. I know He has plans for my son. I know that storms are necessary to appreciate the clear sky. I know my son will have to go through hard times, because that is just life, and I know God will be and is there for Him - even in his parents shortcomings.
Don't go insane.
If you suspect abuse though, act immediately.
Keep your eyes open, never assume, always ask when in doubt and be there, ready to talk and comfort when your child needs it.
Cut yourself some slack also. We are human, we error, people error - learn to let go of the unimportant and focus on your precious family you have with your kids. Don't live in anger - the past doesn't matter.
PLEASE, don't give into "seeing the other parent" on the child - if you are angry at the other parent, then block them from your life, focus on your child. Your child is a unique individual, that needs to be form, corrected, taught. He or she has his/her traits, good habits, bad habits, his/her personality - don't look for the other parent on them, don't compare them - they are them - an individual - that comes from two sets of genes to have his/her own - that is precious and worth celebrating.
When you feel impotent, for whatever reason, past, present or future, just look at your child, you will draw encouragement and strength - trust me - we are mothers, God has made us one of the most powerful human beings in earth :)
Blessings to you and your kids - kiss and hug them forever!