Hello, I really would like your help and advice please.

Following the breakdown of my marriage (now divorced with children) I met an Pakistani Muslim man at university His family have been here for n30 odd years, he was born here. We did wrong (I really know that we did) and had a relationship and then a baby. We couldnt marry as i was separated from my husband but my divorce wasnt finalised - although we spoke about marriage many times. Due to my situation and his family it was not at the forethought of his thoughts i guess. Such a complex situation.
He has since left me saying different things from one week to the next. Parents, family, my children and too much responsibility. So now I care for my new child and have reverted (I wanted too and to give her good upbringing in the faith of her father)
He sees his baby speradically, for a few hours every other week in a coffee shop.

However he then wanted to see her alone and take her to his one brothers home. This would be a two hour drive away, each way.
I have always said he can see the baby at my home whilst i work so that the baby is in the comfort of her own home with fimiliar smells, sounds and routine and of course i would be there if need be.
I cant gift wrap it any more for him but worry as to the fact that because i have said no, she is to remain in her home, he is now considering legal action!!!!
I really wish that he has a relationship with his daughter but as her father, and husband, not a dad. The way Islam guides as a parent when we are blessed with a child.

Maybe it will not get to that as his parents have never seen her and told him to have nothing to do with me (Cultural) I cant understand why they go against everything that Islam teaches regarding parenting and responsibility! Culture V's Faith!!! I took our daughter to the family home when she was just weeks old to see if actually seeing her would change their minds - and seeing me too. However i was told to leave by his mother, that I had led her son astray!! SO NOT the case. I have been so dignified and respectful to him and his family and have been treated so unfairly in return.

There are so many things that I have now learnt that he should have known as a muslim man - NOT to have started this in the first place for one, NEVER to be alone with me - the list goes on.

I am trying to do the best for my daughter and took her to see the Imam who said prayers for us. This was really important to me and the Imam is being very supportive. He cant understand why the brother is behaving like this - he is now a father himself!!

I have had no contact with her father now for about three or four weeks since he said he would go to solicitors to get to see our daughter and that I shouldnt contact him. I have NEVER said that he cant see her, only that it should be in her own home.
I am devastated as I gave this man my heart and soul and we shared so much. We talked for hours of us and our life going forwards - complicated though it would be - Allah would reward us for doing the right thing??

Please advice is really appreciated. Our baby is only just approaching 6 mnths and my divorce will be finalised in two weeks now. My father passed away so he cant go to the family for me and on my behalf. I fear anyone else will be as humiliated as i was. His family are practicing.
Thank you