Really think I want to switch doctors it just sucks I would have to drive an hour away for my…

Really think I want to switch doctors it just sucks I would have to drive an hour away for my appointments. The doctor I have now I still have anger towards. When I had my miscarriage in May he came in to tell me and he said "there is nothing you did wrong. It's more common then you would think. The only way you could have prevented it was to have an abortion (which I am completely against) or to stop doing drugs." Before I got pregnant The first time i smoked weed. And I stopped smoking two weeks before my boyfriend came down to live with me. I did heroin but I quit December 23rd. So please tell me how I was doing drugs when I got pregnant?? Am I wrong to want a new doctor because of this??