Vbrake rubbing on Black SPV fork

I've been messing with the hardtail again, trying to get it trail ready for kicks. I put on my black spv fork and had to go to full V brakes because my disc wheel went to my son.

When i stand up and lean the bike to the left, or take a left corner or track stand (with wheel turned to the left) my front V brake deflects so much that the pad rubs the rim. So much so that it messes up my trackstand because it practically stops the front wheel from moving.

I just can't remember so far back when I actually ran it that way with V brakes on the front to recall if it was this bad. I tried another front wheel and it still did it so I'm thinking it is something with the fork lowers.

Won't mess with it too much because Vbrake on the front is so bad that I must go Disc. Rear I can live with a little bit.