I have the same issue bro and I went to a urologist about it. He told me its an issue that affects many young men particularly if you are right handed as the penis rests on the left side in the boxers. So the muscles on the left become stronger. But with time as one begins to be sexually active the muscles on the right of the penis strenghthen.