It depends a lot on your interests. If you are more drawn towards abstract thinking, it may be better to start -like me- the math and learn physics along the way, while if you are more drawn towards understanding physical phenomena, it may be better to start -like most physicists- with physics and learn math along the way. In the end you'll need both anyway for a thorough understanding.
You may also want to choose based on what you'd prefer to end up with in case you'll not have the stamina to complete both studies.
In any case, studying one subject properly (according to the syllabus of your university of choice) should not deter you from learning as much as you can about both sides of the coin.
People with different educational background and/or preferences will often develop different preferred approaches, though they learn of course the traditional ones, too. This diversity is an advantage, as different points of view complement each other.
If you plan your life actively rather than have it determined by circumstances, what you specialize on will mainly depend on what you want. Cultivating strong and well-defined interests is a definitive advantage, as it simplifies everything - choices, understanding, motivation, recognizing possibilities and open doors, etc.