Even though physics constantly applies the rules of mathematics, it’s two different sciences. That’s why this question can’t have a definite answer. Some people find physics much more manageable than math. Others have no idea how to explain the natural rules, but they can quickly solve any equation. It depends on your predispositions.
You can’t compare it because it is the same as if you compare music and art. Probably, for some people playing the piano will be much more complicated than learning natural phenomena. But it doesn’t mean music is more challenging than physics. There is no general answer to this type of question.