The difficulty of a language to learn depends entirely on your native language and what you've already learned from cultural context.
Despite reconstruction, Japanese and English are still mostly alien to each other since they do not share a common linguistical root. For example pretty much all the European languages derive huge amounts of their vocab (and syntax) from latin. If a German or Spanish person were to try and learn English it would be much easier to learn than Chinese or Japanese since English is a weird mishmash of all the other Latin derived languages. They share many words and the sentence structure is very often similar. All of that changes for Chinese and Japanese. Chinese and Japanese are derived from some sino- tibetan root that predates written language (I think, I'm not an expert, but I know its not latin). To make matters worse, china and japan were literally as far from Rome as it was possible to be at the time and they had almost no direct trade so their cultures were almost entirely isolated from each other until Marco Polo wrote about China and the Portuguese missionaries settled in Japan.
All this to say, English is not inherently difficult to learn - its just harder for people from asian cultural spheres to learn.