In order for someone to read and navigate your site, if it just looks like a mini version of your desktop page, they have to *pinch* to enlarge, and zoom around to see the page. Or they could try to read it by turning their device sideways. As a web designer, you want to design to avoid making them do this.
That doesn't mean that you absolutely, positively, have to go redesign your web page. It will display just fine, and people who want to read your page will zoom, pinch and turn. It just falls under the category of being annoying. But annoying someone who is reading your page can be enough to discourage them, and make them click away. And with more and more site optimized for viewing on a 320-pixel screen, people will begin to click away immediately after seeing a web page like this.
As I tell my students, I don't even believe the light goes out in my refrigerator when I close the door. So inspect what you expect!