It doesn't account for phys_base like it should, fix by using
page_to_pfn().

While we're here, make virt_to_page() use pfn_to_page() as well, so we
consistently use the asm/memory-model.h abstractions instead of
open-coding memory model assumptions.
Tested-by: Kristoffer Glembo <kristoffer@gaisler.com>
Signed-off-by: David S. Miller <davem@davemloft.net>