I went to Osaka in early feb, still pretty cold, not Hokkaido with my baby when she was almost six month, it was fine and we had a great time together!
Btw, Japan is a pretty baby friendly country, i think your baby will be just fine.
Just to let you know indoors always have heater on, so u better put different layers on your baby so that your baby won't be over/under-dressed.