Took my first child swimming for the first time when she was about 5 months old. I know of people who take their babies swimming from the very beginning--maybe 1-2-months-old. Some babies are born using Waterbirth and that technically would be their first swim, right? LOL.
It shouldn't matter as long as your baby is healthy and you're being safe with him or her in the pool.