It is difficult to find a clean beach/sea to swim in but not impossible.
You can try, Pui O and Cheung Sha Wan Beaches (both on Lantau); both are accessible by Tung Chung by bus/taxi, or by Mui Wo by bus/taxi. (Tung Chung is accessible by MTR/airport bus, and Mui Wo is accessible by ferry from Central).
My favourite beach in HK is Tai Long Wan- it is clean, as is the sea; but this is because of it's inaccessibility. There are a number of ways to get there- you can hike from Sai Kung (not really an option with children); you can catch a ferry from the Chinese University pier and then a small hike gets you there (ferry is very infrequent and I last took it nine years ago, so not sure if still running); or you can hire a junk which will take you right to the beach. However the junk operator will charge you an extra $400 to take you to Tai Long Wan (because of it's distance) and junks can't always make it there because although the sea round the beach is calm, the sea that you have to sail through to get there is often very choppy.
You could also try more remote beaches on Lamma- although again this would require hiking or hiring a junk. Turtle Beach/Island is suppose to be lovely- but people are banned from the beach during certain months of the year, when the turtles are laying/hatching their eggs.
Such a shame that we are surrounded by sea and yet so much of it is so filthy!