Definitely a personal decision, but we never take our helper on holidays in Asia. We do, however, stay at resorts that can arrange baby sitting for a couple hours in the afternoon or night so that we can scuba dive, hike, have dinner without baby. Have found that a baby sitter for a couple hours per day/night is more than enough.
We do take our helper to the US, but that is more because she really enjoys the opportunity to visit the US, we have a large place to stay there where she can have a private room and bathroom and it doesn't cost us like renting a second villa or room at a resort would, the trips are a bit longer and we both often have to work while there.
Have friends who do take their helper on all vacations, but they also have their helper do a lot more for their child at home than we do and they are very well off so they just rent another villa or a much larger villa so they can still have privacy.