I can give a perspective from my own experience - me and my siblings started learning violin from between 3-4 years old, with piano following a year or so later. My parents were both quite musical and they considered it important for us to learn musical instruments, as well as learn theory of music.
I can't tell you how many times during my childhood I wished I could give it up, and begged to give up the violin.
Now, as an adult I can't tell you how many times I thank God that they didn't let me give up!! I really appreciate them sticking to the commitment of me learning an instrument. It has added so much to my life and I still use those skills regularly. As someone with a genetic predisposition towards being musical, I think I would have been very frustrated not to have the skill that I need or want to be able to express myself musically.
I have also been a teacher of piano and violin and seen that the children who succeed are those whose parents are also committed. They don't push and pressure the kids, and learning is still fun, but they do take a gently firm stance that this is something they've committed to and will see through. I think starting earlier is better (eg 4), personally, as it becomes a natural part of their lives and activities, as long as you can trust yourself not to make it a stressful experience for your child.
This is PURELY my own experience and not to be taken as any kind of generalisation for everyone!
HKAussie