I am pregnant with my second child right now and will give birth early this year. However, my thoughts are thus:
Babies are a lot of work--even if they're the cuddly type (and my now 3-year-old was not the cuddly type). To me for the first 18 months of my son's life it was a whole lot of work with very little reward. But, then when my son turned into a toddler, all of a sudden my love for him started to grow a lot. Now he's absolutely hilarious and entertaining and even though he still requires work, the bond we have is really tight. In the beginning, I felt like he was a huge investment of time and effort but we didn't get to see the "fruits of our labor" or much payback from that for a long time.
You've had a lot longer to bond with your first-born and every child and circumstance is different. Maybe your first born's personality and yours are similar or there are other strong things that particularly endear him to you. How was the pregnancy and birth experience with your second child? Babies require 10 hours+ day of constant attention but they can't communicate verbally their needs and perspective on life. They don't offer much in the way of companionship--every conversation you have with them is mostly one-sided. To me, when my son started talking, wow, that was when things got a lot better for me--because all of a sudden he became more "human" to me--like before he was just a needy thing that I didn't understand well but his personality started to shine through more and more as he became verbal and more physically able/independent. So, I've fallen in love with his personality. Sometimes it's hard to really love a baby's personality--especially if they are fussy and constantly rob you of sleep and sanity (as was my parenting experience).
I don't blame you for wanting to spend time with the toddler more than the baby. Some mothers even feel bored with their babies because well, babies sometimes aren't the most exciting humans around, right? Toddlers keep you on your toes--full of imagination and exploration. Some babies just sit there, poo themselves, spit up, cry and in other words require a lot of attention but at the end of the day they don't climb up on your lap and say something silly in your ear that makes you think, "Wow, that was all worth it." Baby care is pretty thankless in my opinion.
Also, this is your second motherhood experience so it may not seem that exciting to you compared with the first. With the first, us moms don't know anything (even though we think we do sometimes) and we spend all of our time taking photos and getting excited (or nervous/worried) about the littlest thing. The second go around must be a little less exciting--you've seen it before and you're ready for it. Maybe the love for our kids isn't as instant the second time around but grows deeper with time? I think it must be deeply individual.
I hope you will find (as will I) that eventually you'll love both of your kids equally but differently.
With my second though, we had complications in the pregnancy and because of that, I felt FIERCELY bonded with her even before she was born. When I first saw her, I already felt like I knew her right away. The bond was MUCH more intense and "early" compared to my first born. However I love them both completely the same. Just the way that our bond formed was different. With my first, it was a growing process after she was born and as I learnt how to be a mother... with my second, it started because I was so scared of losing her.