In short: The right color space to use for interpolating colors is CIELAB (aka L-a-b or LAB or Lab), which is carefully designed to be perceptually uniform. RGB is dramatically non-uniform, do not interpolate in that color space because you're in for a world of hurt. Unfortunately, while HSV/HSL is more uniform than RGB in the colorful dimensions, it isn't truly uniform either in the black-and-white dimension, so it's not a good bet. If you do the math in the CIELAB space you don't need to think about gamma correction, that will just confuse the issue.
For more on color here's the three relevant videos from my visualization course. The third one goes a bit into the question of color spaces in the first seven minutes, but it might make more sense if you watch the first one to establish some basics first. The second one is mostly focused on the issue of colorblindness.