As a newcomer to this great sport, I can offer some first hand perspective of why I think there seems to be limited growth: Exposure.
Let's face it, most people (like me) are typically exposed to public waters.
What we see happening on these waters directly influences what others tend to take up on the water, especially those new to boating or lake lifestyle.
We have spent the last several years on open water, initially with an I/O boat. 99.9% of what "I" saw during these times were surfing, tubing and wakeboarding.
It wasn't until several years later hanging around the lake, at 7AM while walking about with my coffee, some very new friends asked me to come and spot them while they skied. Sure, no problem.
Turns out that he is quite an accomplished shortline skier (now I recognize that).
When I saw that from the boat for the very first time, the light went on, and realized how cool it was and that I'd like to do it also.
I was a convert from that point forward.
Now, here in lies the issue - MOST people simply aren't up and running around in boats at 7AM unless your a diehard skier or fisherman.
That's when the good water is on public water. After the wee hours in the morning, it's gone and the lake is a boating zoo.
The young crowd has NO INTEREST in getting out on the water at 7AM unless they are already hooked on a sport that demands it - so they more often than not don't get to experience it from the boat.
So so many people never even get to see a slalom skier - just due to when and where it happens.
I think this sport for public exposure on public water is limited simply due to the harsh constraints (for public lakes) required to do it: no wind & no boat chop.
I HONESTLY believe that if the younger generation simply had more direct exposure to it, this sport would attract a lot of newcomers.
My .02 perspective.