's office.

A few weeks ago I went on site for a client and I seriously fell in love with their location, their team, their office environment, the view from their office, etc. Everybody seemed really chill (one of the reasons I love working IT in So Cal) but also very very sharp and knowledgeable and the type of growing team I've always wanted to be a part of. I had kinda forgotten about it but then I saw a job posting for the same client on one of the job sites I regularly get e-mails from.
I enjoy the freedom working for a consulting firm provides but I feel my skills are stagnating already as I spend very little time on any given technology or product before moving on to something else. What I was trained on 3-4 months ago, I've already forgotten and I am afraid that if I stay here too long I will lose a lot of forward momentum in my learning. If I spent more than 2-3 weeks on a particular product and actually got to know it a little better I would probably be happy but right now I'm just constantly jumping between radically different technologies and getting lost in the weeds a bit.
I know that the grass always looks greener on the other side, but I'm just wondering... has anyone else ever wanted to go work for a client? Has anyone ever done it? Is it... ethical? They're not a competitor, and obviously I wouldn't disclose any proprietary information about having worked here if I did hypothetically get a job there, but I've never been a consultant so I don't know the ramifications of trying a move like this.
Edit: I should note, I've read my employment contract and there's nothing that says I can't go work for a client. I'm just wondering if this is something that's generally frowned upon.