It is a challenge at times.
We have a 4 year old and another due soon. My wife is great and supports all my hobbies... however, like you, most of the time I'd rather spend what "free" time I get making the most of family time.
When the 4 year old gets a little older I will definitely start taking her to the clay ground.