With wife and kids, my experience has been positive with a compact
Peak Design backpack (I just travelled for a week using the Zip and a minimal kit). When I expect frequent manipulation of the camera and lenses, I much prefer a
Mindshift Gear rotation34L backpack. The rotation system is fantastic, unrivaled on the market.
In any case, I have recommended using regular backpacks with a custom insert in the past, but I don't advocate this anymore. A dedicated camera backpack will make it so much easier to access your gear. Backpacks have improved tremendously in the last years, so much so that you can reach the same level of comfort and quality, but will include dedicated photo features.
Also, whatever solution you choose, I strongly recommend having a look at Peak Design's Capture Clip. This also makes it so much easier to function, especially with the family around.