The house and the grounds are really lovely, and make for a nice history lesson. I visited in late September, on what turned out to be a very hot, humid day. Along with that, although the gardens were great to look at, they were also packed with mosquitoes and we didn't manage to stay very long. So I would recommend late autumn to early spring for a more comfortable visit, or just check the forecast if you're a local. Also, many rooms in the building aren't in original Taisho-period condition-- you'll see a fair number of dingy carpets and circa-1980s doors. It detracts from the experience, just a little.

Overall worth a visit, especially on a cool day, and if you're interested in Taisho period Tokyo. But if you've got the time to go to Kyoto, I'd say the examples of traditional Japanese architecture there are more remarkable (Nijo Palace and Ryoanji in particular). The Asakura house is remarkable half because it's a nice buildings and half because it's an extreme rarity in Tokyo- a wood building that survived WW2.