The police chief in Sanaa said one person was injured in the attack that happened in the east of the city on Monday.
The building and neighbouring houses were damaged in the blast, he added.
The Houthis, who control large swathes of Yemen and are regarded as enemies by Sunni fighters including those in al-Qaeda's Yemeni wing, sealed off the area soon afterwards to prevent access to it, local residents said.
Footage showed that the windows had been blown out of the building and bricks were missing from around the front door.
The attack came a day after a bomb explosion in a mainly Shia district in Sanaa killed at least four people including a reporter and wounded 25 others, according to officials. Al-Qaeda in the Arabian Peninsula (AQAP) claimed responsibility for the attack.
The Shia Houthi rebels seized control of the capital in September as part of a protracted power struggle with President Abd-Rabbu Mansour Hadi. Their critics say they are a front for Shia Iran, charges the rebels deny.
They have expanded into central and western parts of the country, leading to direct fighting between them and AQAP in some areas.
A series of bomb attacks by AQAP in recent weeks have struck Houthi targets, including a street celebration in which 26 were killed on Wednesday.