Klifurhúsið is an indoor bouldering gym located in Ármúli 23, 108 Reykjavík. There is a small play area and climbing wall designed for little kids but they seem quite welcome in the main part also.
Family Hours are from 12:00-15:00 on Saturdays and Sundays. Price: 950kr. with shoes for each family member that climbs. Children 7 years and younger pay 500kr for shoes only.
For kids 5-17 years: They have courses for ages 6-17 years and shorter courses for 5-8 years with parents or guardians. Here is a link to these.
Check out their website https://klifurhusid.is (which has a translate function into English) You can also email email@example.com.
It's a great place for school groups, sport clubs, birthdays or workplaces. My kids both LOVE it. I have completely failed in preventing my kids climbing on and over everything at home so this is a very positive way to channel their climbing inclinations.