According to the HTML5 spec, setCustomValidity can be used with <fieldset> elements.

However, current browsers don't really implement this and won't render the error message if you use setCustomValidity directly on the <fieldset> element. Instead, you have to use setCustomValidity on one of the <input> elements contained in the <fieldset> element.

When using a radio-button group for a mandatory single-valued property, you have to make sure that a value is provided. Either you make a default choice (e.g., the first radio button or one which makes sense), or you check on form submission, with an event handler, if at least one radio button has been selected.

When using a check-box group for a mandatory multi-valued property, you check on form submission, with an event handler, if at least one checkbox has been selected. The same applies if more than one value is required for a property rendered as a group of check-boxes (e.g., at least 3 out of 10 choices has to be made).

In both cases, the error message can be attached to the first <input> element in the <fieldset>, and the error message will be shown by the browser, close to that element. Visually, this is not the perfect way, but it allows to see the error, so the user knows what is wrong with the form data.