You can create a pre controller for the app/controllers/backend/profiles.php in your add-on, with a code similar to this one:
if ($mode == 'add') {
if ($auth['user_type'] === UserTypes::VENDOR) {
$total_company_admins = db_get_field(
'SELECT COUNT(user_id) FROM ?:users WHERE company_id = ?i AND user_type = ?s',
$auth['company_id'],
UserTypes::VENDOR
);
if ($total_company_admins >= 5) {
fn_set_notification('W', __('warning'), 'too many vendor admins');
fn_redirect('profiles.manage&user_type=V');
}
}
}
You can implement additional logic and use language variables if you wish.