Disable creation of Vendor Admin

You can create a pre controller for the app/controllers/backend/profiles.php in your add-on, with a code similar to this one:

if ($mode == 'add') {
    if ($auth['user_type'] === UserTypes::VENDOR) {
        $total_company_admins = db_get_field(
            'SELECT COUNT(user_id) FROM ?:users WHERE company_id = ?i AND user_type = ?s',
            $auth['company_id'],
            UserTypes::VENDOR
        );
        if ($total_company_admins >= 5) {
            fn_set_notification('W', __('warning'), 'too many vendor admins');
            fn_redirect('profiles.manage&user_type=V');
        }
    }
}

You can implement additional logic and use language variables if you wish.