For Nagios XI is usual web access URL http://some.domain:port/nagiosxi
In that case, Nagios Core access URL is http://some.domain:port/nagios and that address should be used in Nagios API URL configuration field.
To check if the URL is correct, try in the web browser access http://some.domain:port/nagios/cgi-bin/objectjson.cgi?query=hostgrouplist
some.domain and port should be replaced by actual values from your installation.