You Are right. No need for an external router if all you need is wired. The XB6 has two LAN ports, port 1 is enabled by default. Also nothing wrong for most users with that modem. Good basic AC Wifi (Wifi 5)...
If you want full performance/control you will want to separate modem - router - access points....
Are you still in bridged mode?
