Microsoft recently released a system update that aims to make downloading through the Xbox One significantly faster. If you have internet speeds over 100 Mbps, then you should get an 80% increase in download speeds. If you have under 100 Mbps, you should get a 40% increase in download speeds.
Other notes include more performance and stability enhancements. I’ve found that the Xbox downloads are way slower than PS4 and definitely slower than PC download speeds, so hopefully this latest Xbox 1 update helps!