Outlook has a built-in mechanism that will disable a plugin taking a long time to load. One of the causes for the delay in loading is determined to be in the loading process of the .NET libraries KOE depends on (up to two seconds have been reported).
There are several external factors that have an effect on the loading process as well. The most common one is Outlook trying to update a certificate revocation list from windows update (http://ctldl.windowsupdate.com/) on a daily basis. If this URL is blocked in your firewall it has great impact on the startup of Outlook.