Based on a rich database of all 2390 Norwegian new ventures established in 2003 that export in the period 2003–11, we explore two research questions: (1) Do new ventures follow the incremental internationalization process by starting to export to countries close to the domestic market and gradually expanding to more geographically distant markets? (2) Does the speed of export start-up matter for the firms’ future export success and expansion? Both these research questions are answered with a yes. In particular, the speed of export start-up reveals interesting patterns. Firms starting to export already in the first year of operation are more successful in exporting than firms starting a year later. The latter firms, in turn, are more successful than those starting in year three, and those starting in year three are more successful than firms starting to export in year four. We conjecture that success of first-year exporters is mainly explained by better resources (management, products and networks). We also propose that the later the firm starts exporting the more it is engrained in idiosyncrasies of the domestic market, constraining its perception of opportunities abroad.