The Global Risks Report 2024

However, despite substantial state intervention – and in some cases, because of offensive economic policy – production will remain heavily concentrated.Barriers to entry remain high, and there are limits to the extent to which state policies can lower them. Sizeable upfront capital expenditure for innovation and infrastructure, economies of scale and scope, a niche talent pool, information asymmetries, and proprietary data pools will continue to favor established companies. 60 Vertical integration could become more prevalent, as producers of foundation models increasingly expand to downstream uses or partner with platform companies that control online data pools or offer cloud services. 61 Regulatory controls on downstream applications could entrench market power further. For example, the use of a licensing regime could embed the power of existing players, even as it enhances oversight of frontier AI. 62 As governments seek to manage the higher risk applications, widespread dependence on the underlying tech stack (the technologies used to develop an application) will likely lend tech leaders a disproportionate influence on legislative discourse, shaping industry norms and standards over the next decade. While downstream applications are far more competitive, upstream commercial motives – rather than public interest – could become the guiding force of AI development and deployment. This trade-off can already be seen in the distinct lack of consistent self-regulation by the industry, with responsible AI teams among the first to be subject to redundancies as the sector downsized in recent years. 63 Tech companies could be left largely in charge of prices as well as privacy, and they may hold excessive sway over preventing competitive innovation. If monopoly- or oligopoly-led profit maximization is the primary objective of AI deployment over the next decade, the consequences for applications across healthcare, education, military, legal and financial sectors will be vast. In healthcare, for example, as the volume and granularity of health data increases exponentially, the commercialization of related data pools for downstream AI applications could compromise individual privacy and erode trust in healthcare systems. In the absence of strong ethical guardrails, medical data obtained from a fitness tracker, for instance, could individualize advertising, facilitate discriminatory profiling for health insurance, or underpin new, more invasive forms of employee monitoring. Even as data access enables new healthcare solutions and early diagnosis, medical research and development could be geared towards the wealthy – those who have the resources to afford this type of pervasive daily data collection and monitoring that is then used to train AI for various applications. Additionally, the influence of upstream companies could mean that accountability for related risks, from biased algorithms to diagnostic errors, is pushed downstream in some jurisdictions, particularly in countries with more limited market power, in return for access to these technologies. The next global shock? BOX 2.5 Breakthrough in quantum computing Quantum computing could break and remake monopolies over compute power, posing radical risks in its development. 64 Criminal actors have already launched harvest attacks (also known as “Store Now, Decrypt Later”, or SNDL) in anticipation of a cryptographically significant computer. Trade secrets across multiple industries, including pharmaceuticals and technological hardware, could be compromised, alongside critically sensitive data such as electronic health records and sold to the highest bidder. 65 Large or even global infrastructure – such as banks, power grids and hospitals – could also be paralyzed. Yet it is not just widespread proliferation of this level of compute power that is concerning. Although arguably a tail risk, 66 if cryptographically-significant quantum computing capability is covertly achieved and subsequently uncovered, it may rapidly destabilize global security dynamics. Mohammad Rahmani, Unsplash Global Risks Report 2024 53