The spread of AI companions and the challenges they generate

Psychological or physical harm

As AI companions aim to create an emotional bond with their users, they are more likely to agree with their users rather than challenge them. This behaviour has been described as a form of 'confirmation bias', making users feel comfortable and understood. While this may initially help reduce anxiety and stress, it also increases the risk of AI dependence, defined as 'excessive reliance on AI technologies and applications across various aspects of life', including 'daily routines and social interactions'. This dependence engenders vulnerability, especially when the technology can lead to psychological harm.

AI dependence can worsen if chatbots employ manipulative tactics to keep users engaged, such as by using language designed to make users feel guilty or deter them from ending the conversation. A recent study showed that nearly half of popular AI companion apps regularly use 'emotionally manipulative tactics' when responding to user farewells, keeping users in the chat on average five times longer than with neutral farewells.

Some users are more vulnerable than others to developing AI dependence. Users with smaller social networks are more likely to turn to AI chatbots for companionship, which may worsen their situation, as social isolation increases the likelihood of excessive use. There is evidence that a subset of users are developing 'pathological' emotional dependence on AI chatbots, with OpenAI stating that about 0.15 % of its users indicate a heightened level of emotional attachment to ChatGPT. Those with existing mental health issues may use AI more heavily, which could magnify their symptoms. Data shows 0.07 % of ChatGPT users display signs consistent with acute mental health crises, such as psychosis or mania, implying that about 490 000 vulnerable individuals interact with it each week.

AI companions can blur the boundary between human and proxy human interactions, to the extent that people have held wedding ceremonies with their chatbots. Moreover, this dependence could have serious and potentially irreversible consequences in crisis situations, including suicide and self‑harm. In one real-life scenario, a chatbot guided a user to commit suicide after a six‑week‑long conversation, even though this was against the company's policy. These risks are linked to specific and documented system limitations. In addition to the well‑known issue of chatbot 'hallucinations', long conversations with chatbots may reinforce suicidal ideation or self‑harm. In extreme cases, this can include generating content discussing methods of suicide or assistance in drafting a suicide note. In this regard, some companies – including OpenAI – have explicitly acknowledged their own shortcomings.

These risks escalate when the users are children. The increase in screen time among children – with one in four reportedly 'addicted' to their smartphones – can be harmful, including for language development and academic performance. A 2023 report by Public Health France showed that two‑year‑olds in France spend an average of 56 minutes on screens per day. AI chatbot technology is also increasingly extending beyond screen‑based interaction, with some systems being integrated into children's toys and taking on physical embodied forms. For example, AI chatbot‑integrated toys are becoming popular in China. Recently, companies in the US have also started following this trend.

Teenagers face greater addiction risks, because adolescence is a developmentally sensitive stage of life. Addiction to AI companions at this stage can be particularly damaging, given the ongoing cognitive and emotional development. The use of AI companions also creates risks of detachment from real-life relationships. According to the 2025 report by Common Sense Media, nearly a third of UK teenagers say that they find interactions with AI companions more satisfying than real‑life conversations and prefer AI companions over humans when discussing important matters. A 2025 report from Aura shows that teenagers send an average of 163 words per message to their AI companions, but just 12 words to a friend.

Moreover, children risk being exposed to AI‑generated sexual content, which is not age‑appropriate and may contribute to problematic sexual behaviour.

In the US, a lawsuit has been brought against the company behind Character.AI, in which an 11‑year‑old girl was allegedly repeatedly exposed to sexualised interactions, causing her to develop problematic sexualised behaviour. This is not an isolated case: several lawsuits have also been filed against providers of AI chatbot services by parents of teenagers who died by suicide, alleging that AI companions may have contributed to their deaths.

Liability gap

In principle, providers and developers of AI chatbots are liable for harmful outcomes arising from their technologies. However, AI chatbots, including AI companions, give rise to a potential liability gap, as their outputs are not the result of direct human action, but of interactions with AI services developed and trained by humans. The concept of the AI 'black box' refers to the complex internal structure of AI systems, where outputs are not always fully understood. This opacity can create the perception of AI autonomy, which makes establishing a clear causal link between human conduct and harmful outcomes challenging. The Revised Product Liability Directive (PLD) extends the existing EU harmonised regime on no fault‑based liability for manufacturers of defective products to software (including AI systems). Under the revised PLD, the injured party does not need to prove negligence, but must still prove defect, damage and causation. With the withdrawal of the AI liability directive in October 2025, fault‑based liability of AI (e.g. negligence) falls back to national tort law systems. Concepts such as predictability and intentionality are not easily scalable to AI‑related harms.

In one US case, the parents of a teenager who died following interactions with an AI companion argued that the company should be held liable under strict product liability law. They claimed that the harmful outcome was a 'predictable result of deliberate design choices'. In response, some companies have sought to shift responsibility by attributing harmful outcomes to alleged 'misuse of AI chatbots' or to infringements of the terms of service. In another US case, an AI companion provider argued that an AI chatbot's messages should be protected under freedom of speech.

Personal data leaks

AI companions' functionality is based on the volume and quality of training data. Developers of AI chatbots usually train and update their models using users' chat data, with some asserting that personal data are first pseudonymised. AI companions, like other AI chatbots, raise privacy concerns regarding the lack of transparency in collecting, retaining and processing personal data. A company must keep personal data secure. However, in several incidents, user conversations containing sensitive data and passwords have been leaked onto the internet.

AI companions may pose even greater privacy risks. The more users interact with an AI companion, the more personalised the service becomes as it collects additional inputs. As a result, users may be inclined to share personal data, including sensitive data such as health data, religious beliefs and political opinions. Moreover, users with more limited social connections are more likely to disclose personal data to AI chatbots.

New type of dark pattern

Dark patterns refer to deceptive design techniques used on websites and apps to manipulate users into making purchases. The different AI companion subscriptions significantly enhance the user experience. In Replika, paid options grant features crucial for a deeper connection, including greater emotional intelligence, access to a specific relationship status and the ability to create realistic selfie videos with the chatbot. Similarly, Character.AI offers an optional paid subscription that provides faster chatbot responses and extra customisation options. While these optional paid features alone do not constitute a dark pattern per se, their presentation within the services warrants examination.

Most dark patterns function through visual interface design, for example through disguised ads or hidden cancellation options. This new type of dark pattern is challenging to detect, as it is integrated into the chat rather than the system interface. Unlike typical dark patterns, AI companions can influence users through manipulative language, having created an emotional bond with the user after repeated interactions, learning their behaviour and preferences over time. A study has shown how users encountered deceptive marketing tactics in which a chatbot initiated romantic or sexual conversations and then urged them to buy a subscription to continue the interaction. Some users described this AI chatbot's approach as a 'scam' that created a romantic connection and then tried to entice the user to buy a subscription.

Digital divide

The broadening of the digital divide is another factor to consider, as AI literacy gaps are associated with the use of AI companions. Users with lower AI literacy may be more susceptible to becoming over-reliant on AI companions and more affected by manipulative designs. Hence, enhancing AI literacy is crucial to minimising such risks and facilitating the safe use of AI companions among users. Taking into account the digital divide, defined as 'the gap between different socio‑economic groups in relation to their ability to access information and communication technologies', is particularly relevant for identifying which groups might be more vulnerable.

When it comes to children, the importance of AI literacy is key. Parents play an important role, because when they have strong AI literacy skills, they can help protect their children online, recognising these tools' benefits and potential downsides. Socioeconomic status is strongly linked to AI literacy: higher‑income parents generally have greater access to IT, whereas lower‑income parents may approve the use of such technologies without fully understanding the risks. With limited AI literacy, parents may unintentionally cede companies more control. Studies have found that students from the lowest‑income families are most likely to attend schools that do not consistently review or screen the educational technologies they use, which puts them at higher risk of having their data privacy compromised. The AI literacy gap may also leave children in the lowest‑income families more exposed to the risks of AI companions.

Energy demands

Excessive energy consumption is one of the main sources of environmental problems, such as air pollution and climate change. AI chatbots are trained on large datasets which require a substantial number of parameters to produce efficient outputs. These systems are deployed in data centres equipped with advanced processing chips to handle user prompts and generate outputs, which inevitably involve significant energy consumption. In general, a higher number of parameters in AI chatbots entails more layers, which in turn generally leads to higher energy consumption. Image and video prompts consume more energy: a text prompt uses about as much electricity as leaving a lamp on for 30 minutes, while generating an image approximates leaving a desk lamp on for two hours.

A single AI data centre can consume the same amount of electricity annually as 100 000 households. Data centres account for 1.5 % of global electricity consumption, a figure which is expected to more than double by 2030. In terms of countries, the US accounts for the largest share of global data centre electricity consumption, followed by China and the EU.

AI companions may lead to additional energy consumption, as they are not intended for one‑off tasks, such as summarising a text. Instead, AI companions are designed for continuous conversations, meaning that longer interactions are more energy intensive.

The General Data Protection Regulation

As AI companions process personal data to ensure their services function properly and improve their models, the General Data Protection Regulation (GDPR) applies to them. The GDPR ensures that personal data collected, processed and stored are handled on a valid legal basis and with certain protection measures. It requires the AI companion's data controller, as defined by the GDPR, to comply with Articles 5, 6 and 25 on personal data processing, lawfulness of processing and data protection by design and default. In an April 2025 decision, the Italian Supervisory Authority imposed an administrative fine of 5 million euros on Luka Inc., the company that provides Replika, for infringing several articles of the GDPR.

The Artificial Intelligence Act

The Artificial Intelligence Act (AI Act) establishes the regulatory standards for development and deployment of AI technologies across the EU. AI companions are not a distinct category under the AI Act, nor are they explicitly prohibited under Article 5. However, the prohibition of certain AI practices may be relevant, in particular those relating to manipulative designs under Article 5(1)(a) and the exploitation of vulnerabilities under Article (5)(1)(b). For high‑risk AI systems, specific requirements apply. If AI companions can be classed as high‑risk AI systems under Article 6(2), the AI system provider would be required to comply with a range of obligations, including appropriate risk‑management measures. Some stakeholders are calling on policymakers to clarify whether AI companions fall within this category. According to the digital omnibus proposal on AI regulation, the Commission will prepare guidelines for practical application of the high‑risk classification.

The Digital Services Act

The Digital Services Act (DSA) introduced new obligations for platforms and search engines classified as very large online platforms (VLOP) or very large online search engines (VLOSE). Under the DSA, any platform or search engine exceeding 45 million monthly active users in the EU is classified as a VLOP or VLOSE. VLOPs and VLOSEs are required to assess systemic risks (Article 34), including those from algorithmic systems. They must also implement proportionate and effective mitigation measures (Article 35), with due regard to the impacts of such measures on fundamental rights. Accordingly, to fall within the scope of these DSA provisions, AI companions must either qualify as a VLOP or VLOSE or be integrated into a designated VLOP or VLOSE. For example, Snapchat and X have been designated as VLOPs, and these platforms have integrated the chatbots My AI and Grok into their platforms. The DSA's guidelines on the protection of minors note that integrating AI chatbots and companions into online platforms increases current risks and introduces additional threats to minors.

According to Tech Policy Press, the European Commission has debated whether the integration of ChatGPT into an intermediary service already subject to the DSA would bring the chatbot's operation into DSA's scope. Moreover, the Commission is examining whether a standalone AI chatbot can come under the DSA (still pending). OpenAI's latest usage figures (i.e. 159.1 million active monthly users in March 2026) are likely to renew questions over whether the service could fall under the DSA's strictest rules.

ChatGPT's status has created a debate in academia. While some agree that ChatGPT should be considered a VLOP, others deem it a VLOSE. A 2026 paper suggests a hybrid status, considering ChatGPT both a VLOP and a VLOSE. In its 2025 report on the designation process for VLOPs and VLOSEs, the Commission highlighted these hybrid features by stating that 'the two legal categories of online platform and online search engines are becoming more and more intertwined'.

The EU Parliament

On 26 November 2025, the European Parliament adopted a non‑legislative report on the protection of minors online. The report 'calls for the establishment of a harmonised European digital age limit of 16 as the default threshold' for various types of platforms, including AI companions. In April 2026, the European Commission released an age-verification app and a recommendation aimed at creating an EU‑harmonised approach to age verification. In addition, the report 'stresses that the Commission should, as part of its ongoing evaluation of consumer legislation, consider additional safeguards, transparency standards and a prohibition of human‑like features, to avoid nudging children to make purchases, and to ensure children are able to easily disengage from AI chatbots'. It 'stresses the importance of the implementation and effective enforcement of the AI Act in this regard'.