AI chatbots in healthcare: What you need to know

Artificial intelligence (AI) chatbots are changing the way we think, learn, and perceive the world around us. This change is evident in many areas of life, but perhaps one of the most sensitive and often problematic is the increasing use of genetic artificial intelligence (GenAI) tools in the healthcare sector.

Big tech companies have entered the healthcare space, launching services like Copilot Health, ChatGPT Health, and Amazon’s HealthAI. These platforms help users interpret their medical records and ask questions about symptoms, test results, and treatment options.

See more articles from iGuRu.gr when you search for news on Google.

iGuRu.gr preferred source

However, there are significant risks when an AI tool is asked to take on the role of a doctor, warns Phil Muncaster from the team at ESET, the world leader in cybersecurity. The problem is not limited to the possibility of incorrect advice. Users may also be sharing highly sensitive personal information with systems whose privacy procedures, data management practices and legal obligations differ from those of a doctor or hospital. As a result, their data may end up with unknown entities or be exposed to third parties.

According to a US patient safety organization , the misuse of artificial intelligence chatbots is now the most significant risk in the health technology sector.

From theory to practice

The reason AI modelers are getting into this space is obvious: chatbots have become an incredibly popular way to find medical information, explains ESET’s Muncaster. According to Microsoft, people discuss their health and the health of their loved ones more than any other topic on their mobile devices. Chatbots are available 24/7 and offer instant answers to almost anything, often with a tone of assurance that can reassure anxious patients.

At a time when national health systems are under increasing pressure, many people are likely to try to diagnose their problem with the help of artificial intelligence before deciding whether to seek medical help. The time, effort and potential cost of accessing the often complex health system make at-home “diagnosis” particularly attractive.

However, serious concerns have already begun to be expressed. The first concerns the so-called "hallucinations" of artificial intelligence and incorrect advice.

A study by the University of Oxford, published in February in Nature Medicine , found that:

• Users often did not know what information they needed to share with large language models (LLMs).
• The LLMs gave very different answers even when the questions differed only slightly.
• The models often provided a mix of correct and incorrect advice, while users had difficulty telling the difference.

“Despite the excitement, AI is simply not ready to take over the role of a doctor,” warned the study’s lead researcher, Dr. Rebecca Payne. “Patients need to be aware that asking a large language model about their symptoms can be dangerous, as it may give incorrect answers and fail to recognize when there is an urgent need for medical help.”

Revealing privacy risks

But there are also risks that are not directly related to health. The most obvious is that sharing sensitive medical information with a chatbot can lead to that data being used to train models or even being leaked to third parties. There have been documented cases where models have revealed data that their users had provided.

Some providers use the data to improve their models unless users opt out. Others offer stronger assurances that they won’t use health information for educational purposes. In any case, users should know exactly what kind of service they’re dealing with before uploading any sensitive files.

Health data is not like a stolen credit card that can be canceled and replaced. It is personal data that follows you for your entire life and, once shared with an AI tool, can become a permanent digital footprint.

On the other hand, most specialized health chatbots promise not to use this data to train their models. However, training is only one aspect of privacy protection. These services may not provide the same guarantees regarding data sharing with third parties.

Personal medical information may end up with data brokers or other intermediaries between the model provider and the healthcare provider. It may also be shared with advertisers, directly or indirectly through such companies, even in anonymized form.

Users should be extra careful. Health data is extremely sensitive, and anonymization doesn't always eliminate all risk of identification, Muncaster emphasizes.

When the risk of breach multiplies

When sensitive data is moved between multiple organizations, the likelihood of it falling victim to cyberattacks increases. US senators say they have identified $21 billion in losses linked to breaches at data brokers.

Health data is particularly valuable to fraudsters because:

• They retain their value for a long time, as they usually cannot be replaced or reissued.
• They may include insurance information that allows for false claims to be made or medical services to be obtained in the victim's name.
• They can be used for blackmail or other forms of fraud.

The more companies that hold such data, the more opportunities cybercriminals have to hack and steal it. The problem is that many AI tools in healthcare are not subject to U.S. privacy regulations. This means that their providers may not be subject to the strict data protection requirements you would expect from a healthcare organization.

Advice for patients

If you're concerned about a medical issue, avoid general-purpose chatbots and prefer tools designed specifically for health-related questions.

Check if the service explains:

• How it manages your data.
• If it uses your questions to train models.
• If it shares information with third parties.
• If it is covered by personal data protection legislation.

Don't blindly trust answers, even when they come with references or links. Verify information and always consult a health professional or official sources.

To protect your privacy:

• Never upload medical documents, test results, or other sensitive files unless you know exactly how they will be used.
• Avoid giving names, addresses, insurance information, patient registration number, or other identifying information.
• Make sure that training and chat history saving features are disabled.
• You only share the absolutely necessary information.
• Assume that anything you type may be stored or exposed, and act accordingly.

AI chatbots can be useful tools for preparing for a medical visit, understanding a medical term, or formulating questions for your doctor. But there is a big difference between using AI as an assistive tool and using it as a substitute for medical care.

Don't treat an answer as a diagnosis, and don't ignore symptoms because a machine gave you a reassuring answer. Artificial intelligence can complement human medical judgment, but it can't replace it.

newsbot

Although the press releases will range from very select to rare, I said I'd pass...because sometimes the editors hide.