Generative AI is inherently insecure

This will be a controversial one. Consider me Gordon Ramsay stirring the pot during an episode of Kitchen Nightmares. This blog will detail my reasoning behind the claim that Generative AI is inherently insecure. I will back up my reasoning and claims with sources when needed. Yet, as is often the case today, for every source saying one thing another one can be found stating the total opposite. Taken together it is however hard to argue against the general conclusion, that from a security perspective Generative AI is fundamentally broken.

Before going any further it is useful to make some remarks about the applicability of the following. Any references to “AI” should unless otherwise stated be understood to reference “Generative AI”. In this text no distinction are made between Generative AI and Large Language models (LLMs), the same arguments apply broadly to both. It should be noted that “ordinary” machine learning, including many Deep Learning models, has real usage and well-defined and manageable risks. The arguments outlined do not apply to those applications (if implemented correctly of course). Lastly, not every concept can be explained in depth. The option to explore many of the discussed concepts in more detailed are left as possible future blogs. Feel free to reach out if a deep dive in any of the topics would be interesting. With all the dull stuff out of the way it is time to get to the good stuff.

Where are we today and how did we get here?

It has now been more than three years since the first publicly available GPT model (GPT-3.5) used for the flagship product ChatGPT. What has happened and changed since then? Well it depends on who you ask. A general trend that most can agree on is that the adoption and usage, along with overall hype and optimise, has been polarised. Several professions rush like a 100-meter dash to make everything “AI” and implement it everywhere. Three years in now, and this is still the trend in many areas. I am not in a position to confidently say if this is the right or wrong thing to do. What I however can state with confidence is that a surprising amount of security professions seems to be sleeping on the unprecedented risk this is. Even worse, an anecdotal observation is that many security professionals actively encourage hasty and thoughtless implementation of Generative AI solutions throughout the organisation. A not entirely uncommon phenomena are security professionals pivoting entirely into an AI oriented position. This is alarming. Security people should be among the most boring and inert objects in the universe. Every new technology should be made to convince us of its advantages and how its risks can be managed and controlled. In this case it seems that the reverse has been made standard. A healthily sceptical security professions has to answers question on why not Generative AI is a productive boom and why not risks can be dismissed. Fine, let us take this position and argue primary for why Generative AI is insecure, regardless of the implementation details. Furthermore, let us oppose the often advocated picture that Generative AI constitute dramatic and revolutionary advantages/benefits. Starting from first principle in Information Security and the most basic aspects of Generative AI it will hopefully be evident what a security headache this can end up be. Many of these security concerns has been raised before. However, they have seldom been placed in the correct context and as a piece of the total puzzle.

First a quick summarising of the Information Security principles underlying the entire argument (for those not coming from a security background). Then each of these will be explored as applied to how Generative AI fundamentally works, as well as how some of the knowns Generative AI vulnerabilities fits into this picture.

What do we even mean with secure?

Bear with me, we need to take a step back and start with the basics. Without a firm grasp of what secure versus insecure mean it will be hard to make sense of anything that follows. Think of this as laying the groundwork for building our house of pessimism. Remember that what is to be secured is “Information”. This is another surprisingly intricate topic. But do not worry about that right now. An everyday understanding of the concept is enough for the moment. Observe that multiple different definitions of “secure” exist, this blog chooses one which is easy to understand and work with.

One of the most central concepts in Information Security is without a doubt the “CIA-triad” [https://www.cisecurity.org/insights/spotlight/ei-isac-cybersecurity-spotlight-cia-triad] [https://www.sciencedirect.com/science/article/abs/pii/S1467089505000473?via%3Dihub ]. Take a security profession and wake them in the middle of the night, they will know this by heart. The CIA shorthand stands for “Confidentiality, Integrity, and Availability”. I will spare you a detailed description of these concepts. We do not need all the complex details anyhow. In addition, we will focus on how these apply to Information as a valuable commodity. A simplified helicopter view is that:

• Confidentiality
  • Information is not to be made available or disclosed to unauthorised individuals, entities, or processes. A more direct description of the concept is that sensitive information is to be kept secret from those that have no reason to know about it. For example, if you have some juice gossip about someone then you might want to restrict who knows about it, unless it spread uncontrollably.
• Integrity
  • Information has to be accurate and complete, now and in the future. The essence of this concept is that the information has to be trustworthy. If the information can be modified (without notice) or changes at will then what is the purpose of the information? For example, if you store a family recipe you want to protect the integrity and correctness of it.
• Availability
  • Information has to be available and accessible when it is needed. This is an often forgotten core concept when you work with information. If you cannot get to the information when you need it, it might as well not existed to begin with. For example, if you have a deed to inherit a mansion, but it is stuck in an uncrackable Franz Jäger safe then how are you to prove ownership?

When we are presented with the problem of estimating or determining the security posture of a solution, system, application, gizmo or whatever, we always have the CIA-triad in mind. Sometimes we considered it excitability and other times implicitly. But it is always there somewhere in the corner of our mind.

So, what is secure? Well, it should be the case where all three of CIA-triad concepts are managed and handled. We do not want to be unsure if our super important system leaks secrets to anyone, if the information we get is truthful or not (in the context), or if the information even can be accessed when it is needed the most. How this applies depends on the situation. The nature of the object (system, application, solution etc.) we consider result in different preconditions for these concepts. It is also possible to take actions to improve the chances that these concepts are upheld. With that said, no matter what the preconditions are or what actions we take it will never be enough to fully ensure that each concept is totally satisfied at every time. It is an impossible task. Because of this it is of extreme importance to then turn the attention to examine and determine how these concepts can fail in the current context. Depending on the scenario different deficiencies are conceivable when trying to adhere to these concepts. A takeaway point is that we should always be mindful of how any one of these concepts can falter and how it will affect us. From this we land in the following informal definition: Something is “secure” if we have addressed and managed the aspect in the CIA-triad, and have an understanding of how and when these can falter.

A last remark about an important and often forgotten aspect of Information [Katz, Amnon (1967). Principles of Statistical Mechanics]. Even though an everyday understanding of information was stated to be enough it is still important to mention how Information is important and valuable. What is the reason that information has to be protected? Do information have any intrinsic importance? For example, a page with scrambled random letters contains information in some sense. Still, most would consider this gibberish and nothing of value. In this form information is purely a quantitative concept. Taking this one step further, the letters on a page can be arranged such that the language is correct. How is the value of this page then determined? To answer this we have to pair the information on the page with prior knowledge, experience and intelligence. With that the pairing only make any sense if the page is chosen to be in a very specific format (letters arranged) such that it is useful to us. It is critical to understand the difference between any information and useful Information. The latter only exist in a very specific form which is intimate connected to a target audience with a precise level of knowledge. This distinction is of some importance when considering the security concept of Integrity. Even more important is this distinction when evaluating the usefulness of any information system. It will be clear how this fits together with the argument later.

With a framework in place which can be used to examine the security of any technical solutions, including Generative AI, we are almost ready to outline the argument. Before that we have to make a quick pit stop and explain some of the core elements of Generative AI.

The quickest crash course ever on Generative AI

To explain the inner workings of Generative AI is no easy or straightforward task. It would be impossible to do that without making this excessive blog even more bloated. I will spare you from this. Multiple good resources exist for the reader who wants to make a deep dive [https://www.ibm.com/think/machine-learning] [https://developers.google.com/machine-learning/] [https://www.oreilly.com/library/view/hands-on-machine-learning/9781098125967/]. What follows will be greatly simplified. It will be important later on when discussing risks to have a common understand of what building blocks of Generative AI are referenced and how they all fit together. These concepts are unavoidable aspects of any Generative AI solution. Note that these are only a small selection of the relevant concepts. Only the one needed later on when making the argument are described.

Oversimplified, some of the core aspects of Generative AI includes:

• Model
  • The more correct term used to identify a particular form of Generative AI solution (the model along with auxiliary parts makes up the total solution). The same terminology is used in Machine Learning and other related fields. A model reproduce and reflects pattern in the training data used to create the model. With this the model can be used to predict/infer conclusions from new input data. This is called Interference.
• Training
  • To construct any model a great amount of training data is needed. It cannot be overstated how important this is. The amount of information is staggering. If not the entirety of the public Internet then at-least a large portion of it is used. Add to this all books and publications that can be obtained. Beside the amount of data, the quality of it is likewise vital. Recall from previous section how “valuable” information can be distinguished from just any type of information. A general principle is that the more and the better the training data the better the resulting model will be. Training is both an initial part of creating a model and an ongoing process if the model is to be augmented with new information.
• Interference
  • A trained and prepared model can be used to make predictions for a new case/data point. Another word for this is interference. Pre-Generative AI the most common form of interference was to predict and categorise new data points according to an existing list of different categories known by the model. In this case the model often produced probabilities. These reflected the likelihood that the inputted data entry matches one of the known categories. For Generative AI the interference process and result is more complex. Roughly speaking, the model makes a series of predictions corresponding to each word in the output (applies only to language models). The combined output is then considered the result. Hence, for Generative AI the interference process is more convoluted than for most other Machine Learning models and produce a result more excessive than simple likelihood.
• Statistical nature (Non-deterministic)
  • Virtually every form of existing models deals only in probabilities. That is, the output is probabilistic. Using the same rough description from before, because each word in the total result is generated in a statistical manner, the final output text can vary even if the same input is used. Moreover, what distinguish Generative AI and the complex input data (prompt) is that the same input data can be parsed in different context. Previous input and other factors are included as input data for the model. This results in an additional layer of probability. The takeaway is that the exact form of the output is entirely probabilistic. Another way of describing this is that the connection between input and output is non-deterministic. It is not possible to say with certainty what output you get for a specific input. This is in contrast to most other computer systems, where the same input always produce the same output. Those are deterministic systems.
• Tuning (Temperature etc.)
  • Beside the training data it is possible to adjust the model by fine-tuning or changing what is known as hyperparameters. Almost always these have trade-offs. When changing their values it might improve certain aspects of the model while worsening other. The hyperparameters could affect the model construction (training) or its output. An often heard hyperparameter specific for Generative models is “Temperature”. This parameter directly affects the output of the model. The temperature parameter is used to regulate the shape of the probability distribution used during interference when predicting increment values (such as words). A higher value increase the probability for lower events (predictions) and a lower value decrease those probabilities further. In other words, a high value for the Temperature increase the models “creativity” while worsening its reliability/deterministic- tendencies. The converse is also true, a lower value for the Temperature produce more predictable and similar responses, while lowering chance of unexpected and interesting output.
• Session management
  • Models are in most respect stateless. That means that they do not maintain any sense of continuation or state, at least not by themself. Each input is processed independently of those the came before. Solutions build on the models often solves this by appending the entire history (previous input and output) to the new input. This memory is somewhat then related to the concept of “context window” which is often heard when discussing Generative AI.
• Input and output processing
  • This will be even more simplified the other aspects. Beside the history part mentioned before, many more things happen before the input reach the model. Depending on the system, the input can be pre-proceed by other models. But most consequential are the pre-prompts added to the input data (prompt). Other names for these are “hidden prompts” or “static prompts”. Regardless, these contain amongst other directives that aim to restrict or limit the model. This could for example mean instructions not to process and output anything relating to dangerous or controversial subjects. Because of the statistical nature of the model, input filters are often not enough to ensure that the final result does not contain undesired content. It is therefore common with output filters before the result is presented to the user which likewise check for dangerous or controversial subjects. The act of subverting these filters is often referred to as “jailbreaking”.
• Agents
  • One of the more recent buzzwords. Agents are meant to augment and extent Generative AI solutions so that they can act autonomously. The goals are to achieve advanced automatisation. This is achieved by for example combining multiple models chained together along with interfaces to use different tools and APIs.
• Hardware requirement
  • Along with the copious amount of training data needed, specific hardware also have to be utilised to create any complex model. This hardware is either Graphics Processing Units (GPUs) or even more specialised Tensor Processing Units (TPUs). A large number of these units are needed to train a model. Creating a model requires substantial computational power.

This was by no means exhaustive with respect to all relevant and important parts of a Generative AI solution. For example, the concept of tokens and reasoning has not been discussed, not has the related notion of ”Vibe Coding” been examined. The aim was to provide more context and have a common understanding of key concepts relevant to the argument later on.

With this we should have everything in place to proceed to the argument.

The Argument

It is time for the meat of the matter. With an understanding of what it takes to call anything even remotely secure, we have the opportunity to reverse the reasoning to say when something is blatantly “insecure”: Just turn a blind eye to any of the security concepts, and you have an insecure system. If the preconditions also are terrible then you have a recipe for disaster. The argument will boil down to the following:

Generative AI has critical flawed preconditions to handle risks relating to Confidentiality, Integrity and Availability. In addition, all existing solutions actively disregard the most grievous of those risks.

An elaboration of this argument is best made by considering risks relating to each category in the CIA-trad. Together we will explore how each of these categories has multiple risks stemming from the fundamental nature of how Generative AI is built, functions and is used. None of these risks can be effectively handled by current solutions. It will also be argued that suppliers actively disregard these risks. The argumentation will draw upon many different sources to lead credibility to the reasoning. Each of the following sections could easily been a blog in their own right. Considered it a summarisation and starting point for further possible exploration.

Without any further ado we dive in, starting with Confidentiality.

Confidentiality – Generative AI cannot handle secrets

On the nature of training data

It is hard to argue against the notation that nothing is as fundamental to a Generative AI solutions as training data. The entire existence of these systems rest on the fact that humanity has produced such a copious amount of data that many relevant questions that could be asked has been discussed (but not always answered definitely). The topic of how this data is collected and processed is as often the case in this blog a complex one. Two important aspects are these: data has to be preprocessed before being used for training, and the available data for training is starting to run out. Starting with the first aspect. It has been known for a couple of years [https://time.com/6247678/openai-chatgpt-kenya-workers/] that companies such as OpenAI employs people to manually examine and classify data before allowing it to be used as training data. This is done in order to prevent the model from learning undesired patterns (dangerous or illegal). The data not deemed suitable for training is either discarded or used to train a separate model used for filtering. But what if something slips past these manual controls? Something that is not dangerous, but is sensitive for you or your organisation? The most common object to this would be that it would be impossible for someone to obtain that kind of sensitive information. It should not be publicly available to be collected as training data to start with. Well, if the information at anytime has been exposed externally then it could have been collected as training data by a crawler used by the larger AI companies. All the major AI companies do not care about copyright [https://www.theguardian.com/technology/2025/dec/05/new-york-times-perplexity-ai-lawsuit ]. Moreover, what if an employee uploaded a document to ChatGPT for example? The default configuration for ChatGPT is to use this material for training [https://help.openai.com/en/articles/8983130-what-if-i-want-to-keep-my-history-on-but-disable-model-training]. It is therefore not unlikely that some amount of sensitive data has entered the training data set at one point or another.

The second aspect is even more troublesome is some sense. It has been known for a while [https://www.nature.com/articles/d41586-025-00288-9] that the amount of accessible training data is running out. Or at-least, the amount of “good” training data created by humans with intent and purpure (valuable Information). The easiest way to improve and evolve a model is to feed in more and better training data. This has been known even longer [https://dl.acm.org/doi/10.3115/1073012.1073017]. With this the motivation to obtain additional training data is strong for model developers. To illustrate that this could lead to conflict of interest, consider a cloud storage solution. Classical solutions, where the storage itself is the core business, has no explicit interest of the content (disregarding any trade secrets and the catastrophic result of accessing those). Contrast this to a modern AI enabled cloud storage solution. In the latter case, all data is valuable, as it can be used for training of the next generation of models. Here two different interests are in conflict: that to a specific customer that wants to maintain the Confidentiality of their information, and the company and other customers that wants improved models. For private data this conflict has already been lost to some degree. As mentioned before, most AI providers defaults to using conversational history to training future models. In addition, the European Union seems to be ready to gut core components of GDPR in order for AI companies to train their models on personal data [https://www.politico.eu/article/brussels-knifes-privacy-to-feed-the-ai-boom-gdpr-digital-omnibus/]. This could have a dramatic effect on how privacy is regarded in the coming years.

Hopefully I have painted the picture that training data used for models can and probably will contain sensitive data. Either for you personally or for the company you work for. What is the danger our allowing information being part of the training data? A model can often reproduce its training data with varying amount of accuracy. One known example is how Metas AI model could reproduce large extracts from a Harry Potter book [https://arstechnica.com/features/2025/06/study-metas-llama-3-1-can-recall-42-percent-of-the-first-harry-potter-book/]. Interesting here is also how the model and its training seem to completely ignore the concept of trademark, but that is another topic. Smaller extracts of input training data could probably be reproduced in ever greater detail. Hence, if any data has been included in the training set there is always a chance that it could be obtained at a later time by interference. By repeatedly prompting the model with different input the probability for this would likely increase.

The only way to completely eliminate this risk is to remove the sensitive information from the training data and subsequently retrain the model. This is not an option. Training a model takes an excessive amount of time and requires a great deal of computations. An alternative often considered is some kind of post-training. A similar method is often used to augment an existing model with new data. A question that becomes immediately apparent is how do you train a model to forget something? This is still very much an open question [https://research.ibm.com/blog/llm-unlearning] [https://link.springer.com/article/10.1007/s10462-025-11376-7]. No universal method exists at this time. Moreover, even if a method were to exist, how and when are this method to be invoked? At some point the capability of the model will be affected by remove training data. This again becomes a conflict of interest for the model builders. If every request to amend or remove information is fulfilled then how much of the model will in the end remain?

In present day the method used to restrict processing and output of undesired information are hidden/static prompts and output filtering. Simplified this consist of including instructions to not discuss certain topics and stop the output to the end user if certain content is present in the intended output result.

Prompt injection and bypassing restrictions

If the input/output restrictions can be bypassed than any sensitive or undesired information can be accessed. These restrictions are by their very nature obtuse, and it has been shown how these can be bypassed in every case (for all current Generative AI solutions). The act of bypassing these restrictions is known as jailbreaking by prompt injection. This is both surprisingly easy and straightforward. Detailed information about what this entails can be seen in for example [https://portswigger.net/web-security/llm-attacks] [https://genai.owasp.org/llmrisk/llm01-prompt-injection/]. Examples of current research can be seen in [https://www.tenable.com/blog/hackedgpt-novel-ai-vulnerabilities-open-the-door-for-private-data-leakage] and the work done by [https://pliny.gg/]. The takeaway is that all current Generative AI solutions are affected by this type of vulnerability and bypass. Without going into the details, the problem stems from the way the restrictions are implemented. The input restrictions are combined with the input data (in some way or another). This means that conceptually the user input can affect the restrictions. As described, the input data is more than just the static prompt and input data. The entire previous correspondence is often appended as well (in order to maintain the session). The model can be jailbroken “slowly” by putting it in an unintended state after several exchanges [https://www.youtube.com/shorts/WP5_XJY_P0Q]. Output filtering can likewise often be circumvented by outputting the result in a specialised format or even sending it out-of-band. It is worth noting that the system prompt (static prompt plus other instructions to the model) also can be reverse engineer or otherwise obtained [https://www.forbes.com/sites/johnkoetsier/2025/08/09/gpt-5s-system-prompt-just-leaked-heres-what-we-learned/ ] [https://github.com/x1xhlol/system-prompts-and-models-of-ai-tools ] making jailbreaking even more applicable.

That an entire genre of solutions can have such a severe and reoccurring type of vulnerability is staggering. Three years in, and every major Generative AI solution has at one time or another been jailbroken. This shows that no effective and universal method exist to mitigate this vulnerability. Most mitigations are just bolted on patchworks of fixes. Compare this to a somewhat similar type of vulnerability that can affect databases, “SQL injections”. Sure this vulnerability has been known a much longer time. But the mitigation of this is straightforward and multiple actions can be taken in-tandem to completely prohibit this type of vulnerability. Even when first publicised [https://phrack.org/issues/54/8] it was clear how to effectively protect against this type of attack with available means and functions at the time (custom stored database procedures in this case). In contrast, it is still an open question on how to definitely address prompt injections. Many suppliers seem to have given up on attempting to solve the problem of prompt injection, instead issuing warning to users [https://learn.microsoft.com/en-us/windows/security/book/operating-system-agentic-security].

The conclusion is that it is likely that sensitive information will end up in Generative AI solutions. In their current forms, it is not possible to remove the information. Restrictions placed on the solutions to otherwise limit the access to this information have been shown to be possible to bypass in every case. Hence, Generative AI solutions cannot be guaranteed to not contain sensitive information, and restrictions impose on access to sensitive data cannot be relied upon.

Integrity – Generative AI can output anything

This will be a doozy. A personal favourite as pertaining to the total argument. We start by continue with our examination of training data and its nature.

Errors in training data and training data poisoning

It was briefly discussed before that all data flowing into a model for training pass some form of processing. Most of this is automatic. Certain actions are performed manually, as discussed in the context of filtering of dangerous material. This preprocessing cannot take into consideration things such as the sensitivity of the information (because often it is only sensitive for certain parties). In the same vein, the quality of training data is hard to gauge. What is good training data? Even more complex, what is truthful or erroneous information? What context do the information occurs in? Is it satirical? Several well known examples of this mix-up exist. Gemini (Googles AI solution) for example reported that one small rock per day was a perfect normal and good diet [https://www.bbc.com/news/articles/cd11gzejgz4o]. By search almost verbatim for the same output it is possible to find the original source for this information. The Onion in this case [https://theonion.com/geologists-recommend-eating-at-least-one-small-rock-per-1846655112/]. Almost everybody knows that the Onion is a satirical news publication. Yet, the model took information from one context and presented it in a totally different context. This morphed the information from a joke into a fact.

A couple of core problems can be distillate from this:

• Because training data comes from external sources (the Internet), all of which cannot be quality controlled, erroneous pattern will be learned by the model for some topics/subjects.
• The training process has a hard time to consider the context in which the training data occurred. Sarcasm, jokes and reverse psychology are just a couple of context that greatly affect how the information should be interpreted.

For both of the raised points no malicious intend is necessary present. Sometime people are wrong when discussing a subject. What is current best practice changes as new discovery are made and research evolves. There are a great many ways information can be half-truth or half lies without any malicious indent. The more niche or specific a subject is the more likely it is that the existing information is in some respect lacking. To reflect the ever-changing information landscape all modern models are continuously trained in a fine-tuning fashion. Yet, the core problems raised above still remains. Fine-tuning then becomes more of a whack-a-mole solution for these.

With that it is easy to see these deficiencies in quality control for the training data could be abused by those that want to maliciously influence the information or behaviour of the model. If an attacker can control what training data is passed to the model pertaining to specific subjects and topics, then that information could overwhelm any non-malicious information. It will then be the base on which the model learns its pattern for those specific subjects/topics. This process is known as “Training Data Poisoning” [https://genai.owasp.org/llmrisk2023-24/llm03-training-data-poisoning/ ]. Recall how training takes place when creating a model, but could also happen as part of fine-tuning after the model is placed in production. What is the consequence of training data poisoning? The details depend on the capabilities of the Generative AI solution (if it has access to tools for example which is the case for Agents). A worse case example is that the model can be induced to perform actions when asked very peculiar questions. These questions could touch upon information that was only present in the attacker-provided parts of the training data. This niche data could even combine prompt-injection techniques to facilitate complete compromise of the Generative AI system. If the system is chat-oriented solution then the consequence is lessened. Still, the reliability of the output information will be compromised. A successful “training data poisoning” attack imply a total collapse of the Integrity of the model.

An outstanding question then is how likely such a scenario is? In the early days it was assumed that an attacker has to control a large portion of the input training data for this to even be possible. For training data taken from the Internet this would be several thousand independent data points from multiple sources. Moreover, because the bulk of training is done at an initial stage (when creating the model), it would not be feasible to poison the model when fine-tuning training is done. The difference in the amount of data between the initial and fine-tuning training would simply be too big. This notion has since then been disproven. It is possible to poison any model (regardless of size) by only submitting malicious data points numbering in the hundreds [https://openreview.net/forum?id=eiqrnVaeIw] [https://www.anthropic.com/research/small-samples-poison ]. With this it becomes far more likely that a model can be compromised. Especially if the poison process is designed to be covert and not bluntly injecting malicious content. Training data poison could become an effective vector to insert disinformation into the model

Hallucination and “Bullshit”

Consider a fictive scenario were all the training data have been verified or only taken from trusted sources. Does that guarantee the Integrity of the model? If we are to take Integrity as any measurement of the trustworthiness of the model or reliability of the answers then no. Information presented by the model will not maintain a definitive and deterministic form. Nor will we be able to even determine if any information or knowledge really exist in the model. The fact that output information can be malformed and untruthful is often referred to as “Hallucinations” of the model [https://genai.owasp.org/llm-top-10/][https://cloud.google.com/discover/what-are-ai-hallucinations]. This is an intrinsic feature of Generative models and their statistical nature. A general tendency is for more complex Generative AI solutions to hallucinate more. This correlate for example to the amount of training data, how the models hyperparameters are tuned, the usage area and if any multistep processing is used (such as “reasoning”). The last point is known to overall induce more hallucinations [https://techcrunch.com/2025/04/18/openais-new-reasoning-ai-models-hallucinate-more/][https://www.newscientist.com/article/2479545-ai-hallucinations-are-getting-worse-and-theyre-here-to-stay/]. Extensive research have been carried-out to better understand hallucinations, and foremost determine methods to eliminate it. Even the most optimistic research [https://openai.com/index/why-language-models-hallucinate/ ] fails to adequately address this but foremost acknowledge that any possible partial solutions to this would greatly affect the ordinary operation of the Generative AI solution. One concrete case of this is the hyperparameter Temperature. This parameter can be configured to lessen hallucination to a degree. Yet, by tuning this parameter the capabilities of the model is reduced, as it directly influence the models’ ability to generalise and create novel output. It is therefore not possible to appease both facets at the same time with the Temperature parameter.

To illustrate this even better consider the system cards for some of the most well known current solutions and their models[https://assets.anthropic.com/m/64823ba7485345a7/Claude-Opus-4-5-System-Card.pdf][https://cdn.openai.com/gpt-5-system-card.pdf ]. These are created by the developers themself (it can be contemplated how impartial these are). In any case, as an example we can consider the system card for GTP5. In it, it is stated that the current hallucination rate for their best model (gpt-5-thinking) is about 4.5%. To get a better understanding what this mean in practice we can do some simple estimations. We are approach Christmas, so let us be generous and say that the hallucination rate is only 1%. For every 100 query then roughly 1 of those are wrong. Remember that it is both wrong but also stated by the system to be true (also known in all other context as a “lie” or “bullshitting”). Taking this further, consider a company with 100 employees during an ordinary work day. Each employee works an 8 hours shift. If each worker submits a query every 5 minutes then that would mean:

{number of people} * {work hours/query interval} * {probability of hallucination} = 96

Almost a hundred wrong results during an ordinary workday for a small company. And these are all very lower numbers. Hopefully these do not affect anything important…

This might be the reason why even Googles own CEO cautions against blindly trusting Generative AI solutions [https://www.bbc.com/news/articles/c8drzv37z4jo]. At the same time Google has no problem attempting to replace large part of the search engine output by Generative AI, so one can wonder if they really care in the end. And most importantly, what is the gain if you have to double-check everything?

In a philosophical and physical sense the statistical nature of Generative AI solutions and the related hallucination concept is even more disheartening. It was previously discussed in the introductory section of the blog what distinguish good or valuable Information. A core component of valuable information is that when taking together with outside factors it conveys meaning and intent. In that sense it is not possible for Generative AI to produce universally valuable information. On the opposite side, the erroneous information that can be produced do not constitute a lie or a mistake. A lie is made with the intent to deceive, a mistake with the intent to do something correctly. To blatantly disregard if something is correct or not is more akin to the philosophical concept of “Bullshit” [Frankfurt, Harry (2005). On Bullshit]. This is explored in the excellent article [https://link.springer.com/article/10.1007/s10676-024-09775-5 ]. Related to this is the inability to produce output with uncertainties. Ask a person a question which they might not know. How do most respond? Almost always by adding something references their uncertainty about the subject, along with a guess if the person can make an educated one. A “Bullshitter” will by their very nature just state the guess without any concept of uncertainty. Disregarding any concern if the answer is truthfully or not. A system operating in an analogous manner will in the end render all information processed as worthless.

The conclusion is that Generative AI solutions cannot be counted on to contain or producing reliable information of value. Information contained in the model can be modified when training data is taken from external sources. By their very nature models have no concept of truth, errors or uncertainty. Likewise, they lack the context to determine if information truly is valuable and useful. The model might imitate a behaviour in contrast to this, but it is only a mirage which can at any time disappear, leaving an unreliable mess of information in its place.

Availability – Generative AI is expensive and difficult to provide

We have arrived at the last CIA-triad concept. This part of the argument will use the fact that Generative AI is an extremely resources and money hungry industry. From these observations have the notion of an “AI bubble” been raised. I will not explore this in any detail. Mostly because this is in the news all the time now [https://www.nasdaq.com/articles/prediction-artificial-intelligence-ai-bubble-will-burst-2026-heres-why ] [https://www.businessinsider.com/ai-bubble-mentions-surge-earnings-call-data-2025-12 ][https://www.theguardian.com/technology/2025/dec/01/ai-bubble-us-economy]. For those wanting a thorough treatment of the unreasonable economics behind Generative AI, the excellent work of Ed Zitron is highly recommended [https://www.wheresyoured.at/]. But let me just summarise some of the more salient points:

• Generative AI require investments at a scale never seen before
  • This includes both money, hardware and power
• No actor, beside Nvidia, makes a profit
  • Revenue is not to be confused with profit.
• Every Generative AI solution is heavily subsidised
• Cost control for Generative AI is nothing like traditional services
  • Each query can consume a non-deterministic amount of resources due varying complexity and because of the statistical nature of the output
  • In general, Generative AI is the most expensive type of computation service that can be used due to the need of specialised hardware.

How does this relate to Availability? Well, building a system on a solution that is widely unprofitable calls into the question if this solution even will exist in two years time. Even if the solution exists then, what will the cost be? Will it even be feasible to pay those costs? This directly connects to the concept of Availability. Uncertainty regarding a solution continues existence and cost directly propagates to uncertainties if the information processed can be accessed. Hence, this line of thoughts boils down to the downfall of the AI bubble. I will not argue about the economics of Generative AI. Playing devil’s advocate the outlook on any possible bubble can be reversed. In this case we expect the demand for Generative AI to explode instead. Could this also affect the Availability for the solutions? Absolutely. However optimistic you are about Generative AI it is hard to argue against physics. Both in present times [https://www.bbc.com/news/articles/cj5ll89dy2mo] and as projected [https://www.technologyreview.com/2025/05/20/1116327/ai-energy-usage-climate-footprint-big-tech/] the energy consumption of Generative AI is so large that it is unfeasible that all of it can be satisfied. Some sort of bottleneck effect will arise as the power grid for example is not ready for such an enormous build-out. So even in the optimistic case you will encounter problems. This causes the same kind of effect on Availability. Chances that certain Generative AI solutions disappears and that the associated costs can skyrocket with short notice. The end result is the same in both cases. It is no wonder that OpenAI wants to build so many data centers, and are having issues doing so [http://forbes.com/sites/paulocarvao/2025/12/06/why-openais-ai-data-center-buildout-faces-a-2026-reality-check/].

A brief mention of two often cited counterarguments against the reasoning above. The first is that something drastically will happen that either makes all of Generative AI profitable (almost overnight) or that the computational and power requirement will be lowered to a large degree (often this focus on the cost of Interference). This kind of reverse-reasoning is hard to build up with any substantial facts. Instead, most likely the more complex models that existing today as compared to two yeas ago consume more power. The cost of Interference is discussed in [https://www.wheresyoured.at/oai_docs/]. The first counterargument involving sudden reduced costs is therefore mostly unsubstantiated. The second counterargument is that locally hosted Generative AI solutions are a saviour. For individual people this might be the case. But for bigger organisations it is not plausible that this could be done. There is a reason for example that most organisations utilise Cloud solutions. Even if this could be done, the associated cost for the hardware and power utilisation would only shift to the organisation themself. Just look at the recent RAM price spike to see how even hardware prices no longer can be estimated and controlled [https://www.tomsguide.com/computing/ram-prices-are-exploding-heres-why-and-everything-you-need-to-know-about-surviving-ramageddon].

The conclusion is that Generative AI solutions cannot be counted on to provide reliable and cost-controlled service. The looming possibility of rampant cost increase could in practice make information stored or processed by a Generative AI solutions inaccessible. Moreover, the high power requirements of Generative AI could mean that the provider cannot guarantee service for all customers, making information inaccessible at unexpected time of high demand. This is all caused by the high computational requirements of Generative AI and the inability to estimate query resource consumption due to the statistical nature.

Benefits versus Risk

With the argument laid bare the question then turns to if Generative AI still is worth it, regardless of these risks. If the combined risks for each category in the CIA-triad were to be graded, each of these would receive a large risk value. (Personally I would put the maximum possible value for each but I guess there are some wiggle room). This is damning. But if the value provided by the solution is extreme then it is always possible to simply accept the risk [https://www.isc2.org/Insights/2025/10/risk-acceptance-sticking-plaster-solution]. This is an establish method in risk management. Because of this it is instructive to briefly explore the benefits or value of Generative AI. Of all the discussed topics this is by far the most controversial and polarising.

It has been observed that when it comes to benefits and usage of Generative AI there are two major points raised. Those are current capabilities and those that might be possible in the future. The most important of these are of course the current, as future capabilities and benefits still are up in the air (because nobody knows the future).

Almost all of the often stated usage and benefits of current Generative AI solutions are anecdotal. Someone can tell you about how fantastic a Generative AI tool is for their specific use case, while still not being able to explain how all of this could not have been done in any other way or why this is vital and saves time. To get a better understand about the current capabilities it is better to consider studies and results for organisations that have implemented Generative AI solutions. As it has been a couple of years since the advent of ChatGPT and the like, a number of studies are staring to appear. If the benefits where to be tangible then most studies would indicate this by way of increased efficiency or productivity. Unchanged or even decreased results seems however to be the current findings of many studies. We cannot explore these in any dept, but the interested reader has no shortage of material [https://cmr.berkeley.edu/2025/10/seven-myths-about-ai-and-productivity-what-the-evidence-really-says/][https://www.apolloacademy.com/ai-adoption-rates-starting-to-flatten-out/ ][https://www.economist.com/finance-and-economics/2025/11/26/investors-expect-ai-use-to-soar-thats-not-happening ][https://www.bloomberg.com/news/articles/2025-12-03/microsoft-shares-slide-on-report-of-low-demand-for-ai-software][https://www.techspot.com/news/110139-new-data-shows-companies-rehiring-former-employees-ai.html][https://papers.ssrn.com/sol3/papers.cfm?abstract_id=5136877][https://www.theverge.com/report/829137/openai-chatgpt-time-date][https://mikelovesrobots.substack.com/p/wheres-the-shovelware-why-ai-coding] [https://fortune.com/2025/08/18/mit-report-95-percent-generative-ai-pilots-at-companies-failing-cfo/ ] [https://arstechnica.com/ai/2025/07/study-finds-ai-tools-made-open-source-software-developers-19-percent-slower/ ]. My favourite is the blog that discuss the non-existing increase of the amount of shovelware (low quality software) in application stores. If Generative AI coding solutions and “Vibe Coding” were so good then it would be easy for many people to start outputting software, flooding the market. But statistics from most software stores (for example Google Play and Steam) indicate a almost constant amount of software as compared to times before Generative AI.

Frequently it is stated that Generative AI solutions should not be judged on their current capabilities but by what could be possible in the future. Repeated phrases heard are “this is the worst it will ever be” and “it will only get better”. It is a misconception that this is always the case for every form of technology. Improvements could mean that the current form of the technology has to be entirely scraped and replaced. For specific technologies this is almost always the case. Iterative improvements have a limit. Multiple reasons exist for suspecting that Generative AI and Large Languages Models in particular cannot be improved indefinitely. Many research articles and experts seem to share this sentiment [https://machinelearning.apple.com/research/illusion-of-thinking] [https://onlinelibrary.wiley.com/doi/10.1002/jocb.70077][https://theconversation.com/can-bigger-is-better-scaling-laws-keep-ai-improving-forever-history-says-we-cant-be-too-sure-270448 ][https://gizmodo.com/yann-lecun-world-models-2000685265 ]. Regardless, benefits should not be measured on speculative and hopeful thinking, especially when many fundamental problems are starting to be identified.

So where does that leave us? To start with, it should be clear that Generative AI should primarily be judgment based on the current capabilities and use cases. All claims of the benefits that will come, such as curing cancer, solve climate change and stuff like that should be dismissed
[https://www.theatlantic.com/technology/archive/2025/04/how-ai-will-actually-contribute-cancer-cure/682607/ ]. In its current form the benefits and usage is almost always overstated and most of the claimed increases in productivity is a mirage. What remains are those anecdotal stories often presented; How some find usage and benefits in certain situations. In those cases I want to remind of and invite to pounder the following:

• Do you care about the reliability of the output? Can you depend on it?
• How much would you be willing to spend on the solution? Would accept dramatically higher cost? Combined with adds? [https://www.bleepingcomputer.com/news/artificial-intelligence/leak-confirms-openai-is-preparing-ads-on-chatgpt-for-public-roll-out/ ] [https://www.businessinsider.com/chatgpt-ads-rumors-openai-nick-turley-2025-12]
• Have you considered any other solution or tool that could accomplish the same result? Are you comparing tools/solutions on equal footing?

In the same vain of anecdotal experience, almost always none of these factors are even considered when speaking to most people. When searching for information for example, a search engine is still vastly superior. The latter allows one to see the source and the context of the information, as well as judge its reliability. Moreover, all current search engines operate as sustainable businesses that already are heavy monetised. An equal form of sustainable Generative AI solution would have to cost much more than now and would probably need to feature adds. Moreover, the very design of modern search engines have been adapted to maximise the profit of the companies behind them by making queries less effective. What would the result be if this same principle were to be applied to Generative AI? It is hard to see how this would be advantageous compared to the current form.

Some usage of Generative AI must exist? Generative AI can be the solution of choice if you do not have to care that much about the correctness of the output. Proof of concepts could be one example, where quick iterations could be made and where the nuance of the output do not matter that much.

Generative AI can be used when the Integrity and correctness of the output information is not that important; When the information processed by the solution is not Confidential; And when any lack of Accessibility can always be managed by backup solutions.

Another way to think of it is that Generative AI is not a universal tool for everything. Instead, we should recall the phrase: Different tools for different situations. Remember: If you have a hammer everything looks like a nail. But in this case the hammer is made of gold and can at any moment spontaneous explode. Oh, and the nail always becomes crooked.

Looking forward

This might have been one of the most negative blog on Generative AI so far. I choose to take this approach as there is no lack of optimise in other places. During the last couple of years these risks have become evident to me. It is important to try to give another side to what is currently an unprecedented hype cycle and scramble to make everything, everything, AI. But I understand that this makes me a downer and a pessimist. Not a fun guy at all. Good thing that I have some company [https://www.schneier.com/blog/archives/2025/08/we-are-still-unable-to-secure-llms-from-malicious-inputs.html].

Generative AI is not the same as any technology. It would never have been acceptable for a database solution 20 years ago to produce different och ever-changing output for the same input. To not be able to remove sensitive information in the database. Nor would it be tolerable if the looming question of access to the database software would be hanging in the air.

What is even more staggering is that these Generative AI solutions are built upon even further to attempt to achieve some kind of automation. So-called “Agents” compound all of these risks and make the end result even more insecure by increasing both the likelihood of catastrophic failures and the resulting consequences. Multiple examples are known such as [https://fortune.com/2025/07/23/ai-coding-tool-replit-wiped-database-called-it-a-catastrophic-failure/] [https://futurism.com/artificial-intelligence/google-ai-deletes-entire-drive]. An additional reason for sounding the alarm bell.

Hopefully this has made some of you more mindful and guarded against all the risks with the current unregulated roll out of these solutions. Depending on the reception I might explore additional related topics in the future. Beside making a deep dive in any of the discussed subjects, I might also examine things such specific solutions/products and their horrendous security implications (looking at you Microsoft) or the inner workings of some of the talked about hacking techniques. Let me know if this would be of interest.

At the current rate with Generative AI we in for a wild ride in the future. So let us prepare together, it is dangerous to go alone! And remember: In the security world it ought to be a complement to be a called a “stick-in-the-mud”.