AI's Entropy Problem
If your child’s hot chocolate is too hot, you can put an ice cube in it to cool it down. You can’t do that backwards though. Even though a room temperature hot chocolate includes fast moving hot particles and slow moving cold particles, we can’t collect all the cold particles into an ice cube and pull it out to heat up the hot chocolate. The hot and cold particles are thoroughly mixed, and it takes a lot of energy to sort them. This is the second law of thermodynamics. The universe moves towards randomness and disorder, and it takes energy to impose order and remove randomness.
We see something similar in information theory. A picture with more randomness or disorder takes more information to describe. If you need to describe an entirely blank, white picture, all you need to say is that it is an entirely blank, white picture. However, if you have a fall landscape with numerous trees of all different colors, you would need to describe the colors of each tree, what kind of tree they are, where they are located, etc. Another option is to lose some of the details so that you can describe the picture with less information. Maybe, we don’t need to know what type of trees they are. The leaves are just kind of blurry. We lose some of the details, but the picture is still good enough to enjoy the beauty of it. This is called lossy compression. We have lost some details, but we have compressed how much information we need.
High entropy and low entropy problems
Generative AI implicates both thermodynamic and information theory entropy. AI agents consume large amounts of energy to try to choose a next word or sentence that is intelligible and not just randomness. Generative AI has been trained on much of the publicly available information, but the training is a form of lossy compression. Information is lost, so the original data cannot be entirely recreated from the compressed version. Some of the nuance and fine details are lost. (This is good because we want to avoid overfitting. Plus, there is some information we might not want in the training data, like a photobomber in our fall landscape picture obscuring details of the picture.) Generative AI systems often struggle with fine details or problems with high complexity. There is a limit to how well they can impose order and produce intelligible results when there is a lot of randomness, i.e., a lot of potential answers or states, or when the result needs to be very finely ordered. I will call problems with high complexity where the fine details matter “high entropy” problems. Examples of high complexity areas include generating new knowledge or generating systems that function correctly almost all of the time. In contrast, I will use the term “low entropy” problems for problems of low complexity or where we don’t care about the fine details or where some hallucinations are okay (we just need good enough).
So, where does that get us? Initially, it is important to recognize that this is a limitation in current generative AI systems. When deciding whether and how to use an AI solution for a problem, we first need to decide whether it is a high entropy problem or a low entropy problem. Solving low entropy problems with AI is somewhat straightforward because you can just hook up the AI and let it rip. Solving high entropy problems is more difficult.
Strategies for high entropy problems
When we want to use AI with a high entropy problem, we need to have strategies for dealing with the limitations of current AI systems in solving high entropy problems. A first option, straight from the second law of thermodynamics, is to throw more energy at the problem. There are many AI coding tools that do this already. Throwing more energy at the problem can include using a reasoning model that uses more energy to think about a problem before generating an output. The agent can generate a plan or to-do list and then execute the steps of the plan. You can also run multiple agents and see which gives the best result.
A second option is to experiment. We give the generative AI system a way to evaluate the correctness of its output. (This also uses more energy to give us more correct results.) Despite the entire AI training process being an inductive process, AI models themselves often try to reason or guess their way to an answer, sometimes with horribly incorrect results. AI models have little common sense or intuition about the operation of the real world and often don’t have the broader context around a request. In high entropy problems, this lack of information or the large number of possible outcomes can make it very difficult to get the right result in all possible scenarios. Experimentation allows a solution to be iteratively improved until the right result is achieved in most or all scenarios. This isn’t just an AI problem. It is inherent in the knowledge generation problem that energy (often spent on experimenting) must be used to generate that knowledge. Training the AI model is one way of using experimental knowledge, but it often isn’t the best approach given the high energy usage, high cost, and large data sets required to pursue this option. Instead, experimental knowledge can be used to refine the prompts or context given to the AI model.
A third option for high entropy problems is to use previously gained knowledge. Much like leaving the window open after running an air conditioner, you can waste the energy spent combating entropy if you don’t keep the knowledge gained. One way to do this is to use documents of learnings or processes already developed by people. Another way is to save knowledge or learning generated by the AI agent (such as those generated with the second option above). The documents can be made available to the AI agent to leverage this existing knowledge. The hard problem here can be making sure that the AI agent has access to the documents it needs without including unnecessary information that pollutes context.
A fourth option is to split problems into human and AI portions with the AI agent handling the low entropy aspects of the problem and the human handling the high entropy portions of the problem. The tricky aspect is whether the AI can identify high entropy problems to be handed off to humans or whether a human needs to be the one to split the problem into subproblems appropriate for humans and AI agents. If this can be determined once and then used repeatedly for the same or similar problems, there are clear gains to using AI. If each problem needs a human to analyze it, the gains from AI are likely to be more marginal.
A fifth option is to split a problem into several low entropy problems that can each be handled by AI, and if needed, the pieces can be reassembled into a more complex whole. Again, we run into the issues of whether an AI agent or a human needs to split the problem into pieces, and if it is a human, whether they can split the problem once or if it needs to be done for each new task.
A sixth option is to solve problems without using humans or AI. Traditional automation and algorithms can reduce the workload and the set of problems that need to be solved by the AI. Organizations can also change their processes or the requirements for their outputs to reduce the complexity of generating the outputs. In both cases, we remove unnecessary complexity and allow the AI to solve lower entropy problems.
Lastly, we need to avoid adding unnecessary complexity to a problem. We need to make sure we don’t accidentally turn our low entropy problems into high entropy problems. There has been a lot of discussion of context engineering and harness engineering. When the context for a model gets too large, the problem becomes more complex for the agent. Some portions of the context are relevant to the desired output, but large portions may not be. When the context grows, the number of irrelevant pieces of information also grows (potentially at a factorial rate if this is a combinatorics problem). The problem of getting the correct output from an AI is sometimes a problem of data management. The AI agent needs to have the right data at the right time that enables it to provide the right answer without having too much irrelevant information that leads it astray. One of the big unsolved problems for anyone using AI is having information in a form the AI can ingest and having a way to efficiently provide the right portion of that data to the AI.
What this means for patent drafting
Often, the best results will be obtained by using several techniques. The appropriate techniques will depend on the particular problem. Fenix AI has been working on using AI to generate patent applications and office action responses. These are high entropy problems with long, energy-intensive, high consequence feedback loops. (A response from the patent office may not be received for years and some mistakes may be uncorrectable. A human can review the output, but such a review requires significant amounts of time from an expert.) The difficult feedback loops limit the ability to use experimentation (over the entire feedback loop) to evaluate the output of the AI. Throwing more energy at the problem isn’t optimal. There is a lot of cost pressure for patent practitioners, so we cannot simply use however many tokens it takes.
There is a generally accepted process for how you write a strong patent application and specific MPEP rules that further specify how patents are drafted. The process and rules allow humans to break the writing of a patent application into manageable pieces. Fenix AI has created tools that do the same for AI. Rather than training our own AI model, we allow our users to start from their favorite model. We use our knowledge of patent practice to split drafting a patent application or office action response into a set of high entropy problems for the human and a set of low entropy problems for the AI agent so your model gives you better outputs. Overall strategy and other high entropy considerations are reserved for the human. Each step in the process (claims, figures, detailed description) is a subproblem to be worked with rules and tools that reduce the entropy (and token usage) and allow the agent to generate a high quality draft. We offload work to traditional automation to reduce token usage and avoid unnecessary context pollution. We capture previously generated knowledge using content libraries, retrieval augmented generation, and the use of previous applications as starting templates, so using your AI agent is no longer like running the air conditioning with all the windows open. The end result is higher quality applications with reduced token usage all in a human understandable and controllable workflow.