An example scenario:

At DMS we recently had two totally different clients each tell us they were thinking of using AI to convert word (docx) files into markdown (md) files. One of them had even gone as far as to test AI. He was surprised to find it changing and attempting to improve the document he wanted maintained as-is 🙂 This seemed surprising to us. Not the over helpful editing by the AI agent but the selection of AI for this task. We are well aware of Pandoc and it is a far more suited tool for this work.

Sound familiar? It is easy for those of us in the know to smile sagely and move on but wait…. What if we, like our clients, didn’t know about Pandoc? What about other scenarios, where we risk getting over-hyped on using AI to solve a challenge? How can we avoid walking into the same trap somewhere else and avoid using AI for a task that it is ill suited for?

The key to the scenario above is understanding that the core problem is highly deterministic. Implication being that the outcome is: (a) determined by prior conditions and rules, (b) without ambiguity and (c) always derived with a correct result that is not open to different interpretation. This makes AI alone a poor tool to solve the target problem. It may be that with a good prompt it can suggest using something like Pandoc, but the moment AI jumps in to solve using its own model, we are rolling die on a resulting output.

Handling other use cases

Ok, what about other situations? Can we give ourselves a better chance of not falling into the trap above? What if we use some base guidelines? At DMS we’ve been casually exploring groups of yes/no questions that can help us. Ideally, we’d like something quick to use that safely guides clients across different use cases. The latest iteration of this looks like the following. It can be recalled from the acronym AIDEUS (apologies to our Spanish speaking readers for destroying the spelling of adiós!).

• Absence – Is there an absence of obvious rules to solve?
• Interpretation – Is interpretation required?
• Different – Is it likely different humans would produce different answers?
• Errors – Can the task tolerate errors?
• Unstructured – Is the input unstructured?
• Summarize – Can summarizing and reasoning add value?

Each question above is designed to be answered yes/no. If the majority of responses are ‘yes’ then AI is likely a strong candidate. However, if you answered ‘yes’ to only one or two, then it is highly likely this is not a suitable AI scenario.

Finally, ensure the problem itself has been broken down. Ideally to it’s lowest suitable sub tasks. Deriving that can be a bit of an art. However if you have a middle/split score (e.g. say 3 yes/3 no), then it is likely you need to break the problem down further. It possibly indicates a hybrid solution. That is one where you still need to identify which sub task is best aligned to AI tooling.

What do you think?

Our aim in reading and thinking around this topic is to distill the decision work into something we can quickly use to minimize the risk of poorly spent AI investment. This is not unique to us and is still something we are still evolving. It can improve with more use cases. Does it align with use cases you are deeply familiar with? Are we missing any questions that may further help?