So this week, a case was making legal headlines about the court’s displeasure with the outcome of the use of AI.

This post isn’t about that case, nor is it about all AI being useless. Of course, there are some products which are useful for certain things. This post is about the massive risk (as I see it) of using useless (or worse than useless) AI to try to solve legal problems, or indeed any problems. 

A light-hearted example for me was asking which club had the best top-flight win percentage against Everton. The platform confidently answered, even with supporting statistics, stating that no club had a 100% record. I knew that wasn’t right, so I pushed further.

When I asked about Carlisle United, it told me that they had one win, and one loss. I then asked for the scores and it gave me 2-3 and 3-0, both in Carlisle’s favour. So I asked it why that wasn’t two wins for Carlisle. It said yes, it is two wins for Carlisle. I then asked it again who had the best percentage top-flight league record against Everton, and it told me it was Carlisle. As an aside, whilst writing this, I’ve just asked the same platform again and it gave me the wrong answer again…

The point is that AI can be dangerous if you do not already know the answer. But if you already know the answer, why are you asking the question… I’ve also unsuccessfully tried it in relation to generic ‘black letter law’ scenarios. It clearly didn’t understand some legal principles, let alone nuances, but it confidently gave me a completely incorrect answer. Out of some frustration, I simply typed ‘but that is wrong’, without explaining why. And to my amazement, the response was to simply agree with me that it had got it completely wrong and it literally went on to reverse its original advice.   

So my concern here isn’t that AI often gets things wrong (of course it does), or that lawyers might be misled (I have little sympathy for the latter), my concern is that non-lawyers/clients might put a legal question into a popular platform and get an erroneous hallucination, which is significantly worse, (or perhaps even the total opposite) of the correct legal position. And if the other party to that same question puts a similar question into the same platform, the chances are they will get the same outcome - so those parties might then directly negotiate a settlement to their detriment without ever understanding their actual rights or entitlements.

It seems very dystopian to me (even more so than Carlisle United spending yet another season in the National League) where this is allowed to happen, and that’s before we even get into whether those controlling AI responses will seek to gain advantage from manipulating the output…