Who Wrote it Better?: AI vs. Human-Generated Law Firm Content Writing

It seems like using AI technology to write your law firm content writing will become a trend whether we like it or not.

At PaperStreet, we believe there is value in having human-written content, especially as bot content infiltrates search results and makes well-written copy hard to find. We anticipate this will be a problem in the future, but we promise to always appreciate a well-placed pun or a fully-fledged thought based on understanding and experience that a computer algorithm can never have (or so we choose to believe). 

Yet, we have to consider – is AI-generated content better?

Our AI vs. Human Law Firm Content Writing Experiment

We crossed our fingers and shook our heads, set in our ways that AI-written content could never be better than what our lawyer writers produce, so we put it to the test. 

In this experiment, we used two different sets of practice area content. We took a webpage on Special Needs Planning from a lawyer website in Long Island, and a Dog Bite Accident page from a lawyer website in Boca Raton. We stripped out the phone numbers and the firm name, replacing each with the same placeholders the bots use.

We then asked ChatGPT to “write a page on Long Island Special Needs Planning for an attorney website.” For the dog bite content our query was the same, “write a page on Boca Raton dog bites for an attorney website.”

We formatted all four articles the same, streamlining the fonts, size and colors, so there was no obvious visual difference between them.

Finally, we asked our team of 30 expert law firm marketing professionals to evaluate the following: 

• What version of each was written by a human? 
• What version of each was written by a bot?
• What page out of the four do you think is the best written?
• What page out of the four do you think is the worst written?

So, Can We Tell the Author?

Generally yes. It wasn’t quite the landslide we hoped to see, but we could identify the author:

• 63% of the team correctly identified the author of the content. 
• 83% of the team correctly identified the author of the Special Needs content. 
• 43% of the team correctly identified the author of the Dog Bite content.

Is AI-Generated Content Better?

So, is AI-generated content better? When the team was asked to rank the articles from the best-written article to the worst-written article, 80% chose the human articles as the best. 

Another interesting insight is that our core content team was able to spot the bot content. Our Blog Manager who is responsible for editing 300+ pages of content each month identified the authors with 100% accuracy. 

Final Thoughts on AI-Generated Content for Law Firm Websites

The more you work with ChatGPT, the more the rhythm becomes obvious. You can see the structure of the answers and they become identifiable as quite simple, lacking in sentence complexity and overall style. For those of us who are in the trenches, the language of bots will become easily identifiable. For professionals who don’t care about adverbs or ambiguous modifiers, the difference might not be obvious, or worse, they likely won’t care. 

AI-generated content could provide an experienced writer very rough first draft, but if used raw with no substantial edits, we will hold out hope that readers will notice – our team certainly did.