Graphic courtesy of Law.com

Introduction to GPT-4 in Legal Practice

Introduced in 2022, ChatGPT quickly garnered attention across various sectors, including law. GPT-4, an advancement over its predecessor, is a multimodal model capable of processing both text and images. Its ability to generate high-quality content from vast amounts of data makes it a potentially transformative tool for legal practitioners.

The Sidley Austin Experiment with GPT-4

Sidley Austin, in collaboration with legal technology firm Relativity, conducted an experiment using GPT-4 for coding documents in a completed case involving the Anti-Kickback Statute. The test involved reviewing 1,500 documents, classified as responsive or non-responsive, against GPT-4’s analysis.

The experiment was structured in two phases:

1. GPT-4 was initially provided with standard review instructions.
2. Adjustments were made to the prompt based on GPT-4’s initial performance, emulating a real-world quality control process.

Findings and Performance Analysis

GPT-4 demonstrated strong capabilities in accurately tagging documents, particularly where its confidence level was high. It excelled in identifying non-responsive documents and those highly relevant to the case. However, challenges arose in coding documents part of a responsive family or containing short responsive messages, due to the limitations of the provided instructions.

Speed and Efficiency

GPT-4 processed documents at a rate of one per second, significantly slower than traditional TAR tools but much faster than human review. While this rate shows promise, it currently limits GPT-4’s utility to smaller document sets.

Implications for Legal Practice

The experiment suggests that GPT-4 could potentially reshape document review methodologies in e-discovery. Its ability to generate explanatory feedback could streamline the quality control process, enhancing efficiency. However, GPT-4's performance heavily relies on the precision and clarity of the review instructions provided.

Conclusion and Future Prospects

While GPT-4 heralds a new era in legal tech, its current iteration shows that human oversight remains crucial. As AI continues to evolve, legal practitioners need to stay informed and adapt to leverage these technologies effectively. The experiment by Sidley Austin is a step forward in understanding and integrating AI tools like GPT-4 in legal practices, promising a future where AI and human expertise coalesce to enhance legal processes.