“Analyses of data from the early 1990s showed significant increases in fatal run-off-road crashes with ABS, on the order of 28 percent. The increase was baffling, given the success of ABS on the test track. However, at that time, many drivers did not yet know how to use ABS correctly. During the mid-1990s, the safety community worked hard to inform the public about the correct use of ABS (“Don’t let up on the brakes”; “Stomp, stay, and steer”)”.
In the mortgage industry, errors in document verification workflows significantly impact the overall accuracy and efficiency of the process. The use of AI has the potential to significantly reduce these errors, provided we carefully orchestrate the human and AI interaction. This interaction has to be designed upfront - and not later added as an afterthought. Or else, as in the case of ABS, we might fail to get the value out of a new technology.
The Challenge
Manually searching for information across multiple documents is tricky for several reasons.
- Volume and variety: Handling a large number of documents in different formats makes it hard to find the relevant details quickly. Documents from different institutions typically have different formats for the same type of information. For example, one offer-letter might list deductions differently from another.
- Searching and cross-referencing: When reading through multiple documents it is easy to miss or misinterpret important details, especially if the documents are lengthy or poorly organized. For example: Validating total income often means cross-referencing details across multiple bank statements, Form 1040, Form 1099, W-2.
- Fragmented focus and attention over long periods: This process of searching and validating information is iterative. It is time-consuming and requires careful attention to detail to ensure accuracy. After the initial analysis and requests for supporting documents, the trail grows cold. The documents could take anywhere from a day to a week to arrive and the loan officer moves on to other open applications. The receipt of new documents reopens this trail. Not at all an easy task to pick up the thread from where it was left a week back.
It is no wonder then that “human error” is a well-documented issue in document verification processes.
Technology Comes to Aid
A typical technology solution for document verification comprises of the following layers:
- Extraction: Manage documents and extract information from the documents. Various off the shelf open-source and commercial tools exist. Example: Tesseract, ABBYY FineReader, Google Cloud Vision API.
- Inference: A combination of AI/ML to identify patterns and predict validity. NLP tools to detect meaning and inconsistencies. Rule-based tools to enforce cross validation across documents.
- Orchestration: This layer orchestrates the business processes, e.g. through integration with the ERP system. BPM tools and Enterprise Integration Platforms fall in this category.
The technology choices and the context at each of these layers affect the design of human, AI interaction. Let us consider an example:
Conventional OCR software typically achieves about 98% accuracy for recognizing text on standard quality document images. Let us assume a superlative performance of 99.9% accuracy through AI. The AI is quite good but not infallible. This 99.9% accuracy translates to 10 errors in a typical form 1040 with 2000 words or 10,000 characters.
These 10 errors could lead to significant issues. An extra digit added to income during the OCR process would create havoc with the downstream business process of income verification. Therefore, the business process using OCR has to ensure that a loan officer validates automated extraction.
The solution design has to aim for the following:
- Not provide too many false positives and draw the loan officer’s attention to incorrect issues
- Not let too many issues escape - leading to more complex issues downstream in the document verification process.
Hence the need to carefully design the business process and add just enough Human<>AI interactions.
AI and Human Collaboration
As illustrated by the “ABS study”, a new technology introduction can throw up surprises. By combining AI's high accuracy with human oversight, organizations can create a more robust verification process. AI can handle the bulk of data extraction and initial verification, flagging potential issues for human review. Once the data is approved, further hybrid workflows of cross- document validations could kick in. This collaboration can lead to a dramatic reduction in overall error rates.