GenAI: The importance of good data management

Businesses are using more data than ever – recent estimates suggest that 328.8m terabytes of data are created each day. Companies are increasingly using behavioral insights and predictive analytics to drive sales and some have devoted entire business models to effectively using consumer data. Effective data management is more critical than ever as this data is scattered across various platforms (such as SharePoint, OneDrive and local personal folders) and as more and more people work and access files remotely.

Generative AI (GenAI) is providing personalized models that allow immediate and access to large volumes of internal data, through simple conversational prompts. The possibilities are near endless. However, the successful use of GenAI solutions really hinges on a robust data management strategy to ensure accuracy, security and efficiency of the data used.  We look at some of the key lessons we have learned when it comes to robust and effective data management. 

DATA CLASSIFICATION

Of course, not all data should be accessible to everyone all the time. Implementing strict data classification helps determine access levels and ensures that sensitive information is protected. Integrating data classification protocols with AI systems will help maintain security and comply with privacy regulations as well as help data providers know what data is being used.

CHANGE MANAGEMENT

AI systems often struggle with outdated data. Without a process to inform or retrain AI models when new data versions are released, users are far more likely to receive obsolete information and the process is more likely to break down. Establishing a robust change management process to keep AI models updated with the latest data ensures accuracy and relevance. 

DATA QUALITY

During SPS’ own AI implementation, we have often encountered contradictory statements within the documents used to train AI. This inconsistency can impact results, as AI might return conflicting information. Prioritizing data cleansing to remove or correct corrupt, inaccurate, or irrelevant data ensures consistent and accurate data for reliable outputs. AI can actually be used effectively in this process to spot patterns and inconsistencies. 

QUALITY-CHECKED DATA SETS

For AI to perform to its best capabilities, the training data provided must be high-quality and diverse. Using accurately tagged datasets that reflect real-world scenarios reduces biases and hallucinations in the AI outputs. We have found that ensuring rigorous quality checks of all training and regularly fine-tuning reference datasets helps enhance AI performance. 

CONTINUOUS MONITORING AND FEEDBACK

Ongoing monitoring of AI outputs against benchmark data is essential for robust results. This process involves periodic checks to detect anomalies or deviations and helps make sure that AI remains accurate and reliable over time. User and system feedback are invaluable for this fine-tuning of AI models. This might involve regular updates to the training set with new examples to help improve accuracy. Integrating feedback loops will help address misinterpretations. 
 
 

OTHER LESSONS LEARNED

You can have too much data. In a recent contract analysis scenario, our AI was initially overwhelmed by the volume of data it was provided with. By splitting the problem into manageable parts and validating each segment individually, we were able to significantly improve the accuracy of the AI outputs. The lesson here is to break down complex tasks and validate iteratively to achieve better results. 
 
You can also trust the response too much. In another example, when querying some policy documents, AI provided the source along with an interpretation. By treating the AI response as a suggestion from a novice employee, users were able to validate the information by referring to the original source. The lesson here is to always thoroughly cross-check AI interpretations against original documents to ensure accuracy and reliability. Always assume a novice ability on the part of the AI. 

CONCLUSION

Effective and meticulous data management is essential to successful AI usage. There are lessons to learn around centralizing data storage, implementing robust change management and classification processes, ensuring data quality and continuously monitoring and validating AI outputs. By putting these data management measures in place, organizations can leverage AI to its full potential.