The 2024 Big Data London Summit spotlighted the sweeping transformation across industries driven by data-driven innovation. Over two dynamic days, leaders from various sectors gathered to explore the cutting-edge trends shaping big data, highlighting its vital role in revolutionising industries like finance, healthcare, and beyond. Discussions centred on the latest advancements in generative AI, data strategy, and architecture, offering attendees invaluable insights into how businesses leverage big data to stay ahead in a fast-changing world. In this recap, we’ll dive into the summit’s key takeaways, and explore what these findings mean for businesses.
Big Data in Industry
Big data’s transformative impact on various industries was a key theme at the 2024 Big Data London Summit, with significant attention given to its application in retail, healthcare, and construction.
Retail
In the retail sector, speakers from companies like Ocado and Dunnhumby delved into how big data is enhancing customer experience and operational efficiency. The discussions focused on real-time analytics to drive hyper-personalised shopping experiences and optimise inventory management. As businesses shift towards more digital models, data is being used to improve demand forecasting and refine precision marketing, driving customer loyalty in an increasingly competitive landscape.
Big Data London Summit
Healthcare
In healthcare, data is indispensable in improving patient outcomes and research advancements. Sessions led by NHS Digital explored the role of data in predictive healthcare models, highlighting how big data is helping to forecast disease outbreaks and tailor treatment plans based on large-scale patient data. However, challenges around data privacy and system integration remain significant barriers to wider adoption.
Construction
Traditionally a late adopter of data technologies, is now leveraging big data to optimise its supply chains and operational processes. Leaders from Travis Perkins shared insights into how they are fostering a data-driven culture, despite the inherent challenges of low data literacy across their workforce. Through initiatives such as apprenticeship programs and hackathons, they are building a foundation for more robust data usage across the business.
Nevertheless, the sector faces ongoing hurdles in fully integrating data solutions into long-established practices, particularly around trust in data quality and the upskilling required to harness its potential.
Across all sectors, speakers stressed the importance of building strong data governance frameworks and the need for real-time data access to drive actionable insights. As retail, healthcare, and construction continue to adopt big data technologies; the scalability of these systems and their ability to integrate with legacy infrastructures remain critical concerns.
Leveraging Data with Generative AI
Generative AI was a central theme at the 2024 Big Data London Summit, with discussions emphasising the importance of context in AI outputs.
Contextualising Generative AI with Data: Rather than relying solely on fine-tuning models, companies like Tecton advocated for retrieval-augmented generation (RAG), a technique that combines the power of large language models (LLMs) with external data retrieval systems to embed rich context in prompts.
Instead of using the model’s pre-trained knowledge, RAG dynamically retrieves relevant information from databases or documents at the time of query, significantly improving accuracy and personalisation. This approach incorporates historical, real-time, and feedback-driven data, allowing businesses to deliver more contextually accurate and tailored experiences in real-time.
Scaling and Optimising Data for Generative AI presents unique challenges, particularly around data governance and transparency. Speakers from Starburst Data and Microsoft highlighted the need for federated data systems where the origin of data is traceable. Strong governance ensures business users can trust the AI outputs, avoiding common issues like hallucinations. Retrieval quality and properly grounded responses are essential for making generative AI systems reliable and scalable.
Real-Time Data and Personalisation: Real-time data is playing an increasingly critical role. Traditional batch data is no longer sufficient, as AI models now require continuous streams of information to remain relevant and responsive. Tools like Kafka enable real-time event-based architectures, allowing businesses to refine recommendations and generate dynamic, user-specific insights.
Optimising Data Flows with GraphRAG: One of the key advancements discussed at the Big Data London Summit was GraphRAG (Graph-based Retrieval-Augmented Generation), which combines knowledge graphs with retrieval-augmented generation to improve how AI systems interact with data. Knowledge graphs store data as interconnected nodes and relationships, allowing AI models to retrieve information in a more structured and contextualised manner.
For businesses, implementing GraphRAG could enhance the accuracy and relevance of AI-driven insights, especially in complex, data-heavy sectors like energy and supply chain management. By leveraging knowledge graphs, AI systems can generate responses that are more aligned with enterprise-specific information and provide clearer reasoning behind the answers.
Building Trust and Governance in Generative AI: Governance frameworks were emphasised throughout the summit, with IBM and Microsoft stressing the importance of building data products aligned with business goals. Well-structured data products, enriched with business metadata, ensure that AI models deliver meaningful insights while minimising risk.
Agentive AI: Another prominent concept explored at the Big Data London Summit. This approach involves the use of autonomous AI agents that can perform complex tasks with minimal human intervention. Unlike traditional AI, which requires human input to trigger actions, agentive AI systems can make decisions and take actions independently by continuously interacting with data and learning from it.
“For businesses, Agentive AI presents an opportunity to automate repetitive and time-consuming processes, such as data analysis and decision-making in areas like supply chain management. By deploying AI agents, large businesses can enhance efficiency and agility across their organisations, allowing humans to focus on more strategic tasks.”
Future Trends in Generative AI and Data Integration
Looking forward, autonomous AI agents—capable of operating without human intervention—were identified as a key trend. These systems will rely on well-governed, real-time data to drive automation across industries. However, challenges remain in scaling these technologies, and organisations must integrate AI and data strategies seamlessly to unlock their full potential.
Creating An Effective Data Strategy
The discussions highlighted the importance of a strong data culture, growing data maturity and empowering employees to work with data.
1.Data Culture
Several companies, such as Travis Perkins and Sage, are cultivating robust data cultures by enhancing data literacy and implementing data governance processes to ensure data quality and access control. These initiatives are revolutionising how their organisations value, access, share, and utilise data. Additionally, they conducted organisational surveys to gather employee feedback on data engagement. This helps them identify current challenges and refine their data strategy accordingly.
2.Data Maturity Assessment
Many sessions emphasised the importance of organisations understanding their data maturity (the level of an organisation’s ability to effectively manage, use and derive value from its data) and finding ways to enhance it. It was noted that many organisations are unsure of their position on the maturity curve and which tools or solutions to leverage. By analysing processes across the organisation and identifying data shortfalls such as poor data quality, data silos etc, businesses can better understand their place on the maturity curve. This insight can help determine whether simple data aggregation solutions might offer more value than complex AI solutions in the short term.
3.Data Flywheels
Numerous talks highlighted the importance of creating data flywheels—systems where interactions with AI or automation generate feedback loops that enhance future performance—for maintaining a sustained competitive advantage.
“SnowPlow Analytics: Technology alone is no longer a competitive advantage.”
Speakers emphasised the importance of organisations differentiating themselves by analysing proprietary data on products, services, and customers to gain better insights and drive innovation. As one speaker from SnowPlow Analytics, a SaaS data analytics platform, noted, “Technology alone is no longer a competitive advantage”. Leveraging the proprietary data that organisations already possess is crucial for standing out from competitors.
A key component of the data flywheel is establishing governance frameworks to ensure data quality, consistency, and security across the organisation, making the data used for insights trustworthy. Additionally, organisations must build infrastructure to capture more data throughout the process, ensuring the virtuous cycle continues.
Modern Data Architectures
Data Products: Having high-quality, ready-to-use data is crucial for organisations to solve specific business problems and support decision-making. Data products (such as dashboards, live data from APIs, and data generated by predictive machine learning models) are often reusable and organised datasets and possess these characteristics. As a result, they are becoming integral to modern data architectures. They are typically designed with specific business needs in mind.
Many speakers from various data companies agreed that data products also accelerate AI innovation by providing high-quality, ready-to-use data for training AI models. Building data products can help create a strong foundation for future AI solutions across the business.
Data Relationships: The summit underscored the importance of establishing structured relationships between data to enhance collaboration within organisations. Data’s true value lies in its structure. This is what makes it knowledge. While data products address specific domain issues, the absence of interconnections can hinder cross-departmental cooperation. This is where technologies like knowledge graphs come into play. Knowledge graphs provide a structured representation of data, emphasising the relationships between data entities such as customers, companies, or products. Organising data in this manner facilitates cross-domain communication. As the CTO of Quantyca aptly put it, “Without knowledge, data is just a liability.” Therefore, while adopting a data-first approach is crucial, it must be supported by a foundation of knowledge.
Data Fabric: Numerous speakers, including Matt Quinn, the Chief Data Officer for Microsoft UK, emphasised that a data fabric is the ultimate solution for an organisation’s data architecture. A data fabric architecture enables businesses to integrate data from various environments (cloud or on-premises) for unified access without needing to relocate the data. This approach allows one department to seamlessly access and incorporate data from another into their workflow.
Data products are the modular units of modern data architectures. A data fabric ensures seamless access to all data products, regardless of their storage location. Knowledge graphs add context to these products by linking them with other data products and entities. For businesses, a phased approach of first building data products, then a data fabric architecture, and finally a knowledge graph could drive data transformation. However, the speed will largely depend on current data maturity level.
Conclusion
The Big Data London Summit 2024 emphasised the critical role data plays in the modern enterprise, underscoring the need for organisations to not just collect data, but to actively manage and leverage it.
Adopting a data-first approach is now more urgent than ever. However, collecting vast amounts of data alone is not enough—businesses need to develop a comprehensive data strategy that ensures transparency, governance, and real-time access to information. This requires breaking down silos within the organisation and fostering a culture of openness across its departments.
In the fast-paced digital economy, those who harness the full potential of their data will set the standard for future success. Businesses must prioritise this to ensure that it remains at the forefront of innovation, rather than lagging behind more agile competitors.
Authors:
Chidi Akurunwa, AI Consultant
Sim Riyat, AI Consultant
