Today, data aggregators and data pools have the difficult job of compiling ecommerce data for hundreds of thousands or millions of products. This task is complicated by the fact that every product manufacturer has a different format for their product content.

The goal of data aggregation is to consolidate product data into a form and a format that can easily be leveraged by a search engine and presented to an online buyer in a standard format. Both search engines and human users expect to see product data presented in a uniform format.
Search engines need a uniform format to improve search accuracy. Human users want to easily evaluate and compare products.

Product content can include both structured and unstructured data sources such as:
- Website content
- Manufacturer Data sheets
- Manufacturer Spec sheets
- Manufacturer Brochures
- Spreadsheets
- Product Images
- Testing or lab results
- CAD files
- PDF drawings
- Product Documentation
- User Guides
- Databases
- Product Reviews
- User Generated Content
The Ecommerce Data Consolidation Process
It is an enormous project to collect, collate, import and reconcile all of this electronic information outlined above. For the purposes of this discussion, we’ll assume that all of the content has at least been captured electronically or is available in some machine readable format.
The first step is to ingest all of the source content. For a manual ingestion process, it can take a full time dedicated staff to collect and collate all of the content. However, artificial intelligence (AI) can now help to automate the ingestion process. AI does this by understanding learning what to look for in each source file type and then extracting and organising the relevant content into a uniform output format.
AI learns what is important and what’s not important by first watching a human process the data. As a human highlights, copies and pastes the important information from a source document, the AI learns from what is important and isn’t important.
A new machine learning model is like a infant. It knows nothing. As the AI observes an operation, it can begin to learn how to extract the content on its own. Every AI machine learning model requires training data that has been tagged as the basis for learning. Generating training data can be a large part of the training process. A target data evaluation data set is also required to test the prediction accuracy of a model. A key requirement for any machine learning process is a large enough data set for training and evaluation before a model can be deployed to a production workflow process.
Automated Validation Ecommerce Data
The validation and reconciliation of information is a critical step in the data aggregation workflow. All of the information extracted must be reconciled to ensure that the source documents agree. But how do you know that the correct information was selected for reconciliation?

A properly engineered system can identify conflicts in the source material and either make a decision or flag the data for review. This can identify down rev’d or spurious material and force it to be rejected as irrelevant. It can also identify new information and reconcile the new data into the correct attributes. Machine learning models can help to weight the decision of which data elements are correct.
Conclusion
When a data source is unstructured data such as images, the information in the images must be properly extracted and then tested for quality. A properly trained machine learning model can automate this process. With the state of the art in AI, it’s now possible to train and deploy machine learning models using the skills of business users. Data scientists will need to be involved in the structuring of the workflow or pipeline process and the selection of machine learning models. However, AutoML is quickly evolving and it won’t be long before much of this is effected by AI decision making.
The outcome is the ability to scale AI-based systems to learn faster and operate on a broader range of data without a massive retraining cycle.
Further reading: how SwiftERM works.
Further reading: SwiftERM case studies.
Evidence: NIST AI Risk Management Framework.
Further evidence and related SwiftERM analysis
Ecommerce data: the current operational framework
Ecommerce data can now be considered through SwiftERM’s autonomous individualisation. Product relevance is recalculated for each identifiable customer from that customer’s own behavioural and transactional evidence, rather than being assigned by a fixed segment, campaign or predetermined journey. This distinction matters because an individual’s interests can change between visits, purchases and periods of inactivity. A current decision therefore needs current evidence.
SwiftERM evaluates the products that are becoming most relevant to each person and applies that decision autonomously. The operating model does not depend on a marketer continually creating segments, selecting products or scheduling another campaign. It is designed to use first-party behavioural and transactional signals already associated with an identifiable customer, so the decision remains specific to that customer rather than a broad cohort.
For Ecommerce data, the practical test is not whether a retailer can create more rules. It is whether the system can recognise a change in an individual’s buying intention, reassess the available products and act without waiting for a manual campaign cycle. The output must remain traceable to the customer evidence and the ecommerce catalogue used to make the decision.
Implementation also needs disciplined data handling. Product identifiers, availability and customer events must remain consistent enough for relevance to be recalculated. Existing ecommerce operations should retain control of catalogue truth, order truth and customer permissions. Autonomous individualisation then operates within those established boundaries instead of inventing a parallel customer record or an ungoverned product feed.
The commercial value of Ecommerce data should be assessed through observed customer response, orders and turnover, not through the volume of campaigns produced. That keeps attention on whether product selection becomes more useful to the individual and whether the retailer captures demand that would otherwise remain unrecognised. It also separates autonomous decision-making from conventional automation that simply executes a previously written rule faster.
This current framework preserves the original article’s subject while bringing its explanation into line with SwiftERM’s present capability: one identifiable customer, individually recalculated product relevance, and an autonomous decision made from that person’s own changing evidence. The same principle applies as catalogue choice, customer behaviour and commercial conditions continue to change over time.
In operational terms, the retailer retains the commercial context while the software performs the individual calculation. That division is important: the catalogue and customer record remain governed business assets, while product selection responds continuously to evidence rather than to a static assumption made when a campaign was first configured.


