Building Trust in AI Part IV: Trust in Business
What happens when the model behaves exactly as the business asked?
Building Trust in AI Part IV: Trust in Business
The first three parts of this series began with the data, moved through the algorithm and ended at the prediction placed in front of a person. There is still another system in the room. A company decides which data is worth collecting, which model should be built and which result counts as success.
The algorithm may be technically sound and its explanation faithful. The business can still ask the machine to optimise the wrong thing. Trust in artificial intelligence therefore reaches beyond the artificial intelligence and into the institution that gives the system its objective.
The Currency That Cannot Be Stored
The digital economy is producing more information than any person can examine. Data is copied, combined and transformed into more data. Every search, purchase, photograph and pause on a screen can become another observation. More information doesn’t guarantee more truth. It creates more candidates for attention.
Attention is the scarce part of the system. A platform can store millions of items, but it still has to decide which one appears in front of a person now. That decision becomes the commercial centre of search engines, social networks, advertising systems and recommendation platforms.
Personalisation can be useful. A search for a restaurant improves when the system knows the user’s location, and a film service becomes easier to navigate when it remembers what the viewer enjoyed. The same machinery can decide which offer, price or version of the world each person receives. The difference does not lie inside personalisation itself. It lies in what the business has chosen to optimise.
If the objective is relevance, the system learns to reduce noise. If the objective is attention, it learns what keeps the person looking. If the objective is revenue, it learns which version of the experience makes the person most valuable to the company. The model follows the metric, and the metric follows the business.
The Experiment in the Feed
In 2014, Facebook researchers published the results of an experiment involving 689,003 users. The company reduced the amount of positive or negative content appearing in selected News Feeds, then measured the emotional language those users later posted. The observed effects were small, but their direction was measurable. People exposed to less positive material used slightly fewer positive words, while people exposed to less negative material used slightly fewer negative words.
The controversy did not begin with the size of the effect. It began with the experiment itself. The users had not been told that the emotional balance of their feeds was being altered for a published study. Facebook could argue that ranking experiments were already part of operating the service and covered by its terms. Critics could reply that product testing had crossed into behavioural research involving people who were not meaningfully aware of their participation.
The technical act looked ordinary. A ranking parameter changed. The human act was harder to name because a company had altered part of the emotional environment surrounding hundreds of thousands of people to see what happened next.
This is where the older language of research ethics becomes useful. The Belmont Report centres respect for persons, beneficence and justice. The Declaration of Helsinki places the rights and welfare of research participants above the interests of science and society. Neither document was written for the continuous experimentation of digital platforms, and neither automatically settles the legal status of every product test. They do, however, expose the missing question. When a company can experiment on behaviour as part of normal operation, where does product improvement end and human-subject research begin?
The Laboratory Without a Door
A conventional experiment has a visible edge. A participant enters a study, receives information, gives consent and eventually leaves. A digital platform can alter the environment without creating that boundary. The interface remains familiar. The user opens the same application and sees the same kinds of objects while, behind the screen, the order changes, the frequency changes and the signals used to judge the response become more precise.
The laboratory has disappeared into the product. This doesn’t make every experiment unethical. Companies need to test whether systems work, whether interfaces confuse people and whether a change causes harm. The difficulty is that the same infrastructure can be used to optimise behaviour without the person knowing what is being tested.
A business can run thousands of small experiments and treat each one as harmless while the combined system shapes attention, mood and habit at a scale no individual test reveals. The ethical question cannot be reduced to whether one adjustment produced measurable injury. It must also include the power created by continuous observation.
The company can see the response. The user cannot see the experiment.
Bad Is Stronger Than Good
A broad body of work on negativity bias suggests that negative events often carry more psychological weight than comparable positive ones. This doesn’t mean the brain follows one fixed ratio or that everyone responds in the same way. It means threat, loss and social danger can command attention quickly.
The prefrontal cortex supports planning, inhibition and other forms of higher-level processing and decision-making. Under stress, those capacities can narrow as the nervous system prepares for a more immediate response. The limbic system is a rough label for several networks involved in emotion, memory and motivation. It is not a separate animal brain waiting to overpower a rational human one. The systems remain connected.
The commercial implication is simpler than the anatomy. Material associated with outrage, fear, humiliation or conflict may attract attention more reliably than material that leaves the person calm. A platform trained to maximise engagement doesn’t need to understand anger. It only needs to notice that anger often produces another click, comment or return visit.
The system may then learn to surface more of whatever keeps the metric moving. Nobody has to instruct it to divide people. The instruction to maximise engagement may be enough.
The Bell in the Pocket
A notification is a small event. It makes a sound, changes an icon or lights a screen. The phone is checked because a message may be waiting, although most checks reveal nothing important. Occasionally one delivers social approval, useful news or contact from someone valued. The irregularity strengthens the loop because the next check might be the rewarding one.
The mechanism resembles classical conditioning, but the digital version can observe the response and adjust the cue. It can learn when the person is most likely to return, which wording attracts attention and how long the interval should be before another prompt appears. The system is not merely ringing Pavlov’s bell. It is testing which bell works on this person.
This is why the language of addiction can become too convenient. It places the weakness inside the user and leaves the design of the environment untouched. The person is not encountering a neutral tool and failing to exercise enough discipline. They may be encountering a system that measures the limits of that discipline continuously.
The business knows what the metric rewards. The user sees a notification.
A Different Front Page
A newspaper prints one front page for many readers. A personalised feed can create a different front page for each person. That can improve relevance, but it can also weaken the shared surface on which public disagreement depends.
Two people may search for the same event and receive different rankings. They may follow different networks, encounter different emotional cues and come away with incompatible impressions of what everyone else believes. The system doesn’t have to invent false information to change the public world. It only has to decide which true information each person is most likely to engage with.
This gives businesses a form of influence that traditional media did not possess at the same resolution. The platform can observe the individual response, update the model and alter the next presentation. The public square becomes a set of private rooms joined by one ranking system.
That system may be designed by engineers, but its objective comes from the business. If success is measured through time spent, advertising exposure or behavioural response, the ranking will be pulled towards whatever produces those outcomes. The company cannot call the result neutral merely because no employee selected each item by hand. Automation does not remove editorial power. It makes the editor harder to see.
The Business Behind the Machine
The central question is not whether companies contain unethical people. Most organisations are filled with people trying to perform their roles well. The difficulty is that each role is shaped by the metric above it.
A product team may be asked to increase engagement. A data scientist may improve the prediction of who will click. An engineer may reduce the delay between the signal and the response. An advertiser may pay for the resulting attention. Each part can succeed while the whole system becomes harmful.
Technical transparency cannot solve this on its own. A company can explain how the model ranked the feed while refusing to question whether maximising engagement should govern the feed at all. The objective function is not a technical detail. It is a statement about what the organisation has decided to value.
A trustworthy business should be able to explain why the system exists, what it is permitted to optimise and which profitable behaviours it has deliberately placed beyond the model’s reach. It should distinguish experimentation that improves a service from experimentation that manipulates vulnerability, create review for tests involving sensitive behaviour, provide meaningful routes for objection and measure harm rather than waiting for public outrage to make the harm visible.
Some opportunities should remain unused. That restraint is part of the product.
The Last Layer of Trust
Data can be representative, the algorithm validated and the prediction explained. None of those protections can rescue an objective that treats human attention as a resource to be extracted without limit.
Trust in business begins when the organisation accepts responsibility for the desire placed inside the machine. It requires more than asking whether the model is accurate, fair or transparent. It asks what the company is trying to make happen and whether the people affected would recognise that outcome as legitimate.
The algorithm follows the objective. The business chose the objective.
Comments