A manifesto?

After finishing grad school, I am currently stay-at-home-parenting because we don’t have childcare lined up for our youngest until the fall of 2026, so I have a small amount of free time to build and conduct R&D. One of the things I’m most passionate about is privacy and digital governance. I think Large Language Models (“AI”, these days) are an incredible tool. I use them to write code all the time. I use them to check my grammar all the time. I use them instead of the increasingly-useless Google search results when I can’t remember e.g. the command line flags for rsync.

But I don’t want to interact with LLMs socially.

Just before Christmas last year, a VP at Facebook announced they were going to debut AI-powered fake profiles to try to attract young people. I don’t know about you, dear reader, but I sure as heck don’t want to see an LLM with a fake bio on my socials. A while ago I did a deep dive on persuasive writing and in the course of that, read the Federalist papers, so I tried my hand at summing up how I feel about the state of our digital society in a Federalist-like tone:

Towards a human digital future

The Internet, our global connection of machines upon which the World Wide Web operates, is perhaps humanity’s greatest achievement. This achievement has made instant the global exchange of ideas and information. Through it, people engage in commerce, develop friendships and camaraderie, and maybe even fall in love, all despite never having met in person. Sure, we have split the atom and subjugated Mother Nature, but no feat other than the creation of the Internet has had such a transformational impact on our society. It is the single most democratic machine or engine ever devised. And yet, the Web is no longer the unmediated republic of voices it once promised to be. Increasingly, the words and signals that fill its channels do not originate in human minds, but in scripts, farms, and engines built to mimic them. On certain platforms, the distinction between human and machine is already lost; in these online spaces one cannot say with confidence whether the hand behind a message belongs to a human being or to a server process in a datacenter outside Moscow or Tehran. This is no innocent curiosity. Our society increasingly conducts its debates, its commerce, and its governmental processes through digital means, and the erosion of trust in the humanity of our online counterparties is an erosion of society itself. The incentives that reward fakery are not transient, but permanent features of the system. If we do not prepare for the day when artificial voices are indistinguishable from our own, we risk losing the very possibility of authentic online interaction and discourse.

Few today would trust the authenticity of an Amazon review; experience has taught us better. And for the unscrupulous, good news! Upvotes, retweets, and Likes are all easily purchased online. Nor is the problem confined to product reviews. The very engines we once trusted to guide us (Google foremost among them) are themselves compromised. Search results are now a morass of automatically-generated spam masquerading as content, and users increasingly turn to Large Language Models (LLMs) like ChatGPT to learn information, which requires the AI companies to continually scrape blogs and other sites with punishing frequency, requiring yet more centralized services like Cloudflare to protect indie servers from the barrage of web requests. “Web 2.0” forums died years ago, as their users left for “Social Media”: walled gardens that provide rewarding and addictive dopamine treadmills via push notifications, exacerbating the trend of increasing centralization. Those platforms that remain are subject to continued disruption, both benign and malicious. Foreign state actors posing as locals on municipal or neighbourhood Facebook groups, undermining public confidence in elections integrity, amplifying polarizing topics, and generally sowing discontent. Content marketers can create plausible fake identities and reviews or blogspam at the click of a button.

The point, in sum, is not that the internet is dead, but that the living, vibrant Web we inherited stands in peril, and will not long endure in the shape we currently recognize. Its future is jeopardized by an ever-increasing proportion of synthetic content, by content farms driven by perverse financial incentives, and the slow but inexorable trend of consolidation into a handful of walled gardens.

The writing has been on the wall for some time. Indeed, the problem of synthetic content existed long before ChatGPT was born. Spam contributed heavily to the death of Usenet. If a Usenet user misbehaved, it often required speaking to their university or their employer, an inherently slow and unscalable process. With the Eternal September, that same decentralization came to be seen as a vulnerability, because only a centralized moderation service could effectively shield users from forged NNTP Cancel messages, or vitriolic flame wars. The same remains true for email, where small independent mail servers fight an uphill battle against the capricious and arbitrary whims of behemoths like Gmail. It is no longer sufficient to implement DKIM, DMARC, and SPF. Opaque, byzantine processes around “IP reputation” exist, often silently dropping mail deemed insufficiently reputable, with no way for individual users to contact a human sysadmin, with the end result that many small-time mailserver operators simply give up in frustration. The very same mechanism began with the ascendancy of Google, with its PageRank algorithm. As the web grew, so did its commercial impact which attracted other titans of tech, resulting in today’s familiar names: Amazon, Facebook, Netflix. Even legacy villains like Microsoft got their chance at a redemption arc, but the dream of automating human tasks was too tantalizing for the technocrats in these companies. We saw abortive (and hilariously disastrous) attempts like Microsoft Tay, a technical leap beyond the primitive Markov chains that once entertained chatters on IRC. Some tech companies, like Meta, aim to embrace synthetic content, having already piloted AI personas interacting in feeds. The line between organic and synthetic content has never been blurrier.

Compounding this problem are so-called “content farms”, whose existence is well-documented. These “farms” vary from boutique “guerilla content marketing” firms serving upscale clients with LLM-boosted tools, to sweatshop-type warehouses full of people managing fleets of second-hand phones. The impact of these farms is also well-established, and their proliferation has resulted in a veritable flood of rapid-fire articles, blogs, and videos to manipulate content algorithms on platforms like TikTok or Twitter into generating clicks or other game-able engagement metrics. Churning out high-volume, low-quality clickbait to chase an ever-shifting landscape of trends. Some of this is done under the auspices of advertising, but other content farms serve the interests of nation-states in pursuit of geopolitical goals. Whether delegitimizing Taiwanese independence, spreading false rumours of the surrender of Ukrainian forces, or eroding public trust in Western countries’ elections and other institutions, these groups leverage synthetic content and fake profiles. Foreign influence operations tend to wax and wane, and some studies have shown that fake profiles make up about 12% of conversations on social media as a baseline, with that number rising above 50% on the eve of geopolitically-important events such as state visits, elections, etc.

It is critical to remain cognizant of the fact that these phenomena are not the plots of moustache-twirling storybook villains but the natural outgrowth of political and financial incentives. For the unscrupulous retailer selling imitation products on Amazon, there exists essentially no risk in paying for five-star reviews, and substantial potential upside to doing so. In the event the vendor is caught and its account is suspended, the same vendor can be back online in a matter of days with a new name along the lines of ZQMFBG, selling the same white-labeled or counterfeit products. Similarly, the potential geopolitical upsides for state actors are stratospheric, and the risks essentially zero. No modern-era state has yet declared war in response to another state’s online disinformation campaigns, and actors of states possessing a nuclear umbrella have little to fear in this regard. Simply put there is nothing to lose (and everything to gain) by misbehaving.

And what better place to deploy one’s forces than in a “target-rich environment” where a given piece of synthetic content might have broadest reach? In the Anglosphere, one could conceivably reach the bulk of Internet users by targeting just a small number of platforms. Indeed, a list of the twenty most-visited websites on the planet, irrespective of language, includes household names like Facebook, Instagram, TikTok, Youtube, Twitter, and Reddit. This consolidation is both unprecedented and accelerating.

Some might say that these concerns are overstated. That the proportion of synthetic content remains small, and that the great bulk of the internet’s traffic is still undeniably human. This is likely true in quantity, but not in consequence because influence is not measured in averages. It is measured in leverage: a small faction of well-connected actors could bend the entire current of public discourse.

Others may object that fakery is not new, that propaganda and what we call “astroturfing” today have accompanied every age of communication. This too is granted. Yet the scale and automation available to even a lone hobbyist with a laptop can multiply the mischief by orders of magnitude. What once required an army of agents provocateur may now be accomplished by a single operator.

Still others will say the “free market of ideas” will correct itself. That consumers who tire of fake reviews or vote manipulation will turn elsewhere, and in so doing induce the platforms to better themselves. But confidence once lost is not so easily restored. As the saying goes, distrust seeps quickly and cures slowly; irreparable harm may be done long before any correction arrives.

Finally, some caution that in seeking to secure human spaces we will destroy privacy, anonymity, and the freedom that made the early internet vital. This danger is real, but it is a question of design, not destiny. There is no need to trust a corporation to safeguard one’s retinal scans. It is possible to devise means that confirm our humanity without surrendering our liberty. To do less is to resign ourselves to a world where human presence is always in doubt, and the public square becomes a masquerade of shadows.