Having a passport in your pocket can at times feel almost magical. Borders fade away, immigration queues disappear, and suddenly the world feels more accessible. For those of us lucky enough to have widely accepted travel documents, it can mean bypassing border control and visa requirements that limit travel for most of humanity. Vaccine passports have played a part of that immigration process for decades, but now the term is being unveiled for a very different type of system, one of mass surveillance and segregation.

Historically, vaccine passports took the form of the emblematic “yellow card.” Technically titled an International Certificate of Vaccination or Prophylaxis, the decidedly low-tech document simply lists and certifies an individual’s vaccine history. While the current form hasn’t been updated since 2005, the basic framework hasn’t seen much change since the scheme was adopted by the International Sanitary Convention in 1944.

The goal of yellow cards has always been twofold: protect public health while harmonizing the global standard for reporting inoculation, making it easier for countries to certify and confirm travelers’ health history. As larger and larger swaths of the population are inoculated against Covid-19, it makes complete sense that yellow cards would play a part in our response, helping countries with low Covid-19 rates prevent the introduction of additional cases or novel strains.

The World Health Organization has proposed a digital yellow card, a secure cryptographic vaccine record that could be shown at the border. This standard has numerous technical questions, but it’s largely unassailable. After all, we have global standards for encoding electronic passport information, using RFID and other technologies, so why not do the same for vaccine data?

The controversy comes from a very different type of vaccine passport, not one being rolled out at the border, but envisioned for neighborhoods across the US. While this type of local tracking system is often called by the same name, its implications are quite different.

For New York mayoral candidate Andrew Yang, vaccine passports shouldn’t be confined to travel, they should be the key to accessing your “office building or restaurant or whatever.” The presidential candidate turned TV commentator is not alone. A growing number of startups and tech giants are hoping to develop a daily vaccine tracking system.

When the Vaccination Credential Initiative launched last month, the partnership between tech giants and insurers (including Microsoft, Oracle, and Cigna) promised their technology would enable the return to life as usual. For one participating organization, the Commons Project, this would mean a new digital key to access modern life, “whether it’s getting on an airplane and going to a different country, whether it’s going to work, to school, to the grocery store, to live concerts or sporting events.”

A very appealing vision for some could turn into a dystopian nightmare for millions. Such restrictive passports could mean being locked out of your job, education, or even the ability to shop for food. At a moment when vaccine distribution is highlighting inequalities both locally and internationally, when communities of color and lower-income communities are being systematically underserved, vaccine passports would amplify our medical segregation.

Suddenly, Black and Latinx Americans might not just see themselves being excluded from vaccine queues, but from nearly every aspect of public life. And at a time when one in five Americans lack a smartphone, and many others use devices too old to support such an app, we would likely see millions of vaccinated Americans wrongfully blocked from public spaces. To make it even worse, none of this effort is coming from public health authorities.

As recent voter suppression laws prove, imposing even a seemingly minor obstacle can effectively shut out millions from exercising their basic rights. Requiring a drivers’ license to vote feels like a trivial step to many, but it is truly insurmountable for millions of potential voters. Vaccine passports may not be intentionally discriminatory in the way voter ID laws are, but the impact can be just as stark. Requiring a smartphone app to work or get an education is the same as excluding the millions who lack a smartphone to begin with.