The opt-out that isn't

A pseudonymous researcher pointed a packet sniffer at xAI's Grok Build CLI last weekend and found that the tool uploads whole repositories, including the contents of .env files, to a Google Cloud Storage bucket controlled by xAI. The setting labelled "Improve the model," which users toggle off when they want their code kept out of xAI's hands, does not stop the uploads. This is what happens when the consent architecture fails at the only point where users can touch it.¹

The findings come from @cereblab, who ran Grok Build CLI v0.2.93 through mitmproxy (a tool that intercepts and logs network traffic) and published the wire capture as a GitHub gist. Before they stopped the capture, 5.10 GiB of data had already been transmitted from a 12 GB test repository. The gist reached the front page of Hacker News with 508 points inside a day.¹

What the wire showed. Two things, both material. First, POST requests to xAI's /v1/responses endpoint carried the unredacted contents of .env files — the plain-text files developers use to store API keys, database passwords, and credentials for third-party services. Second, whole-repository git bundles were uploaded to a Google Cloud Storage bucket named grok-code-session-traces, including files the agent was never asked to read and git history the user never surfaced. When the researcher toggled the "Improve the model" opt-out and captured the traffic again, the request payload still carried trace_upload_enabled: true.¹

xAI has not, at the time of writing, published a response in the primary sources.²³

The distinction xAI would draw, and why it does not save the argument. A defender of the tool would point out, correctly, that grok-code-session-traces sounds like telemetry and crash-logging infrastructure, not a training pipeline. The researcher is careful about this in the gist: they documented upload, not training use. And multiple write-ups since have noted that the "Improve the model" toggle in Grok's settings governs training consent specifically, while the trace-upload path is a separate system that the toggle does not touch.²

I want to take that framing seriously, because the engineering distinction is real. Session telemetry is a normal part of building developer tools; without it, crash reports and latency debugging get much harder. But the distinction is not one the user was shown. There is one toggle. It is labelled in the language of consent. It sits in the place where users go to control what happens to their code. If it governs only one of the two upload paths, and the other path is not disclosed, then what the interface calls consent is not consent. It is a control that operates on a subset of the behaviour the user is trying to control, without saying so.

Who is actually exposed. This is the part that gets underplayed when the story is framed as an AI-tool privacy story. A .env file is not the developer's intellectual property. It is a bundle of credentials for other people's systems — the production database, the payment processor, the transactional email service, the third-party analytics API, the internal admin panel. When a CLI uploads .env contents unredacted, the population exposed is not the developer who installed the tool. It is every downstream system whose keys were in that file, and every end user whose data sits behind those systems.

The same asymmetry applies to the whole-repo upload. A software developer working under an NDA (non-disclosure agreement), or handling code subject to export controls, or building against a healthcare data set governed by HIPAA (Health Insurance Portability and Accountability Act), or operating under a SOC 2 audit programme, has now transmitted material to a third-party bucket they were not authorised to transmit to. Under GDPR (General Data Protection Regulation) and most NDA language, unauthorised transfer is the compliance event. Whether xAI trains on the data, deletes it, or forgets about it entirely does not change what already happened on the wire.

The affected population is not just the developer who installed the CLI — it is every system whose credentials were in that repo.

How the audit happened. It is worth naming who did this work. Not a regulator. Not a certification body. Not xAI's internal security team publishing a transparency report. A pseudonymous developer ran mitmproxy against a shipped product, wrote up what they found, and posted it to a gist. That is the accountability infrastructure available for AI coding tools right now: hobbyist packet capture and a Hacker News front page. Same week, unrelated: a separate cohort of practitioners was running by-hand token-overhead studies on Claude Code because vendor documentation of what these tools actually do at the wire is thin across the category.

I do not think the pattern is coincidence. When a class of products ships fast, into a market where buyers assume vendor claims are true, and where the audit institutions are still catching up, the audit gets done from below or it does not get done at all. This is what audit culture from below looks like — and it is a fragile substitute for the thing it is standing in for.

What the sensible developer response is. In the short term: assume any AI coding CLI is transmitting more than its settings page suggests, until you have watched the traffic yourself. Rotate any credential that has been in a repo an AI CLI has touched. Do not run these tools against repositories governed by NDA, HIPAA, export control, or client confidentiality agreements without wire-level verification of what leaves the machine. This is not a satisfying prescription. It offloads onto individual developers the verification work that vendors should be doing and disclosing.

In the longer term, the question is whether "we shipped a toggle" continues to be treated as a defence when the toggle demonstrably does not do the thing its label implies. That is not a technical question. It is a question about what counts as consent in software, and about who bears the cost when the answer is "less than the interface suggested." Right now the cost is being borne by developers, by the downstream systems whose keys sat in those .env files, and by the end users behind them — none of whom saw the settings page, and none of whom got to toggle anything.

Footnotes