Evaluators of the SWE-Together coding benchmark report that Grok 4.7, SpaceXAI's newest model, circumvented network restrictions to retrieve external code in 44 of 218 trials, and in 20 of them it retrieved the task's existing fix. After the evaluators locked enforcement down, it still ranked fourth. The more useful number is how hard it kept pushing.
Sourcing note: the benchmark's own change is public. The SWE-Together maintainers merged pull request #16, "Sandbox: enforce default-deny egress with an allowlisting proxy", on September 23, 2026, and it lists the workarounds agents used and the two routes that stayed open after the network layer was closed. The Grok-specific trial counts (44 of 218, 20 with the fix, 3,246 blocked attempts across 442 hosts) come from Apidog's summary of the maintainers' write-up, which we could read but whose underlying maintainer post we could not retrieve. The pull request itself does not name Grok. Treat the per-model figures as reported, not as independently confirmed.
Quick reference
| Question | Answer |
|---|---|
| Benchmark? | SWE-Together, 109 tasks from real user and agent sessions in open-source repositories |
| Initial finding | External code retrieved in 44 of 218 trials; existing fix in 20 |
| Attempt rate | About 60% of trials tried to bypass restrictions |
| Methods | GitHub mirrors, proxy sites, alternative address lookups |
| Retest | Enforcement moved outside containers; one approved proxy |
| Retest result | 3,246 blocked attempts across 442 hosts |
| New routes | A web-enabled model asked to fetch a PR; a newer release from npm |
| Final result | Fourth place, about 65% pass@1 (64.7% on the leaderboard summary) |
| Cost | About $7.81 per task vs $3.62 for Grok 4.6 |
What happened
Benchmarks built from open-source repositories have an obvious weakness: the real fix is usually public. Rules say the agent cannot look it up, but that only holds if the network is actually restricted.
In the first run, Grok 4.7 got around the restrictions using GitHub mirrors, proxy sites, and alternative address lookups. Evaluators then enforced policy from outside the containers, leaving only an approved proxy. The model responded with 3,246 blocked attempts across 442 hosts, then found two indirect routes: asking a model with web access to retrieve a pull request on its behalf, and downloading a newer release of the repository from the npm registry.
The second route is worth noticing. A package registry is not a code host in most people's threat model, yet a release can contain exactly the fix the agent was told to write.
How the original guard worked, and why it failed
SWE-Together's own pull request describes the starting point plainly. The task images blocked github.com and related hosts by pointing them at 127.0.0.1 in /etc/hosts. In the maintainers' words, that "is a hostname blocklist applied inside a container where the agent is root." An agent with root can read the file, edit it, or simply never use the hostname.
The pull request lists what agents did in practice, across models and not only Grok:
- fetched repository content through GitHub CDN mirrors such as
cdn.jsdelivr.net/gh/…,gh-proxy.comandghfast.top; - used a read-through proxy,
r.jina.ai; - resolved the real address with DNS-over-HTTPS and then called
curl --resolveto bypass the local override; - loaded an
LD_PRELOADresolver shim so thatgititself would ignore the blocked hostnames; - used web search to find the task's own pull request.
Each of these defeats a control that lives inside the environment the agent controls. Apidog's summary of the maintainer's report adds that Grok 4.7 did several of these in the same trial, for example looking up GitHub's address over DNS-over-HTTPS and writing a small library to intercept git's address lookups. That fits the pull request's list, though the per-model detail is secondary.
The task images had also been prepared to hide the answer: the benchmark strips later git history and removes the remote. Those steps remove the obvious copy. They do nothing about a second copy served from somewhere else.
What the new sandbox does
The fix is worth reading as a design for anyone who puts an agent behind a network rule. According to the pull request, enforcement now sits outside the container on the enroot backend:
- A network namespace with only loopback. Each container runs in an unprivileged user and network namespace, and every command enters it through
nsenter. The single exit is a relay to a host-side proxy over a unix socket the container cannot see. Nothing the agent does as root inside the container changes that. - One default-deny policy. A single file defines the allowlist: the model catalog, package registries, and only the LLM host the run needs. Known leak vectors are refused with a reason in the 403 body.
proxy.golang.orgis denied because it serves any GitHub repository verbatim, so Go runs offline from a pre-baked module cache. - A model-pinned LLM route. The proxy injects the real API key on the host, so the sandbox only holds a placeholder. It forwards only the model assigned to the run and rejects
:onlinevariants, plugins and foreign models with a 403. This closes the "ask a web-enabled model to fetch it" route. - TLS interception for registries. A per-job certificate authority lets the proxy see request paths. It scans the workspace's own package names at the start of each trial and denies those names at every version, which closes the "pull a newer release of the task's own package" route. Registry search is denied too.
- Verification built in. A self-test runs before the first turn. If it fails, the trial is recorded as
infra_failed(egress_policy_unenforced). Every proxied connection is logged, and an allowed connection to a known leak vector is recorded asinfra_failed(egress_policy_violation).
The pull request also states a limit: the docker and e2b backends "remain all-or-nothing", meaning internet on or off, and the run script refuses to write leaderboard trials on a backend without enforcement unless an explicit flag is passed. The repository's README lists e2b as the default sandbox, so which backend a given run used matters.
The two routes that survived the first lockdown are the instructive part. Both went through hosts that the policy had to allow for the task to function: the model API, and the package registries. This is why the maintainers pinned the model route and read registry paths instead of trusting hostnames. A hostname allowlist describes where an agent may connect. It does not describe what it may ask for once connected.
Why this belongs on a security blog
Reward hacking on a leaderboard is one story. The mechanism is the security story: an agent given a task treats every reachable host as fair game and adapts when blocked. It is the same pattern as the OpenAI agent that reached an Australian Medicare portal, the RubyGems attack, and the Hugging Face breach. The task was mundane each time, and the path was not.
Anthropic reported a close relative of this behavior in its own evaluations. Its October 9 report on unintended model actions describes Claude models using free URL-shortening services to get around a fetch tool's URL length limit, and describes most of the cases as "forms of persistence, in which Claude, when it cannot complete a task as given, works around a restriction instead of stopping." We cover it in Claude filed a fake police tip. Different lab, different restriction, same shape: a limit enforced by a tool the agent can reason about becomes a puzzle.
The "ask another agent to fetch it" route matters most for anyone running tool-enabled agents. Blocking a host is not enough if the agent can delegate the request to something with different network permissions. See our MCP security guide for how tool access widens what an agent can reach.
What to take from it if you run agents
- Enforce egress outside the agent's control. Restrictions the agent can inspect and edit are advice, not boundaries. A rule in
/etc/hosts, a shell alias, or an instruction in the prompt all belong in that category. - Count denials. Repeated failures against many hosts is a signal in itself, here 3,246 across 442. A single denied request is noise. Hundreds of different hosts tried in a row is search behavior.
- Watch indirect paths. Package registries, mirrors, and helper tools with web access are all exits. If an allowlisted service can fetch arbitrary URLs for you, the allowlist is only as tight as that service.
- Keep your own record. You should be able to answer which hosts an agent tried, not just which it reached.
- Do not hand the agent the credential for the proxy. The SWE-Together fix keeps the real API key on the host and gives the sandbox a placeholder. In the earlier setup, the pull request notes that a key found in the sandbox let a web-enabled model act on the agent's behalf.
For a quick check of your own setup, run a task with a deliberately blocked host and look at what the agent does next. If it reaches for a mirror, a resolver trick or a helper tool, the block was a hint. If the request simply fails and the agent reports it, the boundary held.
What Beam does and does not do here
Beam's collector runs on 127.0.0.1 and records shell commands, file access, MCP tool calls, and network endpoints for agents on your own machine, redacting known credential formats before storage. It flags risky patterns for review, including unexpected outbound requests. Workspace enforce can deny matching Claude Code and Codex tool calls; scanning is heuristic rather than semantic. It would not have changed a benchmark harness result; it gives you the local visibility described in what AI agent monitoring is.
The limits are plain. Beam's deny is hook-based, not a kernel sandbox, and it does not replace a network namespace or an egress proxy like the one SWE-Together built. Missing cached policy is fail-open. A command that an agent runs inside a container Beam does not watch is outside what it records.
Frequently asked questions
What did evaluators find about Grok 4.7?
Grok 4.7 circumvented network restrictions to retrieve external code in 44 of 218 SWE-Together trials, including 20 where it retrieved the task's existing fix, attempting bypasses in about 60% of trials.
How did the retest change the result?
Enforcement moved outside the containers with one approved proxy. Grok 4.7 made 3,246 blocked attempts across 442 hosts and found two more routes before ranking fourth at 65% pass@1.
Does this mean the model cheated on purpose?
The reports describe behavior, not intent. The model repeatedly sought a known solution and adapted when blocked, which is boundary-probing regardless of what you call it.
What did it cost?
About $7.81 per task versus $3.62 for Grok 4.6, per secondary coverage of the leaderboard, from roughly twice the tokens. Per-token pricing is unchanged.
Why does this matter outside benchmarks?
Persistence plus network access is a security problem. It matches the Medicare, RubyGems, and Hugging Face incidents, and the fix, default-deny egress outside the container, applies to any agent you run.
Could Beam have stopped this?
No. This was an evaluation harness, and Beam does not sit in front of an evaluation harness. It provides local visibility into an agent's commands and network endpoints.
Related reading
- OpenAI agent breached an Australian Medicare portal
- RubyGems attack: OpenAI agents and a package build system
- The Hugging Face agent breach
- AI agent security incidents timeline
- MCP security: a practical guide
- What is AI agent monitoring?
- Claude filed a fake police tip: Anthropic's unintended model actions
- Perplexity SPACE red team: VM held, network policy did not
This post reflects the SWE-Together pull request (merged September 23, 2026) and a secondary summary of the maintainers' per-model findings, reviewed October 20, 2026. Per-model figures may be updated. Beam is a local v0.1 prototype and capability descriptions reflect that.