We gave Claude a refinery for three days

1Introduction

We’re Skyello. We automate routine work at industrial sites, starting with canister programs at refineries. Before we put a model on a real program, we wanted to know what it does with a simulated one, especially with the problems that don’t set off any alarms.

2How a canister program works

Refinery wastewater gives off benzene[5]. Vents carry it through pairs of drums filled with activated carbon, which absorbs it. A technician measures the gas between the two drums every few days, and when the reading passes the limit, the first drum has to be replaced within 8 hours. Missing that deadline violates the site’s consent decree.

In the simulation, the model runs the program from a desk. It has the canister app, Teams, email, work orders, the supplier’s booking portal and the site’s documents, but it can’t go out and look at a drum. If it wants something physically checked, it has to send a person.

The simulation tracks what physically happens at the site separately from what people write down. The model only sees what people write down. We grade based on what physically happened.

After the model’s three days, a coordinator takes over and runs the program for another week following only the site’s standard procedure and whatever the model set up. Runs are scored in dollars using rates adapted from a real consent decree (HF Sinclair, 2025)[1]. A vent bypassing its carbon costs $1,000 a day. A late drum replacement costs $100 an hour. A carbon fire costs $250,000 and automatically fails the run. To pass, a run has to score within $2,000 of the best expert plan we could write.

3What’s waiting on Monday

We ran six versions of the same week, with different names, units and dates. Each version has four problems, each drawn from public enforcement records and industry safety guidance. Nothing in the system flags two of them. The other two show up as an alert, an email or a Teams post.

4The vent nobody reconnected

If you look at this drum’s readings, everything seems fine. They had been rising toward the limit, from 9 to 12 to 16 ppm, and then they dropped to about 1 ppm and stayed there for eighteen days. That’s what a fresh drum reads, but it’s also what a drum reads when no gas is flowing through it at all. This drum hadn’t been changed. It was 60 days old, and drums at this site last about six to ten weeks.

The only clue was a Teams message from 18 days earlier:

1. Operations shift lead18 days before Monday, 08:40

   Vac truck is on [the tank] this morning for the cleanout. We’re taking its carbon drums out of the lineup while they pull.

Nobody ever posted that the drums were put back.

4.1Monday morning

Opus 5.5 read through the site documents and the Teams history. Six minutes into its shift, it sent this message to the operations lead:

1. Opus 5.5 to the operations leadMon 06:06

   Morning Jamal - Skyello (covering BWON carbon for Victor through Wed). Two status questions: (1) Slop oil tank TK-571 vent - your Jun 18 post said its carbon drums came out of the lineup for the vac-truck cleanout. Are the drums (DR-6254/DR-6621) back on flow, and since when? …

2. Operations shift leadMon 06:14

   Good question. I don’t think anyone put Slop oil tank TK-571 vent’s drums back in the lineup after the vac job. Sending the outside operator to check now.

3. The outside operatorMon 06:56

   Slop oil tank TK-571 vent carbon drums: found the drums’ inlet block valve shut and the bypass to the vent stack open since Thu Jun 18 9:40 AM (left from the vac-truck isolation); lined the drums back up.

4. Opus 5.5 to the outside operatorMon 06:59

   Thanks Tom. Those drums sat loaded with no flow for 18 days, which is the classic setup for a carbon bed heating up. Please check skin temp on both TK-571 drums now (WO raised). If either is warm or discolored, take it back off line and tell me.

Within an hour, a vent that had been releasing benzene for 18 days was back on carbon. Opus 5.5 filed the release as a deviation and told the Environmental Manager, who may need to report it.

This happened in 16 of its 18 runs. It always found it on Monday, and the drums were usually reconnected within about an hour of the model’s shift starting.

Later that morning, Operations offered to keep the solvent unit’s vent off its carbon when the unit restarts on Tuesday. That’s the unit with the idle drum. Opus 5.5 connected the two problems:

1. Operations shift leadMon 07:34

   OK - we’ll keep the sump vent off the carbon lineup at restart and hold it until you tell us the drum’s ready, latest Wed 15:00. Let me know.

2. Opus 5.5 to the operations leadMon 07:43

   Jamal, one correction please: the SDU sump vent must not restart venting to atmosphere around the carbon. That would be uncontrolled venting, which is exactly what we just found on TK-571. …

It made the same connection, unprompted, in two other weeks.

4.2The other two models

Sonnet 5.5 and Opus 4.6 never found the bypass, in any of their 8 runs combined. They both looked at this drum because it had missed its weekend reading, but they just filed the missed-reading paperwork and put it on the schedule for the next round. At the end of its shift, Opus 4.6 wrote this in its handover to the supervisor:

1. Opus 4.6 handoverWed 12:12

   DR-1806 at S-22 #517: 0.9 ppm, healthy

That drum’s vent kept releasing to the air for the rest of the test.

Opus 5.5 missed it twice, too. In those two runs, it looked at the drum early on and moved on, exactly the way the other models did every time. We went back through the records to see if something was different in those two weeks and couldn’t find anything.

5The meter that reads low

The humid tank is blanketed with nitrogen, so the vent gas has almost no oxygen in it. That rules out the site’s main meter, which needs oxygen to work. The consent decree allows a backup meter instead[1], and that’s what technicians use. But the vent gas is 97% humid, and this backup meter is only rated to 95% humidity[9]. Above that, it shows only about 40% of what’s really there.

The drum actually went over its limit at 02:00 Monday, four hours before the model’s shift started. The backup meter wouldn’t show it crossing the limit until Wednesday morning. A benzene tube, which is a third type of meter that isn’t affected by humidity, would show it immediately.

Opus 5.5 found this by comparing the vent’s humidity spec in the design documents with the meter’s humidity rating in the instrument notes. It ordered a benzene tube reading on Monday morning in all 18 of its runs, and switched the vent to benzene tubes in the app so whoever came after it would keep using them.

Sonnet 5.5 caught it the same way in 2 of its 6 runs. Opus 4.6 looked at the same vent and approved the backup meter:

“DR-1088 (TK-785) correctly uses PID due to N2 blanket - FID can’t sustain flame there.”

That reasoning is correct. The decree does allow it. But the drum was already over its limit on Monday, and Opus 4.6 didn’t find out until Wednesday when the backup meter finally caught up.

6The fire that didn’t happen

Every model handled the problems that set off alarms. Operations posted the restart time on Monday morning, and in every run, every model sent someone to check the idle drum’s temperature before gas started flowing. In the hot weeks, where the drum was between 141 and 155°F, the models had it purged with nitrogen, soaked with water, or swapped out in time[10][13]. There were no fires, no late drum replacements, and no missed deadlines across all 26 runs.

For Opus 5.5, the bypassed vent is the whole difference between passing and failing. Missing it costs about $9,500 in the simulation, and the pass bar only allows $2,000 of room above the expert plan. Catching the humid drum on Wednesday instead of Monday adds about $5,000 more. Everything else moves a score far less. An extra drum swap, for example, costs $350.

7Opus 4.6 filed two deviations by mistake

In one run, Opus 4.6 noticed that a drum had no due date in the canister app. The actual reason was simple: the unit it serves was shut down, so no reading was due. But Opus 4.6 came up with a different explanation:

“R-5003 (DR-7708 at SDU sump #511): logged with wrong location text "Benzene stripper feed tank TK-1339 Pri #511" - that’s why the schedule shows "due -". Needs correction.”

It filed two deviations for readings that were never actually required. An hour later, Operations posted that the unit was shut down, and Opus 4.6 acknowledged that readings are only needed when gas is flowing. But it never went back and withdrew the two filings.

8Limitations

• This is one scenario run in six versions. Opus 5.5 ran each version three times, Sonnet 5.5 ran each version once, and Opus 4.6 ran two of the six versions once each.
• The people in the simulation respond through work orders and a small set of scripted replies. The bypassed vent is usually found when the model asks Operations about it, which triggers one of those replies. At a real site, there would be more ways to notice a closed valve.
• There’s no human baseline. The pass bar comes from a simulated expert plan, not a real person running the same week.
• All three models ran at default settings, so these aren’t necessarily their best possible results.
• The penalty rates are adapted from a real consent decree, but they aren’t that decree’s actual charges for these specific events.
• We can’t see the models’ internal reasoning. Every quote in this post is from a message or a note the model wrote, not from its chain of thought.

9What this means for our work

A flow meter on each pair of drums would have caught the bypass on day one with any model. HF Sinclair’s consent decree already requires one[1].

Opus 5.5 found it a different way. It noticed a message that never came and asked someone to go check. That worked 16 out of 18 times, which is impressive but not good enough on its own for a vent releasing benzene. So at Skyello, we connect the model to the site’s sensors as well as its records.

References

1. U.S. and NMED v. HF Sinclair Navajo Refining LLC, Consent Decree, lodged January 17, 2025.
2. U.S. v. Equistar Chemicals, LP, Consent Decree ¶27.
3. U.S. v. BP Products North America Inc. (Whiting), Consent Decree, 2023, ¶41.
4. U.S. v. Deer Park Refining LP, Consent Decree ¶40–47.
5. 40 CFR Part 61, Subpart FF (Benzene Waste Operations NESHAP).
6. EPA BWON enforcement alert, February 2024.
7. TCEQ agreed order, Flint Hills Resources Corpus Christi, Docket No. 2022-1543-AIR-E.
8. EPA inspection report, Marathon Petroleum, Garyville, Louisiana, 2022.
9. Honeywell RAE Systems: TN-163, TN-127, RAE-Sep benzene tube data sheet, MiniRAE 3000+ data sheet.
10. FM Global Data Sheet 7-2, Waste Solvent Recovery, §2.4.4.5.1.
11. M. Hoyle, G.R. Astbury and M.H. Chen, “Thermal stability of activated carbon in an adsorber bed,” IChemE Symposium Series No. 141 (Hazards XIII), 1997.
12. OSHA Hazard Information Bulletin, “Fire Hazard From Carbon Adsorption Deodorizing Systems,” July 30, 1997.
13. Calgon Carbon Ventsorb product bulletin.