I drown in support tickets daily that used to go nowhere. Then people started realizing that I knew what I was talking about and not some standard business user. Now I still have to open tickets, but get visibility of the people who know what they’re doing on their end (having been moved up from lowly support to oversee some ticket filling company that doesn’t know crap). We get tom complain to one another how shit is broken, wait mandatory change periods and then get something working by exploiting loopholes because some process was developed and someone put a lot of money behind it.
Large companies are why large companies are large (in IT/OT at least)
I don’t think so, or maybe it was… IDK anymore. Now I’ll have to go rewatch it.
It definitely is explained as described above. The gate itself, the “event horizon” is all part of the apparatus for accessing the wormhole, not the wormhole itself. I forget the episode, but there is one where something happens due to the fact that they recreated/reverse engineered the “brains” of the device with human design (missing some key failsafes) and some of them get stuck in the “buffer” (dematerialized state) until it’s figured out how to reconstitute them.
Yeah, that's right and I believe that was the episode that introduced McKay. I've been binging Stargate over the last month or so, so it's pretty fresh.
Even more so, if it’s just there you’re making it easy for the customer to find someone bidding less to execute new projects. Guess who they’re calling if they don’t have the plc project (or don’t understand they need it). Job security.
Plus, yea, someone could really make a mess even if they have the best intention. A guy from my company almost uploaded the wrong program to a large bioreactor. At best it would have made the bioreactor just unusable until corrected. At worst it could have broken the equipment or caused a safety issue. This guy had been around the block a few times too. Luckily someone noticed before it was too late.
I don't get into PLCs, per se, but I've done some stuff with other building-scale gear.
Like: We took care of the centralized controls for the inmate portion of a jail that was built in the 1980s. We rejiggered the controls for that whole jail back around 2010 or so with new kit and had been taking care of it since then.
All the wiring for all of the individual doors and intercoms was documented in one old 3-ring binder. (There were probably 3 copies originally -- for the architect, the maintenance room, and the equipment room, but just 1 remained and it had hand-written notes.)
One day, around 2020-ish, they called us because the book was missing. I didn't have it (I never once left the equipment room with it), and the co-workers who had been there insisted that they didn't have it either.
As time passed, we'd get occasional phone calls of "Hey, we really could use the book for our building back if you guys ever find it." They weren't angry calls, but sheesh: That book had real value and getting stuff done without it was difficult. So every time when they'd call again, I'd ask around our shop -- again -- if it had turned up anywhere.
And then, finally: One of our guys cleaned out his truck and found it. He must have taken the book out to look at and have a smoke about some problem or other, and then it just stayed in his truck. For years.
This shit happens. People are imperfect and sometimes we screw stuff up.
Sounds like we have a similar development path. People fail to understand how complex some of these systems can be, especially since you have to have the project to understand what registers/variables are what. Some when Joe the contractor does a job and walks away, you better make sure they give you the project and you store it in a nice safe place.
A good system integrator is worth their weight in gold. Sure they cost more, but getting a whole package turned over to you is worth a million more than 5 years into the lifecycle of a factory when someone wants to make mods/fix a bug/etc and has to reinvent the whole car, not just the wheel.
This industry is always 10-20 years in the past. My company’s preferred vendor only just started supporting virtualization (this is for a DCS) in the past 10 years. I still have to tell my sales people to provide A/V and minimalistic backup and recovery on every project (they essentially cost nothing compared to the rest of any project).
I enjoy watching Cursed Controls on YouTube. For him, the gold standard is a flash drive with the program on it physically hanging out in the control cabinet.
You’d be surprised how insecure some of these facilities are, especially to someone who has working knowledge of what a PLC (or other process controllers) does and how it works. You can easily look like a tech who belongs there either troubleshooting something or working on a project.
I’ve been doing industrial controls for 15 years and surprisingly infrastructure is some of the most poorly funded. I believe a lot of these places are run by operating companies, so it’s bidded out (we all know how bids work I think). I’m not surprised when I walk into these places and see the computers are running EOL operating systems and the networking is essentially flat.
Okay but then is the RF connection really your biggest concern?
I'm kind of worried (probably unnecessarily) that posting ideas would get me on some list, but it seems like there would be many simpler terrorism opportunities once you have physical access.
WWII provides some interesting data points. On the one hand, yes, as Germany occupied numerous neighbours people - even sometimes in the face of group punishment, kept sabotaging the German military and its logistics. In some cases there are literally enemy agents, the Special Operations Executive† but often they're local partisans either working with the SOE or on their own.
But on the other hand, even though the Americans suspected that people who merely looked Japanese might be traitors, AFAIK there aren't any clear examples where the people who were sent to camps actually were enemy spies who'd have sabotaged America given the chance. And in Britain the counter-intelligence operation was so successful that when captured German spymasters revealed their list of agents in Britain, every name was already either working for Twenty Department (20 = XX = Double Cross, we can't resist a pun) or in prison for espionage or dead.
† notably in WWII if as a woman you say you want to be on a ship of the line, or crew front line aircraft and attack the Nazis you will be told women can't serve front line roles and at most you'll be doing delivery runs in relative safety. But if you know the right people to become a spy they will cheerfully send you behind enemy lines even though if caught you will almost certainly be horribly tortured and then probably killed and the government which sent you won't even acknowledge you existed for years.
So, maybe the risk from people who are already where your infrastructure is is much lower that you'd think if you haven't invaded them and occupied their land. On the other hand remote adversaries are definitely always a risk.
Forgive my ignorance but isn't a PLC simply a computer with some GPIO ports? I.e., a Raspberry Pi could be called a PLC? Why are we talking about them as if they are something else? Is it an exotic OS that makes them different?
All the siblings might talk a lot about "but ruggedized!", "but realtime!", "but supported!".
But in many cases, it is just a case of vendor lock-in. Ruggedized is just a casing and decoupled I/O-ports, you can get that for your Raspberry Pi. Realtime is a concern, but modern PLCs aren't really realtime anymore. Those are usually some big multicore non-realtime ARM, plus some small realtime-capable I/O-processors that do the really time-critical stuff. Or even just a CPLD to implement some trivial ladder logic cycles. Support matters, but if your PLC vendor goes bankrupt you've drained a few millions. Might be better to get something cheap where you can just stock up a few hundred replacement parts for cheaper.
That's why actually, a lot of smaller PLCs are Raspberry Pi CMs now. The vendor won't tell you, but if you open them up, you'll see.
And the big customers also wised up to the PLC scam and are now moving towards putting their control tasks on standard x86 hardware in the shop floor basement plus realtime ethernet with cheap I/O extenders on the shop floor. Which is why the big PLC vendors now try to sell "virtual PLC" to those customers, so they get at least a little money instead of none ;) https://www.siemens.com/en-us/products/simatic/s7-1500v/
The term "PLC" is a lot more specific than "computer with some GPIO ports." No one that works with PLCs would consider a Raspberry Pi to be one.
These things are walled gardens. You never see the operating system. You can only change their behavior using the vendor's software. They generally run a single program (that you write using the vendor's software) on a fixed scan cycle. They read the inputs, run your program, write the outputs, then repeat.
While you're giving up the nearly infinite possibilities that an SBC gives you, the benefits more than make up for the lack of flexibility. They run (and have parts and support available) for decades. Modules are easy to diagnose and replace. An electrician who isn't a programmer can follow ladder logic and troubleshoot problems. Integrators can quickly come up to speed and understand your code.
There's a reason companies will pay tens or even hundreds of thousands of dollars for these things.
The point I was making is that PLCs have a software stack. You have to know the software and programming languages defined in IEC 61131-3. For all intents and purposes they are just very repeatable compute in an industrial setting, but if you don’t know ladder logic or how to interface with them you might be SoL. Most don’t have any sort of web interface and if they do it operates outside the controllable logic. Same goes for shells. If you want to program it you typically have to use the vendor software.
PLCs are ruggedised and built to survive for years in the kinds of hostile environment that are frequently found in industrial settings.
And PLC vendors typically provide guaranteed support periods that are much longer (like a decade or more) than those commonly provided by conventional computer OEMs.
The hardware my company uses has a lifecycle of 15-20 (aka what we typically look at when we consider lifecycle of a plant). This is not meant to be the whole life of a plant, but possibly the life of the current controls. Things/people/companies change. I’ve seen some really shortsighted people rip and replace a sub-decade control system because of “bad experiences”. The industry can be great where it’s great and terrible where it’s terrible.
> If a Raspberry Pi had industrial environment ratings and was certified hard real time, then yes it could be considered a PLC.
Ostensibly yes, but so far I haven't seen anyone really use a PLC in a way that requires hard real time (so far). The cycle on eg a siemens S7-1200 is anyway much too slow for anything really exciting, and a Pi might very well be more reliable in actual practice, were it not for the very unfortunate tendency to eat SD cards. :-P
(And revolution pi actually ships a hardened Pi for industrial use. So that's one way to go about it. I'm not a big fan of that brand, but it's a data-point. Meanwhile in personal experience some regular pi's left in industrial cabinets for one-off emergency monitoring purposes have managed to stay annoyingly alive over time.)
Not really, sure you can use a rpi to control some hw but they are not the same, a PLC usually run rtos, is deterministic (you can predict timing) while rpi relies on linux OS and its scheduler, the PLC also uses ladder language or function block compared to rpi high level language, and obviously PLC industrial grade I/O both analog and digital that also deal with voltage noise that usually happens by field sensors, and environmentally rugged and rated to run non stop compared to rpi.
Now, if you really want to use rpi as a plc, you need something like openplc or codesys as a runtime, add some HATs for I/O, and use protocols like modbus. It will be a software plc but you are missing the hardware certification and other features. Rpi is good as edge computing rather than plc, like processing vision or data logging, it’s why in drones you need the autopilot AND rpi or companion computer, each does certain functions.
I used to think codesys on raspberry pi was pretty much the pessimal application of an ARM processor (and nothing has changed my mind so far). Here you have a chip that is famous for staying stone-cold and doing interrupts in microseconds , and instead you run it hot on a 50Hz (default) PLC loop instead.
I think it is. I work for a (relatively) small company. I naturally grew from project engineer to senior/lead/sme (including pioneering tech _for my industry_) to SA. I had also stuck with my company for many years, so I have the industry connections and got to be known as a heavy hitter. That trust relationship with the customers mixed with technical know how = sales and consulting.
Again, because of the size of my company I can make my role fluid (including a good way), but call it what you will I engineer, I sell, I consult.
My kids a bit young for reading HP on their own, but it’s our bedtime book series every night. Over the past year or so we’ve (I’ve) read HP before bed. We just started book the Order of the Phoenix.
Our school system has a reading log system (mandatory 15min/night), but I don’t think it’s very effective.
I don’t know that I’d put my kid down as an avid reader, but for his age range I can set him on a task to read and he’ll find something he enjoys enough to be immersed (albeit usually graphic novels akin to Minecraft, Godzilla and other mangas). But at 8 I’d rather that than an aversion to reading.
Anyway, I’m curious about the statistics around parents reading by example (solo), parents reading to children (bedtime or otherwise and until what age). I have memories of my mom reading with me when I was about his age.
I can say this, me and every other parent in my school district is sick of screen time.
They could just switch their dns back to auto (or statically use google/cloudflare/etc depending on how you configure it), no? Then fix it when you’re back.
You could also set up 2 ssids depending on your WiFi set up. Point one to pi hole and the other to a different DNS provider. Instruction if pi hole breaks is just switch WiFi.
As an OT systems architect I am totally floored. We design and plan for systems lifecycle on a ~20yr scale, with OT hardware (not the controls hardware, that’s closer to 10-20) lifecycle much shorter (~5 yr). Obvious on Earth we can afford luxuries of adopting new things, which actually shortens a total system lifecycle since new tech drives new designs.
I wish (and don’t) I could work on something that had a dependency of “design it once because it’s relatively inaccessible after its go live.” I’ll def check out the documentary.
Video games used to be like this. Once you built the "gold master" CD/DVD/cartridge/etc it was out of your hands. It was kinda nice to have a concrete end to the project [1]. Nowadays, everything is on the 'net, you can send patches, dlc, etc and the notion of a game being "done" is murky.
[1] There was, however, one game I worked on where they had to pull the boxes from stores (delivered, but not yet for sale) and swap out the disk in order to release a critical fix that was discovered too late. Fun times (:
Which resulted in the notorious release of Outpost [1]. I think owners of that would have happily accepted a large series of post release patches over that.
Especially when it was cartridge games. I remember when PC games started to get updates and you'd wait for next month's cover disc to get them. I seem to remember Frontier Elite having about a dozen...
I just checked the one commercial game I developed and there are two patches I can see released by Eidos for it.
I'm curious, what was the bug that was so critical the publisher decided it was best to perform such a (what I assume was) costly operation post-distribution?
I'm aware of one game that the company I worked for made that nearly released and that would have broken every GameCube that played it, Nintendo had to pull 50k discs from distribution just before they were sent to retailers and destroy them.
The issue was that one programmer used an unauthorized system call to make the disc drive spin twice as fast, as they thought it was a great way to resolve some of the data streaming issues the game had. And yeah it worked - but after few hours of playing it would kill the GameCube. It wasn't really noticed because no one tests the game on actual discs right until actual gold master is made(usually), and then when the devkits died it was considered a random hardware fault and Nintendo just replaced them.
Honestly it was just before my time at that studio so I don't know exactly how that was done, but everyone knew about it because it cost us a lot of money and damaged our relationship with Nintendo somewhat. The game actually went on to be pretty successful after that, but yeah, would have been a disaster.
It was a crash bug, but I'm not really sure the details (and it has been some years...). Even at the time, I wasn't personally involved in it, just heard about it through the grapevine.
But yes, my understanding is it was quite expensive and the publisher was none too pleased (:
Large companies are why large companies are large (in IT/OT at least)
reply