Page MenuHomePhabricator

Thoughts on Control Systems
Closed, ResolvedPublic

Description

Here are some terrible, wrongheaded things I currently believe about control systems, as of October 15, 2018. These bad ideas were developed independently of any proprietary information I may learn in the future.

I have no experience building or working with control systems, so most of these ideas are probably awful. I would expect to abandon much of this after even brief exposure to real work on control systems.

AI

AI is a trap and should not be a component of control systems in the long term.

AI is attractive because it will get good results relatively cheaply. But AI is a local maximum, and systems which are not observable will never be reliable enough for high-stakes control systems. If you choose an AI component because the cost to 95% accuracy is much lower than alternatives without considering the cost to 100% accuracy, you're failing to account for a large part of the cost and making a grave error in long-term planning. The cost even to get from 95% to market may be too high to make the project tenable.

AI may need to be a component in the short term, but it should be incorporated begrudgingly and with great suspicion, and replaced once better systems are available.

If you're building a control system, definitely don't put "AI" in the name.

Like "blockchain", AI is "cool" right now but is not appropriate for most problems it is being applied to and won't stick around in the long term. Unless new AI approaches (which make AI observable and explicable) are developed, it will fail to solve any significant problems where outcomes actually matter.

Don't build a "blockchain" control system. Don't build an "AI" control system. Definitely don't call your company "Blockchain Space AI Rocketry".

Murderbots

If AI must be incorporated into a control system, the system should not fail critically if the AI is replaced with a murderbot: an adversarial AI which perfectly mislabels sensor features to make the control system kill humans and cause mayhem. Simulation should routinely incorporate murderbots in test cases. If you don't, you're being irresponsible in assessing the actual cost of relying on an AI component.

All AIs today are murderbots. Most of them can be proven to be murderbots (adversarial input produces a misclassification). The others can't be proven to not be murderbots.

Layers of Simulation

The control system should abstract the world into layers of simulation which are separate from one another. When a scenario results in a planning mistake, it should be routine to reduce the scenario to a minimally complex test case and run it in simulation to reproduce the planning mistake.

There should never be a case where a planning mistake occurs and you can not replay inputs to reproduce the mistake, or can not understand why the mistake occurs.

Simulation layers and components should be modular and readily replaceable. When a planning mistake occurs, replace components (sensors, models, etc) with simpler versions to isolate the source of the error. Then, replace the simplified AI components with murderbots to make sure the behavior is robust.

Sensor modules should be reducible into toys, e.g. this should be a valid production test case for a market-ready vehicle control system and the planner should stop the vehicle (^) before it encounters the unknown obstruction (?). The planner should work as well if the sensors are reading from an ASCII text file as if they are reading from the real world. When possible, planning mistakes should be reduced to ASCII text or some similar minimally complex scenario format.

X?X
X.X
X.X
X.X
X^X

Observability

Each layer should be readily observable. Engineers should be able to review inputs and outputs to each subsystem and understand why the outputs result from the inputs. When a planning error occurs, it should be immediately obvious where the error originated.

Significant effort should be invested in making state easy to understand and explore. The tools to understand system state should be good enough that they can be shipped to market and used by (sophisticated) consumers.

When public planning mistakes have occurred with existing control systems, I haven't seen companies release comprehensible details about system state and the sources of error. I can come up with several explanations:

  1. System state is not observable.
  2. System state is observable but terrifying (looking at the man behind the curtain and seeing that it's a big RNN which no one can actually understand could reduce consumer confidence in control systems).
  3. Revealing system state would reveal proprietary information.

In cases (1) or (2), these systems are probably doomed. They should be thrown away and rebuilt to be observable or they'll never overcome the wall beyond AI.

Case (3) is ridiculous because the high level design of control systems is obvious: sensors build a model of the environment, a planner makes decisions based on that model. All that disclosure would reveal is that your system is sensibly designed.

For every planning mistake your control system makes, it should be easy to extract a high-quality, relatively comprehensible infographic explaining which sensor or planning component or components made errors, why, and how you've remedied the issue.

System Deployment

Some vehicle controls systems today are focusing on reaching the market quickly by solving the simplest and most profitable control problems: cruise control, automated braking, lane assist, and long-distance logistics.

These problems are attractive because the market seems to be within reach, but they are high-speed scenarios and thus high-risk. If the control system fails, major damage and possible loss of life results.

Real-world vehicle control systems should first be deployed to golf carts that drive at 2mph and are covered in bumpers made of soft, downy pillows while making a loud, annoying quacking sound over and over again. Automate tractors and airport shuttles. Automate boats. Automate construction vehicles and warehouse operations. There are plenty of simpler real-world problems that will at least build a case for eventual commercial viability for consumers. Don't start with trucks. Definitely don't touch consumer vehicles. Solve low-speed problems where the cost of failure is low and bystanders are informed and alert.

Look for ways to reduce existing problems into safe, solvable control system problems.

Buy Wyoming, build an autonomous-only road across it, and put pick-up and drop-off stations on either side of the state. Shipping companies can drop containers off outside Cheyenne and pick them up later near Yellowstone.

NHTSA has 5 levels of vehicle automation but they are not evenly spaced.

Level 1 is not a meaningful control system. Levels 2 and 3 are dangerous and should not be shipped to consumers (even though they are in production consumer vehicles today). Level 4 is commercially viable. The distance between levels 4 and 5 is enormous and far greater than the distance from level 0 to level 4.

Level 4 control systems that can fail safe (e.g., stop the production line; pull over to the side of the road; hover in place) can take small steps toward level 5 by allowing operator intervention: operators can physically intervene; operators can take over control remotely; coordinating control systems can recover.

Imagine: you get into a driverless car. It drives very slowly a few hundred feet down the road until it encounters a work crew trimming trees. It pulls to the side of the road and stops. Several minutes later, a second car drops off a driver who takes over control and drives you to your destination. The trip was half price and you opted into a driverless ride, so you don't mind the delay too much. Many years later, you realize you haven't had to wait for a replacement driver for a while. Intermediate steps may involve remote control or delegating control to passengers, allowing them to review and authorize uncertain plans.

Passenger control can be normalized by letting users review and authorize trivial plans like routing. Even though the control system does not need input for routing, asking for it reinforces that the passenger is in control and that the control system is a stupid tool. These are good things to reinforce.

It might even be good to ask users to confirm decisions that the control system is certain about, e.g. have users authorize the control system to proceed through intersections. The control system should consistently try to appear less advanced than it actually is so that actual error conditions are routine. It is much better for the control system to appear annoying and stupid when it's actually pretty good than for it to appear magical when it's only pretty good.

The first major goal should be to reach a level 4 control system that fails safely in all conditions, even if the range of conditions under which it can operate is very limited. Build stupid robots that give up at the slightest sign of difficulty. Anyone interacting with these systems should be annoyed by how cowardly and incompetent they are. It should be inconceivable that they could ever hurt anyone.

UX

Today's vehicle control systems look like cars, except Waymo's goofy little mini car, which I haven't actually seen on the road. Some of them look like cool, sophisticated cars. This is probably a mistake, and Waymo's goofy thing doesn't go far enough. Some of this may be NHTSA requirements, but vehicles should look stupid and harmless as they behave stupidly and harmlessly.

Vehicles should have large rear-facing screens emoting to drivers that they're planning to turn or stop, and that they're scared of hitting things. They should be happy when they're not moving and terrified when they have to drive places. Emotionally, other drivers should experience vehicles as stupid, clumsy children. Vehicles should generally signal their intent and state to their environment as clearly as possible: the more observable something is, the more trustworthy it is.

Current vehicle signaling (stop lights, turn signals, waving at pedestrians, flashing high beams, awkwardly inching into an intersection, etc) is limited by human ability to actuate controls, but this is unnecessary if the vehicle has a computer control system. The vehicle can clearly signal its intent visually, and eventually electronically to other vehicles.

Probable Objects

Sensor systems and modeling systems built on top of them should propagate probabilities into the environmental model. If the sensor says there's a 99% chance an object is a widget, don't put a widget into the model: put a 99% chance of a widget into the model.

This isn't how we normally think about or model the world, but modeling should fundamentally incorporate uncertainty as a part of its view of the world. The model should not discard uncertainty until the last possible step when it must make a discrete planning decision. When the model is later stopped and rewound, planning mistakes that resulted from probabilistic error should be immediately evident.

In the face of too much uncertainty, the system should prefer to fail safe. This generally provides an easy default rule for failing safe: too much uncertainty.

Trolley Problem

Many words have been dedicated to the idea that control systems will need to solve the "trolley problem" and decide which of several bad outcomes is least bad. This is ridiculous and not a problem control systems need to solve before they are ubiquitous and perhaps ever.

But this is apparently a credible idea in the public consciousness, and suggests that the public perception of control systems is largely baseless fantasy.

Control systems should be so un-magical that the trolley problem is obviously beyond their domain. No one expects dogs to solve the trolley problem, but everyone expects dogs not to walk into things. Control systems should be aiming for this. They should sit and stare at you and look cute when they don't understand how to interact with the world.

Architecture

Control systems probably have a well-established architecture which everyone pretty much agrees on, but I haven't read any literature or ever built one. Here's how I think they probably should work:

Sensors gather information about the environment. For example, a camera can see visual light in the environment or LIDAR can build a depth map. Other sensors represent physical device controls, fuel levels, etc. Some sensors may also incorporate feedback from later parts of the system: if planning generates an instruction but the control system can't actually actuate that instruction, that may generate sensor information. When you pause and rewind the system, you should be able to look at the values each sensor generated recently.

Analyzers read one or more sensors and convert raw sensor inputs into useful information about the environment. Uncertainty should be introduced here. Some analyzers will emit completely certain information (the user pressed a button). Some analyzers will emit highly certain information (e.g., if the fuel level sensor reports a momentarily high level, the analyzer can report a very high probability that the actual fuel level has monotonically decreased, because we reasonably know that fuel levels do not make discrete jumps upward across small timeframes). The fuel level analyzer may consider whether the fuel door was opened or whether the vehicle recently accelerated in determining certainty. But this uncertainty should still be part of the system, and when you stop and rewind the simulation you should be able to understand why the sensor read "80%" but the analyzer produced a 99.9% confidence that the real fuel level was 40%. If you have someone sit on the back of the vehicle and fill the fuel tank while it is in motion, uncertainty should accumulate and the vehicle should eventually fail safe (or accept that it can no longer determine the fuel level and can not make planning decisions based on the fuel level).

Analysis includes converting raw LIDAR inputs to a depth map and computer vision AI murderbots.

The simulation is a model of the environment based on the information from analysis. Some parts of the simulation are straightforward; other parts are complex. The simulation should be reproducible to a version of the world that is too trivial to actually realize. The simulation includes a model of the system being controlled, e.g. the vehicle or arm or drone.

Planning examines the simulation and makes actuation decisions, e.g. stop the vehicle or begin a weld or fly forward.

Actuation executes the plan, and provides feedback from hardware controls where possible.

All control system components should be generalizable. If your control system can drive a vehicle, it should also be able to drive a robotic arm or a drone or a rocket by swapping appropriate sensor, simulation, and planning modules. (Building these modules isn't trivial, of course.)

The architectural goal should be to build a general control system, not, e.g., a vehicle control system. It might be good at controlling vehicles first, but if it's a good control system it should generalize relatively easily to other similar control problems and not-impossibly to all control system problems. It should generalize very easily to simulated control problems.

Across all components, the most important property is observability. It is also important that the simulation be accessible, and the simulation's environmental model be reducible. Components should be modular and state should be explicit, snapshot-able, and rewindable.

Revisions and Commits