Thursday 22 March 2018

The Uber Dev Team just Killed Someone

A software development team at Uber just killed someone.

The Uber AV killing is going to go down in software development as one of the "things you must never do" examples up there with Therac-25 and various Flight Control System failures. The datasets and traces of the collision, will, along with all analysis, be reviewed, torn apart and rebuilt into some fairly brutal dissection of the entire software development process by the Association of Computing Machinery Risks Group, which is basically a documentation of fuckups of one form or another.

Without waiting for any analysis, police and others are already saying "it wasn't uber's fault". That's the wrong way to view it.

Better to try and categorise it

  1. Uber's system made a mistake —one that can be fixed and so never repeated. Other AV manufacturers can also use the data from the collision to avoid it in their systems too. 
  2. The collision was unavoidable given the action of the user and the general category of LIDAR-enabled Autonomous Vehicle.
Option 2 means we would now have make a choice. Do we consider the death-rate of AVs "Acceptable", or push for more to separate vulnerable road users from AVs. That could be banning AVs from cities "it's just too hard right now", or trying to do something banning pedestrians from walking around, requiring cyclists to have some beacon implanted in the bike and charged up on a regular basis to announce its presence, etc. That's the category of "admitting your tech can't do what was originally promised"

It is a lot better politically if it does turn out to be the uber team at fault. Or a least blame the driver for inadequate supervision.

How society and industry reacts to this, the first fully self-driving car killing will reflect on the decisions and priorities which society makes. It should be the duty of the software developers to be honest about the incident and open about whether it and similar events can be prevented in future.

Unlike all "human at the wheel" collisions, the actions of the uber car can be fully replicated: with the recorded sensor data from the car and the same version of the software, exactly the same set of actions can be expected. Therefore we can see where it failed.

Here is the list of places

  1. Sensing: failure to detect the victim with the sensors available.
  2. Interpretation: failure to recognise the victim
  3. Anticipation: failure to anticipate their movements (and/or the vehicle's) and/or failure to conclude from the anticipated trajectories of pedestrian & vehicle that a collision is likely
  4. Planning: failure to come up with a plan to avoid a collision
  5. Execution: failure to execute a valid plan.
What do we think?

Sensing: Uber, like the Google/Waymo vehicle has LIDAR: light-frequency radar Works perfectly well at night, wider angle of view than a set of headlamps. Range? Unknown. These are the state-of-the-art in sensing, very expensive, and why Tesla are claiming "you can get by without them". LIDAR is what engineers believe is needed. Yet here it has failed. Does it have known failure conditions? Rain, snow, hail, probably fog and smoke too. Doesn't recognise glass. Presumably doesn't work into low-sun either. "It was dark and they had dark clothes on" is not an excuse. 

Intepretation: First realistic way things could have failed. 

Anticipation: Not a fast moving vehicle, a person crossing the road. No obvious unexpected movements. This should not have been hard. If it is: give up AVs now.

Planning: Fun one. After concluding that a collision is likely, the car has to come up with a set of actions (brake, swerve, brake+swerve, accelerate, sound horn, ...). None of these actions seem to have been attempted.

Execution: Car skids on rain/snow/oil; swerving to avoid one crash triggers a second, etc. No actions appear to have been executed, so unless the car computer couldn't communicate with the steering/engine/brakes. not a failure point.

We suspect then: interpretation and anticipation. Or the sensors got an echo but discarded it amongst all the other inputs it had to deal with in a limited time. That is "prioritisation". 

We humans make exactly the same mistakes all the time. mostly you get away with it. When you don't, well, it could be skid and a dent, or you can be sliding down the crash barrier in a motorway, passenger and driver both trying to keep that steering wheel straight, everyone in the car screaming thinking  they are all going to die. Such events happen with all too much regularity across the country. and generally, unless fatal, nobody gives a fuck. Now its done by software, it gets a lot of press. We shall have to see what the outcome is.

If it was a software/system error, then it can be fixed. Uber can add a new scenario to test their software on, even before running it in a real car, and the limited set of AV manufacturers (Waymo, GM, Ford, ...) can use the same data in qualifying their software. Everyone can learn from it and repeats can be avoided. 

That's if we want that, and it is avoidable. The alternative is blame the victim, exonerate the software, carry on promising a safe utopia, now qualified with "some people may still die". 

That's a major change, as it is accepting that death is inevitable in driving the way we don't accept it for trains and planes. We do make that distinction in cars today, but that's because we all believe that "we" don't make mistakes, or that personal freedom "right to drive across town" is better than safety for all.

This death is going to have to make us spell out our priorities.

For now though, let's add Elaine Herzberg to the list of people killed by cars. She is one of the few to get a mention beyond the local press. That doesn't make her death any better.

Friday 2 March 2018

Bristol: Snow Day == Smug Day

It was pretty chaotic night out there. On the one hand: almost no traffic. On the other, an Audi A3  trying to get up a slight gradient and stuck wheelspinning. After about 15 minutes people start looking out the window thinking "maybe should help", after another five minutes they open their doors, get out and start pushing. Who says there is no camaraderie between passengers in Audi and the driver?

As the residents are all staring out the window "will they fucking get a move on", the passengers help turn the car round so its weights on the front axle, and it manages to get another 15 metres up the hill before being abandoned. Hopefully BCC have suspended overflights of RPZ enforcement drones for the next few days.

One of the wierdest things was the fact that everyone was driving well below the 20 mph limit. This completely throws off your decision making as you think "I'll wait until they get past before pulling out", and yet they take so long you think "I'll just pull out in front of them". It's the Audi/BMW driving experience on a push-bike.

All mini-roundabouts, give way signs and zebra crossings are suspended, and you can stop in the middle of the road to talk to people. Not different from normal, except now you have a better excuse when challenged than "fuck off"

We considered setting out to a nearby pub on the off-chance of getting snowed in and trapped for 2-3 days. But which one?
  1. Duke of York: good pub, risk of being stuck with David Wilcox. Which is fine, except after 24h of cider drinking he becomes a barge evangelist. We don't care for barges and think the remaining waterways of the city would be best built over the way they did with the River Frome to give us the M32. Sooner they do this for the harbour, the better. In fact, we are surprised the WoEP haven't suggested that to Marvin, as it'll be more popular and easier to deliver than a metro.
  2. Miners Arms: Thursday is Pub Night. Risk of a three day lock-in with pub quiz enthusiasts keeping on with the quizzes as something to do. Challenge of explaining to A&S Police how aforementioned pub quiz enthusiasts all ended up being killed messily with the kind of small nail clippers airport security love to confiscate.
  3. Farm Pub: highest chance of becoming stranded, Doom Bar on draft, easiest to acquire locally curated ganja (we are told). 
Tough choice

Anyway, that was friday, what about today?

Rejoice! Today is a day is a great to be smug!

Because yes, those of us who have the equipment to get round can go round, offering smug advice to everyone trapped.

Drivers of 4x4s and SUVs

Landrover Defender drivers have it easy today: that rickety thing they've kept alive through trading bits on ebay can now be used to cruise round the city. Wearing wooly hats, fingerless gloves and warm clothes obviously, given that class of vehicles' heating system.

Urban SUV drivers have a similar option, but need to be careful. Today may be the day that you can finally justifying spending £15K on top of a practical car..

Do: drive round, nodding knowingly at other SUV drivers.

Don't: discover when you get to a hill that you only paid a £12K premium and so have the 2WD model.

Do: use engine braking on the descents. If your car is an automatic, time to work out how to put it into manual mode, or try the cruise control.

Don't: discover while driving down one of the steep hills in Totterdown or Cliftonwood that it doesn't make a difference between 2WD and 4WD when it comes to using the brakes. No winter tyres, sliding into parked cars screaming.

Owners of cars parked on hills

It's too late to move them unless you pay your neighbour with a landrover to tow them somewhere.

Do: sit by the window, looking at your car, camera in hand, ready to jump out and take pics of whichever 2WD SUV drives into the side.

Don't: own a car of any net value.

Mountain Bikers: 

Do: cycle round offering smug advice to people trying to drive up and down hills.

Don't: get off and help, admit that it doesn't work and then, while pedalling of shamefully, wipe out on a patch of ice.


Do: walk down the middle of the road. Its the only bit which will get gritted after all.

Don't: look at your phone while walking round. Not today.

Don't: walk round with your hands in your pockets, in case you need to put them down when you slide out.

Do: get out the skis you've kept under the bed for a decade, carry them to some hill and ski down.
Don't: go on about it for years in any of the pubs our team members frequent We don't care, really.

Mid-life crisis mountaineers

We want to have a special call-out to mid-life-crisis mountaineers, the ones you see out in Gloucester Road or Southville of a weekend, wearing a down jacket branded with North Face, Patagonia or Mountain Equipment,

This is your chance to get out all the stuff you've bought from Taunton Leisure of Bedminster over the years and stomp round the city

Do: get out the most excessive bits of equipment you have in an arms race with all the other mid-life crisis mountaineers.

Do: walk around being smug, especially on the deepest snow parts of the pavement, rather than the middle of the road where it's clear.

Do: remember that if you wear crampons, walk with  your legs wide apart so you don't step on your own toes.

Do: talk to random strangers, casually dropping phrases like "hut-to-hut through the Haute Savoie Alps" and "New Zealand South Island trekking", or even better, "Everest Base Camp"

Don't: get out your stuff to discover that since you last went near any of those places the plastic boots have gone brittle and failed.


Well, with an equipment failure like that, its trapped in the Farm Pub. We just wish that skier would STFU or we'll have to start looking for those nail clippers