36C3 - How (not) to build autonomous robots

36C3 preroll music

Herald: So Sasha is a doctor with a weak
spot for LEDs and, completely abstains

from HDMI adapters these days. He wanted
to share with us the experience of

attempting to build a delivery robots in
the past 2.5 years in the Bay Area. And

so, yeah, let's give him a big welcome.

Sasha: Thank you, Mikael. So just a show
of hands who here has built robots before?

Well, it's quite a few people. What about
autonomous robots, anybody built

autonomous robots? Still quite a few
people. Well, today I'm gonna be sharing

with you the story of how not to build
autonomous robots. Over the course of the

past two and a half years together with my
team, we built the world's largest robotic

delivery infrastructure. We went from a
concept sketch to a commercially viable

service running in three cities. We've had
lots of successes and one or two failures.

So over the course of the next 45 minutes,
I'm going to be sharing with you a couple

of different stories. First of all, I'm
going to briefly introduce myself and I'm

going to share the story of how we built
robots, the different prototypes we had,

the different iterations that we tried.
Then I'm going to jump on manufacturing.

We actually went to China and scaled up
our manufacturing, our production line.

I'm gonna share with you the story of how
he did that. And finally, I'm gonna talk

about A.I. and all the magic that is
artificial intelligence. So we'll be able

to see how we were able to crack that
puzzle. So without further ado, let's do

the introduction. This is me. Right here.
I like to build things. I built my first

website when I was eleven and I built my
first business when I was 13. I was the

iPhone repair business that I was running
in my bedroom. I've been really, really

passionate about building things and over
the course of many years I built a couple

of different startups. One of them was a
food delivery platform. We ended up

running three different cities and doing
hundreds of deliveries a day. By the time

I was 19. So I got to experience startups
pretty early on. I've been really enjoying

that time. After this food delivery
startup failed I went to some

cryptocurrency startups and then went to
work for big corporations. And that's

actually very boring. I dorned my office
with some supplementary graphics. After a

while, I got a little bit bored of this
corporate life. It wasn't really for me.

So I decided to get a one way ticket to
San Francisco. So I ended up in San

Francisco staying on a friend's couch, not
really knowing anybody. And I was really

fortunate to be introduced to an
incredible group of people. And over the

course of about two and a half years, we
started to take a concepts. A scatch that

we had and we built up a robot. At first
it was something that barely even worked.

But then we gradually got to something
that worked a little bit better and better

and better. After a while, we actually
managed to build a whole fleet of robots.

I think at the peak we had 150 robots. So
it was a really, really cool experience.

And during that time, I got to meet the
lieutenant governor of California, how to

figure out how to do manufacturing in
China and most importantly, work with an

incredible team which who I had a lot of
fun with building these robots. So, yeah,

it's a little bit about me and what we
were building. And maybe now we can jump

in to how not to build robots. So this is
our very first robot,this is a really

small prototype. We built is basically a
shopping basket on wheels. There is a RC

car there below, there's a shopping basket
and Arduino Raspberry Pi. The thing barely

work. Honestly, it was really, really
hacky. And what ended up happening is that

most of the time we just dropped off the
robot in front of the customer that

literally just dropped it in front of the
door just to see if they would like order

food with robots. The answer was
overwhelmingly yes. So we decided to spend

some more time building our technology.
There is a small - I don't know if you can

see it here. Yeah, there you go. There's a
small orange holder, that's actually a

phone holder. So our very first prototype,
it had a phone sitting on top of it doing

a video call so that somebody can remotely
control it from Colombia. So we really

started out small, really humble just to
see if it would work. And that's something

that we did a lot of this being really
resourceful in terms of trying out things.

For about a year of this, we moved on to
something that looks a little bit more

like this. So we started playing around
with the shape. We start playing around

with the design. We noticed that people
responded really positively to faces and

to like things that looked like people. So
we actually built in a face. So we took

this little animation that we built and we
put it onto the robot and there's actually

really, really positive. We had a lot of
good responses from the community, a lot

of great feedback. And what we've seen is
that people really love to have robots

that are kind of friendly. There was
another company that deployed robots that

looked like vending machines or almost
like tanks in San Francisco, and they got

banned really, really quickly. So we
decided that we would do our best to make

sure our robots were as friendly as
possible instead of threatening and scary.

So that was a very important part of it.
After another year, we ended up scaling up

our production and we went to China to
manufacture robots. And here this is what

we ended up doing.
music

It's actually, a cool robot. We built it
entirely from scratch. We got our own

chassis, our cabin, our own compute
module. Basically just about everything.

That was a really cool experience. That
was me. So yeah, that's a robot. That's

the one we were rolling around the past
six months. And we also had some failures

in between, as you saw previously, this
one. So we actually tried a couple of

different concepts. So this was one of
them. This was a Kiwi trike. We thought

that maybe we can figure out how to have
robots do part of the delivery and then

trikes to another part of the delivery. We
also tried to do restaurant robots. We had

like robots that sit in the restaurant and
bring food out from the counter to your

doorstep. But what ended up happening is
that it was actually pretty inefficient

and people would wait a really long time
for their deliveries. So it was very

important for us to try a lot of different
things. We tried this robot, the kiwi

TRIKE that not quite worked out as we
expected. We tried a restaurant robot. We

tried a box that would sit behind our
robot. We tryed a hub that would have like

of a bunch of different robots inside of
it. So we really, really tried a lot. And

with every iteration, we constantly tried
new techniques we costantly tried new

manufacturing methods. We really tried
just about everything to see if we can

make it work. And what we ended up
building is a platform that was really

loved by people. We built a platform that
students adored. That was our primary

demographic we're delivering to college
campuses and students really loved our

products. We actually had people dressed
up as Halloween costumes. We had entire

classes go for Halloween in like kiwi bot
costumes. So that was really, really cool

stuff. Had a lot of great support. A lot
of trust from the community as well. And

that's like coming back to the design.
That aspect of having a friendly robot,

that meshes seamlessly within the fabric
of a community is like super, super

important. We've seen other robots around
and they were maybe not as friendly, maybe

they looked a little scary. Maybe they had
something that was a bit off or maybe a

little too industrial. But having like a
friendly robot that could become a meme

that was something truly revolutionary,
something that really changed the

landscape. And as a matter of fact, like
these cute bots are the only robots that

are deployed somewhere in the world where
they coexist. Day to day with a community,

with people. Like you have some limited
deployments of robots here and there,

maybe have a Roomba at home or some like
that. But you don't have any large scale

deployment. We have robots and people
living in the same city all the time. So,

of course, it took us a while to figure
out what to do and how to do it. At first

one of our models was to have robots
deliver the entire meal, like go from the

restaurant all the way to the customer and
we would have a robot do that delivery.

Turns out, it was pretty inefficient.
People would wait like 60 minutes, 90

minutes for their delivery. And we
realized that maybe automating all of that

was not the most efficient approach. So
what we instead did is a multi-modal

approach where we had people and robots.
This is actually a really cool

visualization that my team came up with.
The blue lines are robots. So these are

robots roaming around our Berkeley
coverage area and the yellow lines are

people. So how this would work is that
people would go to restaurants, they pick

up the food and they take you to a
cluster. They take it to a cluster where

you had a bunch of robots, they loaded
into the robot and then the robot would

actually do the last few hundred meters to
your doorstep. And because we were able to

do this, we were able to go and build a
platform, that handled hundreds of orders a

day with very, very few people. I mean,
labor costs are really high for delivery.

You'd be paying somewhere between five and
thirteen dollars to get a meal delivered

in the U.S. And as a student, that's like
super expensive. That's not something that

you can afford do every day. And also
there is a pretty big shortage of people

who want to do this job in the first
place. The trend is really high. People

are leaving all the time because they
don't like to like sit in a car all day

and deliver food. So that's why we have
this parallel like this multi-modal

approach where the people are biking around,
they're enjoying their time outside and

the robots are actually doing other boring
stuff like the waiting. So the robot would

go up to your doorstep and would wait for
you to put on your pants, your shoes and

actually walk outside. So that way we were
able to change the dynamic. We're able to

change our deficiency from one or two
deliveries an hour, as you would have with

like a traditional delivery service to as
much as 15 deliveries an hour per person.

So it made the delivery far more
affordable and we were able to offer

delivery at just one dollar a delivery,
which is a cost that changes completely

the way people approach delivery. In fact,
if we look our top 20 percent of users,

they were ordering over 14 times a week.
So they were very, very happy that they

could get whatever they wanted very
quickly. Of course, not everybody was

super happy. So we did have some people
that didn't fully appreciate the magic

that is kiwi bot. So we did have one
person try to steal it, but they didn't

get away with it. We found them pretty
quickly. They hid it in the trunk. Not a

very smart move. We ended up finding it with
G.P.S. and also triangulating the Wi-Fi.

So this guy decided to steal it because he
doesn't like robots. I don't know why, but

he was clearly very passionate about that
topic. And he stole it and now he's in

jail. So, yeah, don't steal robots. So
maybe some conclusions from our robot

part, like from building robots, from
figuring out like what to do and what not

to do. Really important thing that we do a
lot in software and maybe not as much

hardware is iteration. Like we iterated
through three major revisions and like

lots of small revisions during a really
small period of time. It was really

interesting to see like that transition.
Every single time we try something new, we

try it maybe for like 20 robots at a time,
like not our whole fleet. We just try for

a small portion of our fleet and that we
were able to iterate really quickly and

see what sensors worked or cameras worked.
And just to see what we could do in order

to grow the products, it was very
important to iterate. Communication.

Communication is absolutely fundamental.
And not only communication like inside the

company or anything, but more importantly,
communication with your community. Because

we weren't just building a product in
isolation. We were building a product for

people who live in a city, who have an
established life. And we're kind of

intruding into their lives by bringing in
a new product that takes the sidewalks. So

communicating what we're doing, showing
them what this is and what this robot does

is super important. Actually, very early
on our designs have no text on it. They

had like no information it was just like a
basket case on RC car. And people were

like really confused. The police were
like: "Hey, what is this?", so we had to

add a lot of communication, we had to put
food delivery on the robots really

clearly, we had to add a license plate
with like a phone number that somebody

could reach out to us. So communication is
very, very, very important when it comes

to robots. Also: Scaling hardware is hard,
super hard. I mean, it was crazy. When we

first started it was just Arduinos and
Raspberry Pis and that did not scale

really well. Like, sure, we could have
maybe 10 or 20 units at once. But then how

do you handle updates? How do you handle
those weird things that happen all the

time? So it was really challenging to do
this. We actually killed a bunch of SD

cards. Didn't really know you could
destroy SD cards, but you can. And we

learned a lot of things about hardware
pushing it beyond its normal boundaries.

So yeah, iteration, super important.
Communication is key, like getting buy in

from the community and scaling hardware is
super, super hard. That's something we

actually figured out how to solve by going
into China. So how to do or how not to do

manufacturing? So as every China story
goes, I hopped on a plane and I ended up

in China. And it's really interesting to
see because like you have this perception

of China from the media, you have this
idea of what it would look like. But the

reality is it doesn't look anything like
what you would expect. It was a completely

different world. It was at the same time
Bladerunner and like the most modern city

in the world and it was truly an awesome
experience. I highly recommend anybody who

has the opportunity to go in and explore
the world. But of course, the culture is a

little bit different. We were surprised to
see some things happening there. Was a

weird dichotomy between communism and
consumerism. This is kind of interesting

to see that sometimes. But the reason why
we came to China is for manufacturing and

there is no better place for that than
Shenzhen. In Shenzhen, you have

Huaqiangbei. This huge market. It's a
market that spans several city blocks and

you can actually find anything and
everything you want. We were able to get

components super quickly, super easily.
And you could spend days just walking to a

single building finding different things.
There were entire city blocks dedicated to

like just LEDs or just connectors or just
processors. It was absolutely crazy. You

could really, really, really get lost
inside of these mazes. And what was really

incredible to see and something I've never
seen anywhere else in the world is just

how easy it is to get hardware, to get
things, to get parts. It was super easy.

Just go in and get something and you could
get it at one piece, two pieces, a

thousand pieces like instantly. If you're
anywhere else in the world, that's super

hard to do. So just by this virtue, you're
actually able to prototype things. You're

able to build things incredibly fast.
You're able to go in, you're able to

comission a PCB and get all the parts
almost instantly, which is not something

you see anywhere else in the world. And
also a lot of the manufacturers have their

booths here so these would be direct
booths from the manufacturers so you could

say go up to them, start talking to them
and ask, hey, can you make this product

this specific way? Can you do it how I
want it? And they'll be like, sure, why

not? They'll do it for you. So it was
really, really valuable to just learn from

these people, from the vendors here, from
manufacturers about how to build things.

And it was actually really surprising to
see everything they had in stock. Two

years ago, we built an art installation
here that covered a tunnel with LEDs. We

covered one of the tunnels at 34C3 with
LEDs. And we used this tiny, tiny chip. It

was a five dollar ESP 8266 chip that
basically was able to control all your

LEDs. And over the course of five years,
up to that point, I spent a lot of time

figuring out how to build it myself. I
played with Raspberry Pis I played with

PCA controllers over serial and like I
finally managed to get a prototype to

work, but it was super clunky, it was
super expensive and it wasn't very

reliable and I go to China and I find that
it's available there and much better

quality, much cheaper, much faster, so it
was a really, really interesting shift in

perspective. It's something you can't
appreciate when you're abroad. Even if

you're browsing like eBay or Ali Express
it's kind of hard to appreciate just how

much selection you have and how you can
find just about any tool, anything you

need to find. So it's really, really
incredible. But these markets were cool

but was even cooler are the factories and
during a course in China, we were able to

visit a lot of factories. All these
factories there are super, super

welcoming. They always love having you
over. They invited you to really, really

luxurioius dinners. We had way too much
food. And it was a feast of celebration

every time. Actually, relationships are
super, super important in China. Like a

lot of people in the West, like they have
contracts and they say, OK, this is the

terms of the contracts. Well, China, you
do sort of have contracts, but they don't

matter as much as relationships. Like when
you have a relationship with the

manufacturer, you have to like always go
to dinner with them, drink beer, smoke, go

to KTV like it's a really evolved
relationship. And you're only able to have

good communication based on that
relationship, because if you don't have a

relationship they kind of forget about
you. We actually had a couple of instances

where manufacturers ghosted us. Like they
had a critical component and they just

stopped answering our emails, they stop
answering our weechats, they just

completely ignored us. And for some pieces
they were completely irreplaceable, we

could not just go out and find another
factory to produce a specific part the way

we wanted. And the only way you can ensure
that this doesn't happen is by really

explicitly making sure that you have good
communication, a good relationship with

that manufacturers. It's super, super
important. This is one of the factories we

worked with. It's really crazy. I mean, we
went there and we're just absolutely blown

away by the scale of everything and also
blown away by how manual everything is.

There's actually audio here. Everything
was super manual. People were just like

there with minimal or no protective
equipment whatsoever. Just like building

things that look like they were made by
robots or machines, but they were in

reality built by people with their hands,
which is super crazy to see. And there

were a lot of Blade Runner esque designs,
really bizarre contraptions there in this

factory. This is our fiberglass factory.
The way we build our casing, was actually

prototyping it first in carbon fiber,
sorry, fiberglass and then moving onto a

mold in carbon fiber. And actually Scotty,
he made a really cool video on YouTube. So

if you search for hockey stick factory on
YouTube, you can see a huge video. My

buddy Scotty actually goes and meets his
factory, discover how they make this mold

and how they make these carbon fiber
things. It was actually really crazy to

see it. It was cheaper to make a carbon
fiber mold than it was to make a plastic

mold. So since the tolerances were a
little bit different, since like the

process was a little bit simpler, you were
able to make a mold, that was very, very

strong and very indestructible without
necessarily having to have all of that

expense upfront for like a plastic mold.
So, yeah, that our a fiberglass factory.

Really exciting stuff. Really crazy scale.
These folks like the first night we came

there, we arrived at like 8 pm and there
was 100 people in the factory just like

working at 8pm. Really crazy to see. This
is another factory worked with. So this

was a metal factory. It was actually
really, really, really interesting to see

how they built all these things and at one
hand, you can build super complex things,

you can build a super complex designs. But
on the other hand, we got surprised a

couple of times by being unable to
manufacture really simple designs. And it

took us a while to get a grasp was like,
oh, OK, so we can make really complex

metal that's bent, but as soon as we add a
well to aluminum, you start to have a big,

big problem. So we had to like change a
lot of our designs. We had to really adapt

to the way things were being made in
China. And sometimes you could adapt

yourself, but like at an insane cost. So
it was better to adapt to the way things

were being done there. So, again, very,
very interesting to see how things are

done. No protective equipment, this is
like a two ton press and his hands are

millimeters away from it. So, yeah, it's a
different world out there. Very, very

different. Another factory we visited was
a PCB factory. So this one has a really

interesting story. This factory is not in
Shenzhen. It's just across the border from

Shenzhen. The city actually passed a law a
couple of years ago that has very, very

strict environmental policies. So you're
no longer able to do PCB manufacturing

inside the city anymore. So we actually
had to drive for a couple of hours outside

of the city and over there was a huge
plant. And this plant was kind of semi

automated, semi handmade. Were part of the
process were done by hand, as you see

here. But then parts of the process were
done with machines. So they had this giant

machine, which is basically a black box we
can't really see inside of it. But you had

a bunch of chemicals and it's like take a
PCB and just like move it forward through

a chain. This is really intresting to see.
And this factory also had a really quick

turnaround, they had a three hour
turnaround if you paid a premium and the

standard was 24 hours. You could also ask
them to do PCBA so you can actually get

them to assemble the PCB for you. And we
ended up doing that for some of our PCBs.

We'd give them build materials and we'd
give them our designs and then they

manifacture it. We actually got in a
little bit of a situation with that

because we sent them some designs, we sent
them some parts that we wanted to put in

our PCB and it turns out that one of these
parts was unavailable and they didn't tell

that to us until it was almost Chinese New
Year. So we had to scramble all that to

find another solution. Was very exciting
to see how you would deal with these

factories. There are some even cooler
factories. I think the coolest factory I

visited was a battery factory where they
made lithium ion and lithium polymer

batteries and it was almost entirely
automated. It had giant films of things

going into a machine and then you had all
sorts of liquids and powders it was all

combined together. It was super, super
cool. Didn't allow us to film it

unfortunately, there may be only a dozen
or so such factories in the world. They're

very protective about their technology,
but the scale of how quickly they're

manufacturing these batteries was just
incredible. They would manufacture them at

a crazy, crazy scale. So all these factors
are cool, but actually building things is

even cooler. So we ended up partnering
with a contract manufacturer. I was really

fortunate to find one through my network.
Otherwise, I would have been totally lost.

A couple of days before I ended up going
to China I found a contract manufacturer

that liked to work with start ups and
small scale people and we ended up working

with them to build our first batch of 50
robots. It was really interesting to see

how different our designs were to what
they expected. So they expected things are

really ready. They are very explicit, very
clearly specified. But we didn't have

that. The difference between manufacturing
in the U.S., for example, against China is

that in the U.S., like it's a super long
process and the back and forth takes super

long just to get an idea of what kind of
files they need. Whereas in China, you're

able to sit down directly with the
engineer, with the person in charge, and

you can figure out what they need and they
can help you out instantly. Actually, just

here, I just want to show you one thing.
So this is my designer, Alehandrew, and he was

translating from English to Chinese with
his phone with Google Translate, and it

worked surprisingly well. Google Translate
actually is not blocked in China for some

reason. We were able to communicate almost
all the time with that. Also, Weechat has

a built in translate feature. So Weechat
is like the universal app that everyone in

China uses and has this built in
translation feature that can translate

your text automatically. So it's really,
really cool to see how that worked. One

question that we get commonly asked is
like how do we find our manufacturers? How

do we get this relationship? So about 20
percent of that was through Alibaba. So

our fiberglass manufacturer. We quoted
like 30 different manufacturers and went

with the cheapest one. Of course, it was
far more expensive than we expected and we

ended up producing with them. 20 percent
that, for example, our chassey, it was

built with companies that we already had a
relationship with. So we were just able to

continue working with them. And then 60
percent was through just references so

select networks or getting to know people
and talking to them saying, "Oh, hey, who

did you use for this or this" or "How did
you make these PCBs?" or just getting a

conversation going. So having that kind of
network was really, really helpful in

order to build these robots. So as you can
actually see right here, our design, this

is what we had when we came into China.
When we left, we had our own computer

module like super sophisticated. But this
was like a Raspberry Pi, a pix hock and a

voltage converter like a DC to DC
converter. That was pretty much it. As you

can see, it was not very reliable. It
would break a lot. So it took us quite a

while to translate this into something
that was manufacturing well. So thanks to

the dedication of my incredible team that
we're able to do that. And we kind of did

not know what we were doing so, we ended
up having all of our parts and all of the

components ready just days before Chinese
New Year. So we actually had to do all of

that someday ourselves. We didn't have any
Chinese workers who could help us do that.

So there's our team just assembling things
in the factory like we wanted two days

before Chinese New Year. So that was very,
very interesting. We kind of hacked or

tried to hack Chinese New Year. We
assembled all the robots literally days if

not hours before Chinese New Year and we
shipped them out and everything was great,

except our robots got stuck in customs. We
had a trademark on our box and the customs

agents, they open the box and saw more
trademarks on some parts. We had 3D

printed parts and they were like, no, this
is not going to go through without the

proper paperwork. So our robots got stuck
for three weeks in China, which was really

fun. Little problematic. So, yeah, those
kind of things happen you have to be ready

for it. After we received our robots in
California, we had to spend another like

maybe one or two months refinishing them,
redoing some parts, tweaking them,

flashing them. So there's still a lot of
work to get them to work. The pieces we

shipped out to China was maybe just like a
case with most of the electronics in, but

not all of it. So we still had to do a lot
of tweaking over back home. And of course,

all this going to be impossible without an
incredible team so I was really fortunate

to be with some really, really passionate
people who would work four months in a row

continuously without virtually taking any
breaks. We had plenty of opportunities to

go and take the high speed rail or go to
Shanghai or even Tokyo, but we all stayed

in Shenzhen and spend a lot of time
together building these robots. It was a

really, really arduous journey. So maybe
some conclusions for scaling,

manufacturing, some of the failures we've
had in relationships. I mean,

relationships are super important, like
super, super important, in China far more

important then contracts. If you're able
to have a good line of communication with

your manufacturer, that really, really
helps out. Because if you don't, things go

bad. We've had manufacturers that ghosted
us. We have had manufacturers that

completely ignored us or manufacturers
that just replaced components because they

just felt like it. So relationships, super
important. Don't hack Chinese New Year. We

tried it, doesn't work. It's a thing.
China just sit's down for like two or

three weeks. So it's really, really
important to respect that. People buy

tickets to go to their hometowns like
months in advance and they're not going to

move it for just like some pesky thing
that you're building, especially for like

some small scale thing. So, yeah, don't
try to hack Chinese New Year, it did not

work out well for us. Also, do it with a
team . While I was in China, I saw a

couple of sole entrepreneurs try to build
their own thing and it was super, super

hard, super stressful, having a team is
really great, especially if like a foreign

place where you don't really know anybody.
Having that team there together, to

support you is super, super important,
especially since you can multitask, you

can split responsibilities and do
something together. So it's really, really

important aspect. So that's how we
manufactured and some of the failures

we've had. Now let's talk about how not to
build A.I.. So as we all know, A.I. is

magic, right? Just as Blockchain and IoT
and the cloud. It's absolutely magic,

right? Well, the reality is it's it's not
that magic. So we decided to have a very

pragmatic approach to A.I.. We said, let's
not do anything crazy. Let's just make

something that works. So our very first
iteration of a robot was this. This is

like the control panel for a robot. It was
super simple. We had a video call coming

in from the robot, on the left over there
is literally an iframe, super simple

stuff. And on the right, we had a map, on
the bottom we had some controls so you can

move the robot forwards, backwards. It was
very, very simple. It barely worked. On

the robot we had our Arduino, Raspberry Pi
all running in python and the server was

Java communicating over web sockets. But
this barely worked. So we decided, OK,

what can we do? Maybe we can build an
autonomous robot and we can both say that

would work entirely by itself. We actually
did that. So we built a robot that could

go entirely by itself. It was fully
autonomous. And it was actually really

cool. The way we built it is we had pretty
beefy computer inside. We had it a Nvidia

Jetson TX2. On that, we were running ROSS
and inside of ROSS we were running

TensorFlow and a couple of other
technologies. We had YODA for object

detection and some other cool tech that I
am not entirely familiar with it since I

didn't write that code, but over here what
the robot did is it looked at objects. So

it was detecting objects. It was also
measuring the distance to the objects. And

it also had an inference neural network.
And you can see that on the top left of

the screen here. Basically, based on
trained data, it would know where not to

drive into and it would try to plot a path
based on 12 different directions it could

go into. So it had 12 directions and it
would go in the direction which had the

highest probability of not colliding with
somebody or something. And this worked,

OK. We were able to get like 99 percent
autonomy. But the problem is, since we're

doing a commercially viable delivery
service, that's like offering deliveries

to regular people and not something in the
lab, it really had to do something that

worked all the time. And the challenge
with this is we still needed to have

people in the loop. We still had to have
people who looked at the robot to make

sure it would actually not crash. And what
happens if you have something that's fully

autonomous and people assume it works
well, when it doesn't work well instead

of looking at the screen and being ready
to take over, they're just looking at the

phone and Instagram. So this approach
wasn't the best one. And instead, we

decided to use a supervision approach. So
we spent a lot of time building this. So

this is our supervisors console and it's
actually really, really cool platform.

It's a platform that allows you to connect
to a robot and the robot streams to you

video over Web RTC or like the 4G network
and you're able to control it over web

sockets. So the way to work is you'd have
a supervisor that sets waypoints for the

robot to follow. So the supervisor would
click on the image and he or she would

tell the robot to move 10 meters at a
time. So typically they'd set waypoints

every 5 to 10 seconds. It was a very
interesting approach. We tried a couple of

different approaches. We tried to do slam,
that really did not work out for us. It

took too much resources and it didn't give
us a significant gain. We tried other

things as well. We tried traffic light
detection. So we tried traffic light

detection. There are some amazing models
available online, some great Github repos.

The problem is, yes, they do work on a
very clean data set. But when you actually

have data, we actually have a real life
scenario where we have like glare, you

have rain, you have weird situations, you
have homeless people. It doesn't really

translate that well in the real world. So
we kind of struggled with that. Instead,

we actually had a more middle ground
approach. So we are able to detect traffic

lights really well, but we're not able to
detect the color really well or the which

kind of signal it's giving. So instead,
all we do over here, this automatically

zooms in to traffic lights. So it's very
easy to see. This video actually that

you're seeing is transmitted over very low
frame rate, very low bit rate as well. I

think we're doing 480p at 100 kilobytes a
second. So it's very, very low bit rate.

And when the robot isn't moving, we
actually make it go black and white and

even lower rate frame rate so that it
doesn't waste resources. So yeah, it's

pretty cool stuff. Over here on the top
left we actualy have our latency. So we

managed to build the infrastructure that
allowed us to supervise this robots from

Columbia for 200 milliseconds and less than 20
milliseconds. So it's like a blink of an

eye. It was a really, really cool
technology, it worked or 4G and we did a

lot to optimize that. We had also a map
over here. So this map is really, really

cool. A lot of people ask us like, hey,
did you do mapping? Did you map out your

environment? Did you need to have
something there before you came into a new

place? And well the answer is no. But what
we do instead is we actually map out the

network conditions. So we would map out
the network conditions of a city and we'd

say, OK, these areas like over here. This
is like high latency. We should avoid

those areas because the robot could get
stuck there. It is actually very

interesting to see the network conditions
change continuously. You didn't have the

same network conditions every day, all
day, all year. They'd actually change

every few hours. So it was something that
took us a while to figure out.

Takes a sip of Mate
So, of course, the way this works is we

had two or three people supervising,
sorry, two or three robots for a

supervisor in Colombia, and we have just a
bunch of people. Typically, students who

would just be working part time and they
were sitting in an office in Colombia

doing this. Of course, the press found out
about this and they wrote a very small bit

of text in the article saying like, oh,
Kiwi hires Colombians and pays them two

dollars an hour. And people were really
frustrated about that. We had a lot of

interesting feedback about that. But what
was interesting to see is that this

technology actually helps people in
Colombia. If you're there, it's a third

world country it's a developing country.
You can get a job at a factory. You can

get a job at a textile shop, you can get a
job maybe McDonald's. But there aren't

that many tech jobs per say. The biggest
employer in the country is a phone support

company. So like when you call in to
support line, you get connected to

Colombia sometimes. And that's the biggest
employer in the country. So in order to

get like a tech job, it's really, really
hard and giving people the ability to, go

and supervise robots it's something that
helped them get something on their CV and

help them step up It helped them learn a
little bit more about the technology and

helped them progress in terms of their
careers. Our lead A.I. guy, he actually

started off as a supervisor and he went up
through the ranks and then he ended up

leading the A.I. and robotics team. So it
was really interesting, really inspiring

to see how that transition happened. And
we managed to get our technology to work

so well that we can do this.
Video of the inside of an airplane is shown

So we were able to get it to work with up to
eight seconds latency, which meant that

you can control it literally from anywhere
in the world. So even from an airplane

above the Pacific Ocean. So it was a
really interesting experience. And we

really try to make it simple. So in
conclusion, for A.I., we realized that the

best approach was to keep it simple. We
tried a lot, a lot of different

approaches, like we tried the traffic
light detection. We tried a yellow pad

detection. I didn't mention that. So in
Berkeley, you have these accessibility

ramps and you have yellow pads that blind
people can actually feel them and see them

easier. So we built the algorithm to
detect that and we thought, OK, maybe if

the robot is stuck in the middle of the
intersection, you can automatically detect

this yellow pattern and navigate to it.
It's an approach that worked in theory, in

practice it did not quite work. We tried
segmentation. So that was an approach that

worked OK. But some weird things broke it.
So for example, any lamp posts or bicycle

posts would crash the robot because it
didn't see it. So yeah, keeping it simple

was the best approach, really not going
too crazy. And the approach we ended up

going in the end was to have it more of
like a driver assist type, like a parallel

approach, parallel autonomy approach,
where our robots would help people the

same way that cars would help people stay
in lanes or have cruise control or like

with parking assistance. That's kind of
the approach we're having. I think long

term it is gonna be possible to build
robots more autonomous, it could be

Starship that have some interesting ideas
about how to solve that. But I don't think

it's quite something I could be scaled to
every city just yet. Another really

important thing is, the lab does not equal
the real world. So there are many, many

great examples of fantastic research
papers from some great groups and they

were great with very polished, very clean
datasets. But they did not work when you

deployed them on 100 robots, there were
all different. They all had slightly

different camera calibration that all had
slightly different hardware, it all had

slightly different chassis. It did not
really translate as well. So these

algorithms, these lab best case scenarios,
will need to be modified a little bit.

What else? Yeah, one thing, maybe jumping
back to the keep it simple. We decided to

put in a very simple safety mechanism. So
the robot actually breaks if it sees

something within 50 centimeters in front
of it. So as kind of like a last measure,

precaution, as you saw before, there is a
video like you can supervise the robot

from anywhere in the world, but a lot of
latency. But having this 50 centimeter

like hard break, actually saves us in case
the robot loses connectivity or the

supervisor is no longer able to supervise
the robot. So it's always breaking 50

centimeters away from any collision with
like a baby or a car or whatever. So the

approach we really thought about is, how
can we expand human potential? There is a

lot of talk about A.I. taking jobs or A.I.
replacing people's roles, but we sort of

kind of try to do that and it didn't work.
Like we try to build robots that were

fully autonomous that went from the
restaurant to your door and that didn't

work. People were waiting a very long
time. These robots required an obscene

amount of maintenance. So we ended up
going for an approach that was far more

parallel autonomy where these robots were
like helping people to do more. Same way

the supervisors are getting these
assistive technologies where they able to

set a waypoint to do the path finding and
the robot does the motion planning on

board. We also had the couriers who would
just load food into the robots instead of

the robots picking up food from the
restaurant directly, so really expanding

human potential. I think that's where it's
at. And over the course of the past

century, we've seen a lot of examples of
this. Like we've seen operators of

elevators. Like before, elevators had
operators who would make it go up or down.

And now they're fully automated. We had
switchboard operators who were there to

connect phone calls. Now we can make a
phone call to anywhere in the world

instantly for free. So we're seeing this
transformation of work and transformation

of the way things are done. And I think
this is just the start. The way I see

these robots is really meshing into the
fabric of our societies and solving

physical transportation. Like, sure, you
can move bits from anywhere to anywhere in

the world, but can you move atoms? It's
really expensive to do that. It's really

hard to do that. That's why I see robots
expanding human potential. So.

Conclusions. What we did was really cool
and I think it was a cool experience. One

thing that we realized is that tech isn't
the hardest part, right? We spent a lot of

time thinking how to build something, but
figuring what to build is sometimes very

important as well. And I don't think we
spent enough time asking ourselves that

question. We kind of went in all sorts of
directions we didn't focus as much on

making the best product possible. We kind
of tried things that were really weird and

not well thought out. So like having that
more long term thinking, like thinking

what should we build is very important
because like how, you can just look up a

tutorial on Google and figure out how to
build robots. It's not the end of the

world. One really important thing for us
was interaction. So interacting with

people, figuring out how to make the door
open, when you actually received your food

was super hard, super, super challenging
to do. Actually, the only robot that opens

the door for you. Other companies like
Starship, for example, they have a button

that unlocks a solenoid. So it's like the
experience is not quite there yet to bend

it down. You have to figure out how the
door actually opens. So we spent a lot of

time, a lot of effort in order to optimize
that experience to make it as smooth as

possible for people. Also, one thing we
didn't figure out is financing. I'll come

back to that in a second. That was really,
really hard to do as well. So like tech,

you know, not the hardest, financing
figuring out like how to manage cash flow,

super important. But I think the most
important thing is to work with a great

team. If you're going to be spending a lot
of time with people who you eat, live and

breathe with, it's really important to
choose a team that you really connect with

and then share the same passion as you do,
because you could be miserable making an

amazing amount of money, but if your with
a really crappy team with a high turnover,

it's really boring. I was really fortunate
to work with one of the best teams in the

world and over the course of the past two
and a half years we managed to do quite a

lot. And just last month we actually got
an article in The New York Times. So that

was a really big accomplishment for our
team and we got to share it with our

families. My mom was really proud. So a
lot of great traction and a lot of great

coverage. But unfortunately, we actually
ran out of money, so we kind of ran out of

money last month and we are no longer
delivering things. So I decided to leave

and start my own thing instead of doing
robots. I decided to do data. So now I'm

actually focusing more on building a tool
that helps you tell stories with data. So

this is Glint. This is a data storytelling
tool. You're able to drag in some files

and it tells you the story of your data
without you having to write any code. So

my hope for this is to allow anybody in
the world without any knowledge about how

to wrangle data, how to clean data, how to
analyze data, to be able to tell stories

with their data directly from their
computers. I'm imagining a tool where you

can say, oh, "In December there were X X
visitors to Congress" or "Last summer we

had X X sales" and automatically filled
out for you. That's kind of what I'm

thinking about. If you want to join the
effort, there is a Github. I'm more than

happy to have any contributors. And if you
have any questions or comments we're happy

to answer on Twitter or here in person.
Thank you.

Applause

H: So, as usual, feel free to line up in
front of the microphones or write your

question to the signal angel over there.
That already has one. Um, it's all the way

down. Go ahead.
Signal Angel: OK. Here is a user of your

service who apparently got an e-mail from
you that announced some changes. So he's

wondering, what's up? What you're going,
what you're planning to do there, whether

you're continuing your service or closing
shop?

S: Yeah, it's unclear. We ran out of
funds, so I think the CEO is still trying

to figure out what to do with that. I wish
him the best of luck, but I ended up

leaving with a lot of other people. So we
have like 50 people in November, now we

have like 10 people left in the country.
So it's very ambiguous what's happening,

but yeah, I left.
H: Yeah, microphone?

Audience Member 1: No audio. Okay, now it
works. I'm a little bit confused because

you are presenting a 1970s concept of a
manipulator, because a robot is something

that works by itself, a manipulator is
somebody who has some joysticks and moves

things. So it's nothing special. You just
have a interlinked Internet link for

manipulator and in the 70s, there were
cables. So what's the special thing?

S: Yeah, that's a good question. I think
the magic here is connecting everything

together, figuring out for us how to build
these robots, how to build a reliable

connection, and how about a platform that
works. And as I mentioned, like the how

that's not that interesting. It's more of
the what you build. It's that experience

where you're able to order anything you
want at any time and get it delivered in

under 30 minutes virtually for free. So
that's good.

AM1: So, so far, so good, but evil people
could just buy a remote control car, put a

bomb in it, drive under a police car and
make boom. And so it's the same use case.

You deliver something by remote control.
Audience Member 2: Yeah. You talked about

iterating quickly and rapidly and that's
very good model for conceptual stage and

software. Were you in the stage where you
were leasing your hardware with your

iterations? Because usually a thick stack
of certification has to come in between.

S:So I'm not entirely sure. Are you asking
if we got certified at every single

release?
AM2: I suppose. Yeah. What level of like

recertification was it totally released.
So you had to meet like regulations for

each iteration of that?
S: Yeah, absolutely. We didn't really get

certified because we're not building
hardware product for consumers. So we're

not selling it to anybody. We're operating
it ourselves. So we don't fit under the

same kind of requirements. However, we did
have to have some permits. And part of the

conditions that these permits was that we
had to meet some expectations. But they're

very, very basic. And there were rigid
like an FCC or a CE certification, for

example.
AM2: That was the question. Yeah. Thanks.

S: Thank you.
SA: Another question from the Internet.

"Why did you develop different
applications for Android and iOS?"

S: For the consumer application?
SA: I haven't got any more details.

S: We just did. I mean, we had first an
iOS application. I mean, 80 percent of our

customers are using iOS. So we really
spent a lot of effort like polishing that

iOS experience, making sure that worked.
And at one point, our Android app was

working super badly. So we decided to kill
it. And everybody was really, really

pissed off, extremely pissed off. So we
actually reintroduced it and we started

catching up with features to the iOS
version. Internally, all of our apps are

built in React and React native. So we had
a common framework for all of our internal

apps, but we didn't have that experience.
Where you're expecting the quality of

experience, that we're expecting from a
consumer app using React. That's why we

had two different code bases.
H: Have tried different methods regarding

perception? For example, lidar, radar and
what are your conclusions from that?

S: Yeah, we tried lidar, we tried the
cheap lidar we didn't try the really high

end lidar. So the challenge with having
like point clouds is that you have to

compute, spent a lot of time competing. We
were using a relatively low power device

and it was running from batteries. So we
didn't have the luxury of having like 10

GPUs is in the trunk of a car, for
example. So that was one approach. One

question. Another question is how much
does it cost? So lidars, they can cost ten

thousand, hundred thousand dollars. Our
bill of materials was 'round two and a

half thousand. The last versions are two
and a half thousand. So all of our sensors

were very minimal. In terms of what
sensors we tried? We tried a lot of

different sensors. We tried ultrasonic
sensors. We tried near field infrared

sensors. We tried other sensors. Yeah, we
tried a lot of different sensors. We are

ended up just going with cameras. So we
have cameras. We have six cameras onboard,

all of them full HD. We stitch them into
an image on our compute module and then

the supervisor decides which portion of
the image they want streams. They can

manipulate with the keyboard to see which
portion of the image is streamed. So we

don't stream the whole image. We just
stream a part of it. The really important

part for us was to make something that's
viable, that can be used commercially. I'm

sure lidar is really cool, but I'm not
seeing any commercial deployments of lidar

based autonomous vehicles or robots yet.
Audience Member 3: Thank you.

Audience Member 4: You've tried out many
different concepts how to do it. And you

saw that your company ran out of money. Do
you still believe in the business concept

of robots delivering packages of food?
S: Who knows? I think I think it was a

great learning experience. We learned a
lot. We had a great team. And I think

we'll see some concept of robots. Maybe
not exactly what we were building, maybe

something a little bit different, but I
think it's a little bit inevitable,

especially with the rise of self-driving
cars. Maybe we'll have cars delivering

packages instead of robots. Not entirely
sure what it would look like. I could tell

you, Amazon, they bought one of our
competitors dispatch labs. So they're

making a big bet on this. There are two
delivery companies in the US, Postmates

and DoorDash, that are building products
internally also for... with delivery

robots. And also companies like FedEx are
also building delivery robots. And then

we have companies as Starship, for
example, which are building robots and

doing B2B with companies all over the
world. So, yeah, I think we'll see some

form of delivery robots. I don't know if
it's going to be what we had or what

somebody else is going to have.
Audience Member 5: Were there any safety

certifications you had to satisfy in order
to operate around people?

S: No. So. laughter Well, the thing is,
in the US, like, it's kind of just do

whatever you want. It's very different
from Germany. You can kind of just do

things and you can do them until you get
in trouble. So we kind of had that

approach don't ask for permission, don't ask for
forgiveness. We ended up having to have a

permit in the cities we operate in. But it
was very simple. It was like, OK, you have

to have lights. You have to have a phone
number and you can not go in these areas.

That was essentially all the
authorization, all the permitting and

certification that we had.
H: Yeah?

Audience Member 6: I wanted to ask, did
you try other markets? Like, autonomous

driving is very hard, even way more
than... manage it fully. So like perhaps

elderly care, like you could use this
robots in elder care where you have a

controlled environment where everything is
the same. Did you search after other

markets where it's less...
S: Yeah, that's a great question. Yeah,

there is a lot of potential for markets
like elderly care, for example, also for

mail delivery, for applications inside of
factories. We had a couple different

medical companies that reached out to us
and like, hey, we want to move items, move

packages inside of our facilities. So we
did have a lot of interest. We tried to

keep a focus on the consumer space, like
really building a consumer experience that

worked out before branching out into these
more B2B approaches. Where elderly care

could be one of them. I think one
important thing about elderly care and

services like Meals on Wheels, for
example, is that human contact. So I think

people who are maybe not seeing as much of
their family, of their relatives, they

really cherish that connection they get
from people who deliver them food. So I

think it's a multifaceted approach they
have to have. You have a couple of

different considerations with these kind
of services for the elderly, for example.

AM 6: Thank you.
Audience Member 7: What kind of

personality do Chinese entrepreneurs have?
S: laughs I think, as I mentioned like,

it's really important to have
relationships. So they were very

interesting. They were very deeply in
belief of their government. They had

nothing bad to say about it. They believe
they would bring them everything... the

best possible, even though they still try
to access Facebook and Twitter with VPNs.

So they were very, very loyal
to their governments. They were very, very

diligent. If they committed to something,
they would usually deliver on that. They

really wanted to make sure you had a good
experience. And also what we saw, for

example, with building up these
relationships, like the first few times we

talk, they would try everything to impress
us. So we got taken to these ridiculously

expensive restaurants to make sure that we
were welcomed well and make sure

everything was right. I actually had an
interesting episode earlier this year. I

was going to go to Burning Man and then
all of a sudden one of my colleagues had

an argument with my manufacturer about
whether Hong Kong is another country or

not. And I ended up having to go to China
to deal with our manufacturer instead of

going to Burning Man to make sure we're
aligned in terms of our beliefs. So,

sometimes it's really delicate. You
cannot, like talk too much about the

government there. You can't talk too much
about politics. It's best to just stick to

business and, yeah, focus on building a
product.

H: I guess this was it. Thank you.
S: Thank you so much. Applause

subtitles created by c3subtitles.de
in the year 2020. Join, and help us!