Google I/O 2015 – Making apps context aware: Opportunities, tools, lessons and the future
Google I/O 2015 – Making apps context aware: Opportunities, tools, lessons and the future


WALEED KADOUS: Welcome. My name’s Waleed Kadous, and
with me is Marc Stogaitis. We’re members of the Android
Location and Context team, and we’re super excited to
be talking to you, today, about how to make your
apps context aware. And we’re going to explain why
we think now is the right time to do it, some of the tools
that are available to help you do it, some things
to watch out for, and, finally,
something I’m really excited to do with
you, today, which is to share a vision of
what the future will hold as far as this is concerned. Next slide. I want you to
imagine that, instead of presenting at
Google I/O 2015, I’m presenting at
Google I/O 2020. I want to describe to you
what my day so far has been like making it here
to go Google I/O 2020. I’m sleeping comfortably
in my bed in my house. And my phone alarm goes off. And it’s not that I
programmed my phone to go off at that
particular time, it just knew that was the
right time to wake me up. It knew I had a
late night before. It knows the traffic it
will take to get to Moscone. It knows that I like to
have showers in the morning. So all I said to it the
night before was wake me up in the morning. Now I pick up my
phone, and, of course, it’s automatically unlocked
because it knows exactly who I am, and it knew that it
spent the whole night next to the bed. But even before
I’ve got out of bed, the house has started
to come alive. My room has decided to warm
up to a comfy 72 degrees. The lights are slowly coming
up to welcome me into the day. And of course, the
coffee maker is going because who knows
where we’d be without coffee. As I sit down in
front of my breakfast, my tablet is there
waiting for me. As soon as I sit down in front
of it, the content for the day appears, all my events,
all the exciting talks I’m looking forward to at
Google I/O. And not just that, it’s suggested
some activities for me that it knows about
what I’m interested in. So for example, it
knows I love Asian art. It knows that I’m
in San Francisco. And it knows that
the Asian Art Museum is open late on Thursday
nights, which it is. You guys should go check it out. But it put all of
that together for me so that I can just see
the day ahead of me. And just as I’m finishing
checking out stuff, my housemate joins me. And he’s also enjoying
his morning coffee. And as we’re sitting
there, just as we’re finishing the cup
of coffee, we get a buzz that says that our car
is waiting for us downstairs. And sure enough,
we go downstairs, and waiting for us is
our self-driving car. We hop inside, excited about
where we’re going to go, today. And we notice that
the music comes on. And the music is
specially selected. It just happens to be the music
that both me and my housemate like. And to be frank, that level
of overlap is not so large. We’ll talk about
that another time. And then just as soon
as it gets to Moscone, I notice that my watch buzzes. And on my watch, I
notice that there’s a map that takes me
directly from the curbside in through the front
door, which of course I don’t need to swipe my badge
or anything because my watch already knows who I am,
and I can share that with the devices around me. And it takes me up the
stairs, turn left, turn right. And as I come to
the auditorium, I get a buzz from my friend
that says that she’s sitting in the audience. She’s coming from New York. I don’t get a chance
to see her very often. And my phone guides me directly
to where she’s sitting down. I sit down next to her. We get a chance to catch up. And we’re ready to take
in a day full of knowledge and exciting
information about what’s happening at Google I/O 2020. Now the question
in all of this is, that sounds a little bit
sci-fi, a little bit crazy, but I want you to
disassemble that. And I want you to think
about the interactions between the device and the
user as this whole experience happened. I want you to think about how
many unnecessary interactions were avoided. I want you to think
about how many times the system reminded
me, in context, when it was useful to me; and
how it brought new knowledge to my fingertips that I
didn’t really even ask for, but I knew that it
was very useful; and how it was able
to enrich my life, by letting me sleep in
for the maximum amount by making me feel more
comfortable in the morning instead of a rude awakening. So I really think that
those components there that we mentioned, that type
of simplification of a person’s life, that simplification of
interactions, those things we can build, today– there’s
no obstacle to us doing that. And what I’m going to
try to convince you of is that, not only
is it possible, we already have some
applications that we’ve built, and some other
people have built, that show these type of
contextual experiences. And let me explain to
you why I think now is the perfect time to do it. We have in our pockets
an amazing platform. Sometimes we’re so involved
in the day to day of things, we don’t take a step
back to just think about how amazingly
powerful our platforms are. So inside your
phone is a processor that can process 8 billion
computations a second. Not only that, but
it’s full of sensors. It’s got accelerometers, and
gyroscopes, and barometer, and magnetometers. Some phones even have
humidity sensors. But that’s never really
happened before– that we’ve had that level of
sensors available with us in that way in an
always-on fashion. And we can use those sensors,
like the accelerometer, to determine things
about the user’s context, like if they’re running,
or walking, or driving, or cycling. There’s also other
rich sensors on there, like the camera and the
microphone, that can pick up cues that we can use. And combine that with the
huge variety of techniques available for
communication, like cell, and Bluetooth, and
Wi-Fi, and GPS. And GPS, when you think about
it, it’s kind of a little bit crazy. It allows us to talk
to a satellite that’s 20,000 kilometers away
to work out where we are. So we have to ask ourselves,
have we really taken full advantage of this platform
in terms of really relieving the pressure off the
user in terms of memory, in terms of simplifying
their lives, and easing the interaction? And I think the answer
is that, right now, we have this very, very large
opportunity ahead of us. Basically, in the
history of computing, we’ve never had this
combination of sensors, communication, and
computing power. And we can use that power,
those sensors, that ability to talk to devices nearby, to
understand the user’s context. I think all of you can
think back to the keynote and what [? Aparna ?] was
telling us about Google Now as an example of this. And we can use this idea
of understanding the user’s context to simplify
interactions, to augment human
memory, and knowledge, and also, to make people’s
lives just a little bit better. When I think about
simpler interaction, I just think back to the
broken model we have. One of our primary means of
interactions with devices is that we took a 19th
century technology, which is the typewriter, which
was used to bang letters onto a page, and we
shrunk it down, and tried to get it into a phone. And that’s the
metaphor that we use. There’s got to be a better way. And we should be able to use
the sensors on the device to help take that load to use
those keyboards off the user. So where we’re going
to go from here is we want to walk you through
our experiences building these types of systems. But this technology
is still very new. So there’s a couple
of things– in fact, to be exact, three
things that you need to pay very
careful attention to. And finally, I want to map
out what the future might look like between
here and 2020– that Google I/O talk that
I hope to see all of you at in the future and be
here, presenting again. And with that, I’ll
hand over to Marc, who will walk us through
some of these experiences. MARC STOGAITIS: Thanks, Waleed. So Waleed described the
power of modern cellphones. And he also set the stage for
what 2020 might look like. What I’m hoping to do is to
show you what’s possible, today. I think a good example is
something that happened to me a couple of months ago. As I was driving
to a restaurant, I used Google Maps
to help me navigate. And after I parked, I
walked over to the building. And just as I walked
in, I realized that I didn’t pay attention
to where I parked my car. That’s a terrible feeling
that I’m sure some of you have experienced before. Luckily, I was
able to swipe over to Google Now, which had a card
indicating the last location of where I parked my car. I think that’s a great
example of the type of contextual experiences
that can be built, today. And I’d like to show
you how to build these. So usually context is broken
down into three layers. We start by taking
sensor data, like that of an
accelerometer or Wi-Fi, and then build algorithms on top
of these, which take raw sensor data, and turn it into
higher level signals, like location or user activity. And finally, we
use those signals to build features in apps. So let’s look at these
three layers in a little bit more details. Context starts with sensors. Sensors are what allow
our devices to understand the world around them. They allow them to break
out of the digital world and into the physical world, the
world that you and I live in. And I think a good way
to think about context and sensors in general is
to think about human senses, since we’re really
good at understanding the environment around us. And building up the
strong mental model of how sensors work is important
because a lot of context is built on top of these. So something that’s
helped me with this– it’s also a little
bit of fun– is to ask myself, what if I
had Android sensors instead of human senses. How would I perceive
the world around me? Well, I’d see with a camera, a
light, and a proximity sensor. I’d hear with a microphone. And I’d keep my balance with an
accelerometer and a gyroscope. And I would use these sensors to
understand the world around me and make decisions
based on what I see. As a human, I’m pretty lucky. My brain takes care of all
this sensor processing for me. But how could we
do it on Android? Well, algorithms need to be
built on top of raw sensor data to make sense of what’s
happening in the real world. And there are many
challenges here. We have to come up with ways
of fusing noisy, and sometimes conflicting, sensor data. We have to put together
large data sets that allow us to train and fine tune models. We also have to come up with
ways of enabling and disabling different sensors so that we
can manage power consumption. Luckily, we’ve tried to make
this as easy as possible. So we’ve built several
Android services that take raw sensor
data and turn it into contextual information. These include the Activity
Recognition, Places, Geofencing, the the Fused
Location Provider APIs. I’ll talk about these in
a little bit more details in just a little while. So the key takeaway
is that these allow you to receive context
information in your apps with just a few lines of code. All the processing is
handled under the hood so you get to focus on what
really matters, which is a user experience in your apps. A context can sometimes
be a little bit abstract, so my goal for today is to make
it as concrete as possible. And to do that,
I’ll try something we haven’t done before,
which is to take launched Google products
that make use of context and show you a behind the
scenes look at how they work, so which APIs they use,
how those APIs work, even which sensors they use. Hopefully, this will help
illustrate what’s possible, today. So let’s start with Smart Lock. When I went home for
Christmas, my mom asked me to enable a
password on her device. She saw that I was using a
swipe to unlock my phone. And after about a
couple of months, she emailed me and said,
how do I disable this. Basically, she found
that having to do this every single time she
needed to use her device was just too cumbersome. So everyone will weigh the
pros and cons of security versus convenience a
little bit differently. But this exposed an
underlying problem. When it comes to
unlocking your device, there are only
really two options. You either disable
security completely, or you enable it and have to
perform some kind of action every single time
you use your phone. So is there something– can we
sensors to make this better? It turns out there’s a
lot that can be done here. And one example is
on-body detection. So the idea behind
this is pretty simple. If you’re using your phone,
and you put in your pocket, and you walk around for a while,
when you take it out again, it should still be unlocked. Your phone should
be smart enough to know it’s been in your
continuous possession. So how do we make
something like that work? Well, it starts with
accelerometer data. So as you see here, on the
left is the accelerometer coming from a phone
that’s in a user’s pocket. On the right is a phone
that’s just on a desk, not on the user. So you can visually tell that
they’re pretty different. Even if a person is
sitting fairly still, there will still
be micro-motions that can be detected over time. But is this enough
to build something, like on-body detection? Well, not quite. When it comes to context, there
are usually many real world variations to deal with. One of the problems
that came up was that, let’s say you’re
sitting in a cab and your phone slides
out of your pocket, we need to know that
it’s not on you anymore so that we can lock it. But the Excel that comes out
of a phone that’s in the car looks extremely different
than that of a phone that’s just sitting on a desk. So we needed something
that could tell us that the phone was in a
vehicle so that we could train situation specific models. In comes the Activity
Recognition API. And it’s the first context
API I’d like to introduce. It works by fusing
sensor data to detect the user’s current activity. The way it works– we took
some accelerometer data and used machine learning
on top of that data to try and predict the
user’s current activity. We’ve also added new
signals recently, like the use of Bluetooth,
and trained new models to make the
detections run faster and also improve the accuracy. So let’s look a little bit at
how it works behind the scenes. So what we did is we asked
Google employees to record over 65,000 sensor traces
where they would label the activity
they were doing, as well as the carry position. So they’d say
something like, I’m on a motorcycle and the
phone’s in my backpack, and label and
gather sensor data. And we then extracted
features from that sensor data and trained machine
learning models that could predict the
user’s current activity. After that, we took
those models and made them run directly on the phone
without needing any server interaction. This saves battery,
bandwidth, and also makes the detections run faster. So this is kind of fun. So Kevin, an
engineer on our team, went out and recorded
sensor data along with videos of
what he was doing. And the goals is to basically
look at the sensor data and try and figure out was he
walking, was he on a bicycle, or was in a vehicle. And if you look
at this one– this is coming from the accelerometer
data from his phone. So you can see there’s kind
of a couple of big spikes that happen. Those are usually an
indication that somebody’s moving their legs. Either they’re walking, and
each time they hit the ground, there’s a big spike that
happens or maybe they’re biking. Each time they cycle
around, there’s a big spike that happens. If you add another sensor,
like the watch accelerometer, all we see here are
a lot of vibrations. We don’t really
see any big spikes, like you’d expect to see
if somebody was walking. So these types of
vibrations, you’d usually expect to
see if somebody’s holding on to the steering
wheel of a bicycle and just kind of feeling
the bumps in the road. So let’s take a look
at what he was doing. Well, he was, in fact, biking. So this goes to show that, if
you use accelerometer data, it’s a pretty good
indication as to what the user’s activity might be. Let’s take a look
at another one. So this one– it looks
a lot like walking. The top graph, which is
coming from his phone, has a lot of these spikes. They’re pretty periodic. And the watch also has very
similarly timed spikes. So he’s basically walking. That’s what it looks like. So that’d be my first
guess as the activity. But if we add one more
sensor, which is a barometer. Things get pretty interesting. So a barometer senses
atmospheric pressure. And basically, the
way it works is the lower you go down in
altitude, the more atmosphere there’s on top of you, and
the more pressure your phone will sense. It’s a little bit like
if you go underwater. The deeper you go underwater,
the more pressure you’ll feel. The same thing happens here,
and your phone can sense that. So here, there’s a
continuous upward spike in the barometer, which
means the user is actually decreasing in elevation. So we have somebody who’s
decreasing in elevation and also doing
something like walking. So what activity is he doing? Well, he’s going
down some stairs. So I think it’s a
good example of, if you add another sensor,
like the barometer, you can better detect what’s
happening in the real world. Let’s look at one last one. So Kevin, when he
showed me this one, I had no idea what this was. I’ve never seen something
quite like this. So he told me, well, OK
I’ll give you a hint. I was playing a game
and I had a gamepad that I was playing with. So, OK, well, the
first part of this looks a little bit like running. But then there are
these huge spikes that happen in the middle
graph that I’ve just never seen before. Let’s look at what he was doing. He’s playing on a gamepad. And then, oh, he
starts using his first. Now, it makes sense. He’s cheating. If you’ve ever played
this game before, you’ve probably
tried that yourself. But it’s a good example
of why we introduced an activity called Unknown. This basically allows
are classifiers to say, you know what, this
kind of sensor data, I’ve just never seen it before. And I don’t want to
make the wrong thing and say, oh, you’re
biking or you’re running. So I’m just going
to guess, Unknown. So in summary, on-body
detection works by taking accelerometer
data, as well as the Activity Recognition API, to detect
that a phone was in a user’s continuous possession. It saves users over
20 unlocks per day. But can we save
even more unlocks by using other context signals? Well, the answer is yes. So for me, personally,
if I’m at home, I’m comfortable having
my device stay unlocked. But if I go to an
airport, I want it to lock in case I lose it. So different places
have different levels of trust for different people. And the Trusted
Places feature allows you to keep your device unlocked
in places that you trust. So let’s see how it works. Well, it makes use of the second
context API I’d like introduce, which is the Places API. This API makes it really
simple to get information about the places
around your users. And also, it
contains a component called a Place
Picker, which makes it easy to launch an interactive
map that allows your user to select a nearby place. So let’s take a look
at a little bit of code to see how it works. Well, to use the
Place Picker API, you simply create a
PlacePicker Builder. And then, you call
startActivityForResult. When the user has selected
a place, you’ll get a callback with the
details of the place, like its name and address. It’s really that simple. And the other API used by
this Trusted Places feature is called the Geofencing API. This API fuses location
as well as activity to detect and to give your apps
a callback whenever it leaves or enters a particular region. It allows your apps
to respond to changes in the user’s location while
keeping battery consumption to a minimum. For this feature, it was
used to unlock my device just as I arrived home, and
lock it again when I leave. So to use it, you set a
circular region around an area. And then you decide if you
want to be told about entries or exits of a particular area. We also added a new
transition type called Dwell. This allows you to say, I
want to be notified only when the user’s been inside of
the geofence for a specified amount of time. It makes it easier to
distinguish between the case where somebody walks
by your geofence or actually stops at the place
that you’re interested in. So in summary, we have the
Trusted Places feature, which uses the Place Picker,
as well as Geofencing, to make sure to unlock
your device in places that you trust. And we had this feature
which uses a lot of very kind of location based. And we had the on-off
body detection, which is using
accelerometers quite a bit. What happens if we use
these two sensors together? We could build a feature
called Where Did I Park My Car, which is
one I mentioned earlier. What I love about it is that
it’s completely automatic. I don’t have to remember to
take my phone out of my pocket and press a button every single
time I get out of my car. So how does it work? It makes use of the Fused
Location Provider API, which is an Android service
that uses Wi-Fi, cell, GPS, accelerometers, gyroscopes,
magnetometers– so basically a huge, huge amount of
sensors to try and figure out the location of your device
if it’s indoors or outdoors. And it also provides multiple
controls for developers to allow you to tune for higher
accuracy or better battery consumption. So let’s take a
look at how it was used as part of this feature. Well, we started with the
Activity Recognition API and looked for whether or not
the user was in a vehicle. Once you detect the
user’s in a vehicle, then register for the
Android Tilt Sensor. This is a sensor that can
really, at super low latency, detect that a user went
from sitting to standing. And it helps reduce
the amount of time it takes to detect that a
user came out of a vehicle. And then finally, once
we detect the user just got out of a vehicle, we use
the Fused Location Provider API. And then, there you
have it, the location of where you parked your
car, automatically detected. I’m a huge fan of the
quantified self idea. And that’s why I love
the Google Fit app. It tracks my steps. It tracks my bike rides,
as well as my runs. And it does this
completely automatically. I don’t have to remember to push
a button every time I go out for a bike ride. And to do this, it
makes use of many APIs. It makes us of the Activity
Recognition API, the Fused Location Provider API,
and even the Sensors API to track things like heart rate. And after using it for
about a couple of months, I noticed that, on
weekends, I would routinely miss my step goal. So I thought this was a really
interesting insight for me that allowed me to adjust my routine. And once I did, I noticed
a significant improvement in my skating stride
whenever I played hockey. So I thought that was great. But now, after my
legs were super strong from all that hard
walking, there was a new limiting factor
for me, which were my lungs. They would just burn
after a really hard state. That kind of high
level of intensity just isn’t reached by walking. So we asked ourselves,
what could we do to make fitness a bit better? Well, new devices bring
up new possibilities. Android Wear is
great for context, since it’s packed with sensors,
it’s always on the user, and it’s at a known,
fixed location. So we decided to
study sensor data to see if we could detect
some of these commonly known activities, like push
ups, sit ups, and squats, which have a really
high level of intensity. Well, it turns out it’s doable. So here’s an accelerometer
graph of what doing a squat looks like. So each time the
user does a squat, you can see this huge
spike that happens. And basically, we can write
algorithms on top of this to detect it. So I love starting at
accelerometer graphs, and I could do it all day. But I think a live demo would be
a little bit more interesting. So please welcome, Barbara,
the Engineering Lead of this project onto the stage
to show us this in action. [APPLAUSE] BARBARA WANG: Thanks, Marc. MARC STOGAITIS: Could you switch
over to the [INAUDIBLE] vision, please. BARBARA WANG: So today, I’m
going to do some work out on the stage, and show you
how Activity Recognition can automatically detect what
work out you’re doing and also how many
reps you’ve done. First, I’m going
to do some squats. And while I’m doing
it, you should be able to see the
count on the big screen. One, two, three, four, five. Cool. As you can see, I
did five squats. And while I was doing
squats, the count incremented one by one. The count you see here
is synced from my watch directly over Bluetooth. So the detection is all
happening on the watch. You may also notice, we only
started showing the count after I’ve done three squats. This is because we
want to differentiate between a real squat
versus a random arm swing. The accuracy for
squats is pretty great. Now, let’s take a
look at sit ups. WALEED KADOUS: You want
to move back a bit? BARBARA WANG: One,
two, three, four, five. [APPLAUSE] Awesome. Thanks. Back to Marc. MARC STOGAITIS: Thanks, Barbara. Can we switch back
over to the slides? Thank you. That was a really
cool demo, by the way. So we saw how Smart
Lock, Find My Car, and the Google Fit app
make use of context to improve their app experience. Now these are Google
specific use cases, which were useful to use as
examples since we could really dive into the details. But what I’d like to
leave you with now are a few examples of
how you can use context in your own apps. Context can be used to simplify
interactions with the device. So Waleed mentioned
earlier how keyboards are really small
and hard to type on. One example would be that,
if you have a chat app, you can automatically fill
in the name of a nearby place as a user is typing to
save them some keystrokes. You can also write a
feature that would simplify the output of the device. So if a user’s driving,
you could automatically read text messages
out loud to make driving a little bit safer. Slides are being a bit slow. So you can also do
things like– OK. We’ll go without the slides. Other ways of
simplifying interactions would be if, say, your
user arrives home, you could automatically
turned on the lights. You can also use it
to simplify sharing. So one example would be that,
if you have a text messaging app, as your user
is driving, you could let other users
know that they’re driving by putting a
little icon that says, this user is currently driving. And then it can save them from
having send out a text message when that user’s already busy. It can also be used
to– let’s say, your user just
went out for a run. You could say, all
right, let me fill out a post with the distance
that you’ve traveled. That way you don’t
have to actually go in and, say, OK, I just
ran 10 kilometers. So you make sharing
a little bit easier. And another example would
be an activity leaderboard. So something that I like a
lot from some of these fitness products is that they me
here’s how much you’ve walked versus other users. You could do a lot more of that. For bike rides,
you could show this is how many miles your
friends have biked versus you. And basically, keep things a
little bit more interesting in terms of challenging
for your friends. You can also use it
for use cases that are related to entertainment. So let’s say that
I go out for a run. Well, if I hit a really
tough part of a run, maybe you could play the
song, Eye of the Tiger, to keep things interesting. It can also be used for games. So for example, you
could write a game that’s a treasure hunt based
on activity and location. We’re getting stage
directions as to whether or not we’ll
get the slides back. It’s OK, this part is
actually pretty non-visual. The next part, I think
we’ll need the slides back. So other entertainment
use cases. For example, you could
do movie recommendations. So if I go out and
I go to a ballpark. And then, I get home,
you could recommend hey, you might want to watch
the movie, Field of Dreams. And it can also be used in
quantified self use cases. So I mentioned
how the Google Fit app was great for
quantified self, especially around fitness. I think there’s a lot more
that can be done here. For example, you can show
users this is how much time you spend on commuting. And here’s how much
it was this week. Here’s how much
it was last week. Hey, we’re back. All right. Here’s a quantified
self use case. So basically, you
show your users, this is how much you’ve
done in terms of commuting this month and last month. And then, show, hey, maybe if
you left a little bit early on this day or this
day, you’d actually be able to save extra time. You can also show it
for restaurant visits. So let’s say you
show me a summary of this month how many times
I’ve visited restaurants versus last month, I can
use that to compare or maybe optimize my life a little bit. And finally, it can be used
to explore and discover the world around your users,
like showing nearby events, or even, just as I walk
up to a restaurant, you can show me this is the
menu for this restaurant, here are the best items
so that I can compare. And of course, the best
examples will come from you. So there you have it. I hope that going
through these examples has helped to illustrate the
type of contextual experience that could be built, today. And I look forward to
seeing these in your apps. Now, Waleed will speak to
the challenges of building contextual experiences. WALEED KADOUS: Thank you, Marc. So it’s important to realize
that these technologies and the ideas that
Marc presented to us are really in their infancy. So at a very critical
stage, you can really build some awesome
stuff, but you need to pay very close
attention to a couple of things because the
technology is so new. It’s not like, say,
graphical user interfaces that have been around
for quite a while. So it’s really important
that you pay attention to a few things. And those three things are
user trust– in other words, you need to establish a
relationship with your user to help them understand
how you’re using sensors and what the application
is doing for them. Secondly, recovering gracefully
when things don’t go right. And finally, and perhaps
most challengingly in the current
age, battery life. So let’s talk about user trust. Really here, there
are two key rules. The first one is to
collect only what you use. It’s so tempting
with the huge range of sensors available to
say something like, give me everything you got. But really, what that leads to
is it leads to user confusion. And we’ve seen that ourselves. We’ve seen flashlight
apps that require fine grained location, which
doesn’t make any sense at all. And the other really
important principle is to explain how
you’re using the context to benefit the user. And generally,
what we’ve seen is, if you can present to the user
the benefits of sharing these like GPS and accelerometer with
you, then they will understand, and they will come with
you on the journey, as you launch the app. So let’s go through
some examples that we have in our
own products in Google. The first one on the left
shows the trusted places. So what happens is
the first time that trusted places triggers,
after you’ve installed it and enabled it, is it will
have an interstitial here that explains, by the way, we’re
unlocked right now because we knew where you were. You were in a particular place. And now, we’ve
communicated to the user how we’re using location
to make their lives better. The second example comes
from the on-body detection that we mentioned earlier. And that’s really about
explaining to users how we’re using the accelerometer. And we have a short explanation
with cute little graphics that say, hey, this
is what we’re doing. And we’ve really found that
this helps users to understand and come along with
us on the journey as we make apps more contextual. The second principle
that we have to adhere to is really recovering gracefully. Now as Marc showed you,
these are all machine learning algorithms. No machine learning
algorithm is perfect. So for example, if you’re
driving really slowly, it’s going to be very hard
to tell that from cycling. For example, if you think
about choosing a place, maybe we won’t get you at
exactly the right restaurant, but we’ll get you next door. So we have to
design with the idea and change the user
interface and interaction to have capability for that. So for example, when
you look at Google Fit, although we automatically detect
the start and end of things, it’s really important to provide
the user a way to correct that. Or think about the Place Picker. When you look at
the Place Picker and we’re giving people a bunch
of options to choose from, ideally we wouldn’t even
give them the option, right? We’d just say, you’re
definitely at this place. But what we do is we offer
a couple of other options so that the user can choose
the correct one in case we didn’t quite get it
right, but we were close. And then the third
example is to think back to on-off-body detection
and trusted places, and really contemplate
what we did there. What happens if we get it wrong? What happens if we think
that the phone was on a desk, but was actually
in their pocket. Well, what might happen
is that we might ask them to unlock one more time. And is that so bad? At least we’ve had
the benefit of being able to reduce the
number unlocks that they have to do on a regular basis. This is perhaps the
most challenging thing that you will face as you try to
make your apps more contextual. And I think it’s important
to us to recognize the difference between the
way that these contextual apps work and normal apps work. Most of the apps we develop,
but not all of them, are basically
foreground applications. And when you’re a
foreground application, what you do as far as
power doesn’t really matter so much
because the screen’s going to be the dominant factor
in your power consumption. And so what that means– and
in addition, most of our apps, maybe, the user used
it for 10 minutes a day or 15 minutes a day. It’s not like they’re using
it for 15 hours a day. And in fact, if you
look at the statistics, the average phone is
on for 4% of the time. Now we have to switch
our mental model. And we have to think of,
instead of these huge spikes for short intervals,
we have to think about sipping a little
bit of power all the time. And that means we have to
adopt some new metaphors and ways of thinking
about things and, in fact, change the way the hardware
is designed to facilitate that particular use case. The great thing with Android
is just how flexible it is. It’s an amazingly
flexible system. And you can do things on
Android that aren’t even possible on other platforms. But it also means that that
gives you a lot of rope to hang yourself with. So you have to be
super, super careful in how you use these things. So what we’ve tried to do with
the APIs that we’ve provided is that we’ve tried to give
you powerful abstractions that aren’t very power hungry. And that gives you
a lot of capability to address the common cases. But as I mentioned,
the beauty of Android is just its flexibility. And you can always
go off and build your own contextual experiences. So just to give you an idea
of what we do in our labs, we’re designing APIs for
the entire ecosystem. So we put a little bit
of effort into this. This shows our power
regression testing lab. And what you can see there
is a bunch of [? monsoons ?]. Those are devices for measuring
power very accurately. And what we do is, every time
we make a change to the code that we make available to you,
we run it against these devices to make sure there
are no regressions. Now again, I want to
emphasize that you can always do your own thing. But you do need to pay
attention to power because it’s one of the greatest challenges. So if you want to
get power right, there are two key
principles that we’ve seen. The first one is you have
to make the hardware do the heavy lifting for you. And we’ll talk a little bit
about the design patterns, the subcategories there. And the second thing
is to really think about what does your
application really need in terms of quality. So to really understand
making the hardware do the hard work for you, we
need to step in and understand a little bit about how
things used to happen and what happens now. So traditionally
the model in Android was that you had
sensors that themselves might be quite power efficient,
like a accelerometer, it’s pretty power efficient. But your application
processor that actually does those 8 billion
calculations, and renders the screen, and
makes your game run, and gives you no jank, and
all of that kind of stuff, that thing is actually
pretty power hungry. In fact, it consumes a very,
very substantial amount of power. So the game here is to keep
that application processor on for as little
time as possible. Now the issue is that if
you have a sensor, even if the sensor is low
power– let’s take you back to your college days and draw
one of these sequence diagrams. What happens is that the
sensor sends the update to the. processor. The processor sends
it to your app. So each sensor measurement
you get– and you might be getting 25 or 100
sensor measurements a second– the processor has to
stay awake, and it never gets a chance to go to sleep. So what we did is we
changed the design of the internals of Android. And we said, what if we
can put some low power hardware in between the sensors
and the application process. This will give us a
mechanism to really get to low power possibilities. So effectively, what we
have is something like this. You send the sensor data to this
little low power thing, which we will call a hub for now,
and instead of it waking up the main application
processor, it’s going to bunch together
a bunch of measurements, send them as one package
to the processor, and the package then
delivers that to your app. In the case of
accelerometer data, we’ve seen this cut
our power by 80%. So using this feature is
very, very power advantageous. And in fact, this
is what we rely on for Activity
Recognition to make it even more power efficient. Now we already put
this low power hardware box in the middle of it. But what if, instead
of just gathering up a bunch of measurements and
shipping them off in one go, we actually put more
intelligence on there, and ask that the hardware
check for conditions to be true or false,
and actually start to put programs in there. Then what’s possible is
that you could actually check for the Geofences
in the hardware. And some of the Android
devices in your pockets actually do hardware
Geofencing, which is much more power efficient
than software Geofencing. And think back to
the keynote, where we were talking about Doze mode. Doze mode is built on something
called significant motion, which watches the
accelerometer all the time to see if the
position of the device might have changed
in a significant way. And by using that
sensor, we were able to really extend the
battery life of the device. And it’s really
enabled us to build some very impressive features. So we talked before
about design patterns and making the hardware
do the heavy lifting, but the other thing
to think about really carefully is what does your
application really need. And I want you to think
about a weather app and an indoor navigation app. Clearly, for a weather app, I
don’t need the most accurate location. I just need city level
location to get what I want. The Fused Location
Provider gives you options as far as power,
high quality, balanced, and low power. And in this case, we
would choose low power. But compare that to an
indoor navigation app. The user’s looking
right at the device. I want the highest
quality I possibly can. So then, I would
ask for high power. And not only does this
apply to accuracy, but also to timeliness. For example, if we have
the same situation, do I really care if I show
the city for five minutes or 10 minutes after
I’ve left that city and moved on to, say, Berkeley
instead of San Francisco? We think we can
probably tolerate that. But if you show
the user’s location 50 seconds after
they were there, that’s going to be a
pretty horrible experience. So it’s really about you
telling the system what you need in terms of features. So those were the difficulties
we have to overcome. But what I’m really
excited to share with you in our last
few minutes together is what I think is going
to happen in the future because I really think we’re
on a good track to 2020. So there’s, of course,
going to be a large range of incremental improvements. Location will get more accurate. And the range of activities we
detect is going to get broader. We’re going to detect
going upstairs, and downstairs, and up
elevators, and maybe even more extreme things, like
playing golf or cooking dinner. But there’s also three
critical step functions that I want to
outline to you folks. The first of those is
improvements to location. The second one is moving
to a multi-device world. I think you remember
Sundar’s conversation today about Brillo, and
the Internet of Things, and finally opening
up that low power subsystem that we’ve
just been talking about. So let’s dive right in. So I really think that location
will get very, very much better in the coming while. I think one of the biggest
weaknesses with location right now is it’s not
available indoors. And I think we have some
ways we can improve that. What I want to highlight
is– in the next slide, I’m going to show you a video. This is not close to production. It’s way too power hungry,
and too flaky, and too lab-like to [INAUDIBLE] launch. But I wanted to
show you what we’ve been working on in our lab. What you can see here is a
diagram of one of our labs. And the green line
shows ground truth. And the blue dot
shows what we’re able to compute in
terms of location. And the most impressive
thing about this is we don’t use any sensors that
aren’t already on your device, today. You can see it’s tracking
pretty bloody accurately. It’s about 2 or 3 meters off. And that’s pretty impressive. And we’re really
excited about working to make this a reality
in most people’s devices. Imagine what you could
do in terms of Geofences with something this accurate. The second point we talked about
is multi-device, the Internet of Things, and Brillo. So we’ve already started to talk
about what you can do on Wear and what you can
do on your phone. But what happens when you add
Auto, and Nest, and Brillo into the mix. Two really interesting
things happen. Number one, the ways that
you can sense the environment can change. And that means that our
ability to understand context becomes much more richer, and
we can do a much better job. Secondly, you have a
much greater surface area to surface the benefits of
location and other contextual signals. So imagine, for example, that
you say, turn on the lights. Well, we can answer
that question now because what you can do is
you can know exactly which lights to turn on because
we know your location very accurately. So that’s a really
exciting thing. The final thing that
I want to talk about is opening up that flexible,
always-on computing platform. So through this
talk, we’ve talked about this analogy between the
human brain and human sensors. And I really want to talk
a little bit about what we might be able to do. So historically, what
happens with the human brain is we have the cerebrum
and the cerebellum. And they’re our
powerhouse processors. That’s what we use to
understand language and what you’re using right now
to understand what I’m saying. But there’s this other part
of the brain that many of us are not aware of
called the brainstem. And the brainstem
is the thing that makes sure that you’re breathing
when you’re asleep and wakes you up when there’s a
loud noise, like an alarm, and wakes you up when the lights
are bright, and wakes you up when you fall out of bed. That system is always running,
even if you’re sleeping. This should look familiar
to you because it’s exactly the same diagram that
I mentioned earlier. You have this low
power subsystem that gives you the capability
to process signals, and it’s always on. But what’s incredible
to me is just how capable these low power
subsystems are becoming. So what I did is I compared
what’s called a sensor hub, and in this particular case,
it has the romantic name STM32F411. This is just one example
of one of these devices that can provide this low
power hybrid capabilities. There’s lots of
different designs. Sometimes, it’s in
the sensors, sometimes it’s in the same piece of
silicon as the main application processor. But what I did is
I wanted to compare with a Pentium of 20 years ago. Now believe it or not, the
Pentium of 20 years ago has about the same computing
power as one of these low power hardware processors. I got to remind you that
20 years ago, we still had the internet,
we still sent email, we still browsed the web, we
still put together documents. And so this chip,
at 100 MIPS, was able to provide a pretty
good user experience. Now we have that
same amount of power just to process the
sensor measurements, which is really mind blowing to me. And then the other
thing to think about is just how much
lower power they are. Let’s imagine that I
could shoehorn a Pentium into your phone. Do you know how long would take
to [INAUDIBLE] your battery? You do the math and it’s
like 30 minutes max. But you take one of these
STM32F411– that’s a mouthful. And you plunk it in there. It’s so low power that you
can basically just keep it running all the time. And if it was the
only thing running on the phone in typical
mode, your phone battery life would last for 60 days. Now I want to remind you
again, I think we all know what Moore’s
Law is, but it’s just fun to look at this again. So this is the Intel
Pentium from 20 years ago. And this is STM32F411 that’s in
some of your phones right now. And it’s crazy how the
difference in size works. But here’s what I’m
super excited about. What happens when, instead of
this processor being a fixed function that the developer
who makes the phone gets to decide
what runs on there, and we open it up
to third parties and allow us to develop
the code in a dynamic way. Think about what happened
with GPUs in the ’90s when, all of a sudden, instead
of having a fixed pipeline, you could do flexible
things and render really interesting things. What if we connected, not just
accelerometers, and gyros, and compasses, but
entire systems so that even when the phone
is in low power mode, it can communicate with
nearby devices using Bluetooth low energy. This is something that I’m
incredibly excited about. And just to convey the power
of what this idea means, let’s look at some
concrete examples. Suddenly, think about
it– this whole discussion that I’m having with you
about lower power would almost become irrelevant
because you now have a platform to make things
really, really low power. And what you could do is you
could take that indoor location system and, even in
a deployed phone, make it run at low enough that
you could have it always on. Not only that, but you
could even detect sensors the same way that we
talked about the brainstem. You could detect falls like
when you fall out of bed. You could detect earthquakes. You could detect
even car crashes, and be able to
offer the user help. Now that’s in the phone. But when we come to wear, it
becomes even more incredible because what we can
do is– you might be able to walk up to a
door and just knock on it, and your phone would
detect your knock. And it would talk to the device
of [INAUDIBLE] [INAUDIBLE] and unlock it magically. You could even do weird things. Like imagine that you’re into
juggling, any time you could juggle, we could detect
that, and give you statistics about how often you
juggle and something like that. And perhaps, just to twist
your brain a little bit, one of the really interesting
implications for me, as an old-time RPG
guy, is that we could build a location based
game where you literally cast spells. So as I do things like this
or like some other symbol, we could build a game around
recognizing when you make those symbols in real life. It really does open up
the exciting possibilities to be able to access
that platform. And with that, I’d really like
to thank you for your attention today. I think we’re at a unique part
of the history of computing now with these ability to combine
the sensing, the computing, and communications ability. And we can use these to
simplify users’ interactions with the device. We’ve added a
bunch of rich APIs. We hope that you use them. We can’t wait to see what
you folks will build to make your apps context aware. Finally, Marc and I
will be standing outside to answer questions after this. And we have some special
talks about the Places API and Activity APIs tomorrow. And we’d love to hear
your feedback and thoughts at this page. Thank you, very
much, for your time.

4 thoughts on “Google I/O 2015 – Making apps context aware: Opportunities, tools, lessons and the future”

  1. pnemonic78 says:

    GPS sensor drains battery. How does FusedLocation solve this if it is monitoring constantly?

  2. delatroy says:

    Sounds like a sterile existence where everything is measured, predicted and prescribed as if you entire life is the sum of a series of ticking check boxes.

  3. Billy Arante says:

    +Google Developers thank you!

  4. Александр АнтоноВ says:

    найти перевод на русском

Leave a Reply

Your email address will not be published. Required fields are marked *