[Script Info]
Title: 
[Events]
Format: Layer, Start, End, Style, Name, MarginL, MarginR, MarginV, Effect, Text
Dialogue: 0,0:00:02.01,0:00:04.90,Default,,0000,0000,0000,,In this lecture, we’re going to talk about trying out your interface with people
Dialogue: 0,0:00:04.90,0:00:12.05,Default,,0000,0000,0000,,and doing so in a way that you can improve your designs based on what you learned.
Dialogue: 0,0:00:12.05,0:00:16.80,Default,,0000,0000,0000,,One of the most common things that people ask when running studies is: “Do you like my interface?”
Dialogue: 0,0:00:16.80,0:00:20.99,Default,,0000,0000,0000,,and it’s a really natural thing to ask, because on some level it’s what we all want to know.
Dialogue: 0,0:00:20.99,0:00:23.92,Default,,0000,0000,0000,,But this is really problematic on a whole lot of levels.
Dialogue: 0,0:00:23.92,0:00:28.10,Default,,0000,0000,0000,,For one it’s not very specific, and so sometimes people are trying to make this better
Dialogue: 0,0:00:28.10,0:00:34.36,Default,,0000,0000,0000,,and so they’ll improve it by doing something like: “How much do you like my interface on one to five scale?”
Dialogue: 0,0:00:34.36,0:00:39.40,Default,,0000,0000,0000,,Or: “‘This is a useful interface’ — Agree or disagree on one to five scale.”
Dialogue: 0,0:00:39.40,0:00:42.64,Default,,0000,0000,0000,,And this adds some kind of a patina of scientificness to it
Dialogue: 0,0:00:42.64,0:00:46.75,Default,,0000,0000,0000,,but really it’s just the same thing — you’re asking somebody “Do you like my interface?”
Dialogue: 0,0:00:46.75,0:00:49.68,Default,,0000,0000,0000,,And people are nice, so they’re going to say “Sure I like your interface.”
Dialogue: 0,0:00:49.68,0:00:52.26,Default,,0000,0000,0000,,This is the “please the experimenter” bias.
Dialogue: 0,0:00:52.26,0:00:56.70,Default,,0000,0000,0000,,And this can be especially strong when there are social or cultural or power differences
Dialogue: 0,0:00:56.70,0:01:00.61,Default,,0000,0000,0000,,between the experimenter and the people that you’re trying out your interface with:
Dialogue: 0,0:01:00.61,0:01:05.06,Default,,0000,0000,0000,,For example, [inaudible] and colleague show this effect in India
Dialogue: 0,0:01:05.06,0:01:09.32,Default,,0000,0000,0000,,where this effect was exacerbated when the experimenter was white.
Dialogue: 0,0:01:09.32,0:01:15.91,Default,,0000,0000,0000,,Now, you should {\i1}not{\i0} take this to mean that you shouldn’t have your developers try out stuff with users —
Dialogue: 0,0:01:15.91,0:01:21.80,Default,,0000,0000,0000,,Being the person who is both the developer and the person who is trying stuff out is incredible valuable.
Dialogue: 0,0:01:21.80,0:01:24.52,Default,,0000,0000,0000,,And one example I like a lot of this is Mike Krieger,
Dialogue: 0,0:01:24.52,0:01:30.12,Default,,0000,0000,0000,,one of the Instagram founders — [he] is also a former master student and TA of mine.
Dialogue: 0,0:01:30.12,0:01:32.31,Default,,0000,0000,0000,,And Mike, when he left Stanford and joined Silicon Valley,
Dialogue: 0,0:01:32.31,0:01:36.48,Default,,0000,0000,0000,,every Friday afternoon he would bring people into the lab into his office
Dialogue: 0,0:01:36.48,0:01:39.61,Default,,0000,0000,0000,,and have them try out whatever they were working on that week.
Dialogue: 0,0:01:39.61,0:01:43.22,Default,,0000,0000,0000,,And so that way they were able to get this regular feedback each week
Dialogue: 0,0:01:43.22,0:01:48.01,Default,,0000,0000,0000,,and the people who were building those systems got to see real people trying them out.
Dialogue: 0,0:01:48.01,0:01:52.17,Default,,0000,0000,0000,,This can be nails-on-a-chalkboard painful, but you’ll also learn a ton.
Dialogue: 0,0:01:52.17,0:01:55.12,Default,,0000,0000,0000,,So how do we get beyond “Do you like my interface?”
Dialogue: 0,0:01:55.12,0:01:58.97,Default,,0000,0000,0000,,The basic strategy that we’re going to talk about today is being able
Dialogue: 0,0:01:58.97,0:02:05.04,Default,,0000,0000,0000,,to use specific measures and concrete questions to be able to deliver meaningful results.
Dialogue: 0,0:02:05.04,0:02:10.22,Default,,0000,0000,0000,,One of the problems of “Do you like my interface?” is “Compared to what?”
Dialogue: 0,0:02:10.22,0:02:16.10,Default,,0000,0000,0000,,And I think one of the reasons people say “Yeah sure” is that there’s no comparison point
Dialogue: 0,0:02:16.10,0:02:21.89,Default,,0000,0000,0000,,and so one thing that’s really important is when you’re measuring the effectiveness of your interface,
Dialogue: 0,0:02:21.89,0:02:25.78,Default,,0000,0000,0000,,even informally, it’s really nice to have some kind of comparison.
Dialogue: 0,0:02:25.78,0:02:28.69,Default,,0000,0000,0000,,It’s also important think about, well, what’s the yardstick?
Dialogue: 0,0:02:28.69,0:02:31.18,Default,,0000,0000,0000,,What constitutes “good” in this arena?
Dialogue: 0,0:02:31.18,0:02:33.92,Default,,0000,0000,0000,,What are the measures that you’re going to use?
Dialogue: 0,0:02:33.92,0:02:36.88,Default,,0000,0000,0000,,So how can we get beyond “Do you like my interface?”
Dialogue: 0,0:02:36.88,0:02:41.07,Default,,0000,0000,0000,,One of the ways that we can start out is by asking a base rate question,
Dialogue: 0,0:02:41.07,0:02:46.53,Default,,0000,0000,0000,,like “What fraction of people click on the first link in a search results page?”
Dialogue: 0,0:02:46.53,0:02:50.14,Default,,0000,0000,0000,,Or “What fraction of students come to class?”
Dialogue: 0,0:02:50.14,0:02:54.56,Default,,0000,0000,0000,,Once we start to measure correlations things get even more interesting,
Dialogue: 0,0:02:54.56,0:03:00.33,Default,,0000,0000,0000,,like, “Is there a relationship between the time of day a class is offered and how many students attend it?”
Dialogue: 0,0:03:00.33,0:03:07.61,Default,,0000,0000,0000,,Or “Is there a relationship between the order of a search result and the clickthrough rate?”
Dialogue: 0,0:03:07.61,0:03:11.49,Default,,0000,0000,0000,,For both students and clickthrough, there can be multiple explanations.
Dialogue: 0,0:03:11.49,0:03:16.41,Default,,0000,0000,0000,,For example, if there are fewer students that attend early morning classes,
Dialogue: 0,0:03:16.41,0:03:19.05,Default,,0000,0000,0000,,is that a function of when students want to show up,
Dialogue: 0,0:03:19.05,0:03:22.86,Default,,0000,0000,0000,,or is that a function of when good professors want to teach?
Dialogue: 0,0:03:22.86,0:03:26.22,Default,,0000,0000,0000,,With the clickthrough example, there are also two kinds of explanations.
Dialogue: 0,0:03:26.22,0:03:37.53,Default,,0000,0000,0000,,If lower placed links yield fewer clicks, Is that because the links are of intrinsically poorer quality,
Dialogue: 0,0:03:37.53,0:03:41.08,Default,,0000,0000,0000,,or is it because people just click on the first link —
Dialogue: 0,0:03:41.08,0:03:45.24,Default,,0000,0000,0000,,[that] they don’t bother getting to the second one even if it might be better?
Dialogue: 0,0:03:45.24,0:03:48.87,Default,,0000,0000,0000,,To isolate the effect of placement and identifying it as playing a casual role,
Dialogue: 0,0:03:48.87,0:03:54.16,Default,,0000,0000,0000,,you’d need to isolate that as a variable by say, randomizing the order or search results.
Dialogue: 0,0:03:54.16,0:04:00.33,Default,,0000,0000,0000,,As we start to talk about these experiments, let’s introduce a few terms that are going to help us.
Dialogue: 0,0:04:00.33,0:04:05.48,Default,,0000,0000,0000,,The multiple different conditions that we try, that’s the thing we are manipulating —
Dialogue: 0,0:04:05.48,0:04:12.40,Default,,0000,0000,0000,,for example, the time of a class, or the location of a particular link on a search results page.
Dialogue: 0,0:04:12.40,0:04:18.38,Default,,0000,0000,0000,,These manipulations are independent variables because they are independent of what the user does.
Dialogue: 0,0:04:18.38,0:04:22.24,Default,,0000,0000,0000,,They are in the control of the experimenter.
Dialogue: 0,0:04:22.24,0:04:26.71,Default,,0000,0000,0000,,Then we are going to measure what the user does
Dialogue: 0,0:04:26.71,0:04:31.45,Default,,0000,0000,0000,,and those measures are called dependent variables because they depend on what the user does.
Dialogue: 0,0:04:31.45,0:04:36.01,Default,,0000,0000,0000,,Common measures in HCI include things like task completion time —
Dialogue: 0,0:04:36.01,0:04:38.98,Default,,0000,0000,0000,,How long does it take somebody to complete a task
Dialogue: 0,0:04:38.98,0:04:43.38,Default,,0000,0000,0000,,(for example, find something I want to buy, create a new account, order an item)?
Dialogue: 0,0:04:43.38,0:04:46.84,Default,,0000,0000,0000,,Accuracy — How many mistakes did people make,
Dialogue: 0,0:04:46.84,0:04:51.30,Default,,0000,0000,0000,,and were those fatal errors or were those things that they were able to quickly recover from?
Dialogue: 0,0:04:51.30,0:04:55.38,Default,,0000,0000,0000,,Recall — How much does a person remember afterward, or after periods of non-use?
Dialogue: 0,0:04:55.38,0:04:59.18,Default,,0000,0000,0000,,And emotional response — How does the person feel about the tasks being completed?
Dialogue: 0,0:04:59.18,0:05:01.44,Default,,0000,0000,0000,,Were they confident, were they stressed?
Dialogue: 0,0:05:01.44,0:05:04.35,Default,,0000,0000,0000,,Would the user recommend this system to a friend?
Dialogue: 0,0:05:04.35,0:05:09.08,Default,,0000,0000,0000,,So, your independent variables are the things that you manipulate,
Dialogue: 0,0:05:09.08,0:05:11.98,Default,,0000,0000,0000,,your dependent variables are the things that you measure.
Dialogue: 0,0:05:11.98,0:05:14.03,Default,,0000,0000,0000,,How reliable is your experiment?
Dialogue: 0,0:05:14.03,0:05:17.57,Default,,0000,0000,0000,,If you ran this again, would you see the same results?
Dialogue: 0,0:05:17.57,0:05:20.92,Default,,0000,0000,0000,,That’s the internal validity of an experiment.
Dialogue: 0,0:05:20.92,0:05:24.78,Default,,0000,0000,0000,,So, have a precise experiment, you need to better remove the confounding factors.
Dialogue: 0,0:05:24.78,0:05:30.35,Default,,0000,0000,0000,,Also, it’s important to study enough people so that the result is unlikely to have been by chance.
Dialogue: 0,0:05:30.35,0:05:34.37,Default,,0000,0000,0000,,You may be able to run the same study over and over and get the same results
Dialogue: 0,0:05:34.37,0:05:42.21,Default,,0000,0000,0000,,but it may not matter in some real-world sense and the external validity is the generalizability of your results.
Dialogue: 0,0:05:42.21,0:05:44.90,Default,,0000,0000,0000,,Does this apply only to eighteen-year-olds in a college classroom?
Dialogue: 0,0:05:44.90,0:05:47.91,Default,,0000,0000,0000,,Or does this apply to everybody in the world?
Dialogue: 0,0:05:47.91,0:05:52.00,Default,,0000,0000,0000,,Let’s bring this back to HCI and talk about one of the problems you’re likely to face as a designer.
Dialogue: 0,0:05:52.00,0:05:55.50,Default,,0000,0000,0000,,I think one of the things that we commonly want to be able to do
Dialogue: 0,0:05:55.50,0:06:00.36,Default,,0000,0000,0000,,is to be able to ask something like “Is my cool new approach better than the industry standard?”
Dialogue: 0,0:06:00.36,0:06:03.29,Default,,0000,0000,0000,,Because after all, that’s why you’re making the new thing.
Dialogue: 0,0:06:03.29,0:06:06.96,Default,,0000,0000,0000,,Now, one of the challenges with this, especially early on in the design process
Dialogue: 0,0:06:06.96,0:06:11.03,Default,,0000,0000,0000,,is that you may have something which is very much in its prototype stages
Dialogue: 0,0:06:11.03,0:06:16.84,Default,,0000,0000,0000,,and something that is the industry standard is likely to benefit from years and years of refinement.
Dialogue: 0,0:06:16.84,0:06:21.51,Default,,0000,0000,0000,,And at the same time, it may be stuck with years and years of cruft
Dialogue: 0,0:06:21.51,0:06:25.11,Default,,0000,0000,0000,,which may or may not be intrinsic to its approach.
Dialogue: 0,0:06:25.11,0:06:30.59,Default,,0000,0000,0000,,So if you compare your cool new tool to some industry standard, there is two things varying here.
Dialogue: 0,0:06:30.59,0:06:35.72,Default,,0000,0000,0000,,One is the fidelity of the implementation and the other one of course is the approach.
Dialogue: 0,0:06:35.72,0:06:37.82,Default,,0000,0000,0000,,Consequently, when you get the results,
Dialogue: 0,0:06:37.82,0:06:43.93,Default,,0000,0000,0000,,you can’t know whether to attribute the results to fidelity or approach or some combination of the two.
Dialogue: 0,0:06:43.93,0:06:48.40,Default,,0000,0000,0000,,So we’re going to talk about ways of teasing apart those different causal factors.
Dialogue: 0,0:06:48.40,0:06:53.71,Default,,0000,0000,0000,,Now, one thing I should say right off the bat is there are some times where it may be more
Dialogue: 0,0:06:53.71,0:06:57.33,Default,,0000,0000,0000,,or less relevant whether you have a good handle on what the causal factors are.
Dialogue: 0,0:06:57.33,0:07:01.41,Default,,0000,0000,0000,,So for example, if you’re trying to decide between two different digital cameras,
Dialogue: 0,0:07:01.41,0:07:07.73,Default,,0000,0000,0000,,at the end of the day, maybe all you care about is image quality or usability or some other factor
Dialogue: 0,0:07:07.73,0:07:12.83,Default,,0000,0000,0000,,and exactly what makes that image quality better or worse
Dialogue: 0,0:07:12.83,0:07:17.83,Default,,0000,0000,0000,,or any other element along the way may be less relevant to you.
Dialogue: 0,0:07:17.83,0:07:24.03,Default,,0000,0000,0000,,If you don’t have control over the variables, then identifying cause may not always be what you want.
Dialogue: 0,0:07:24.03,0:07:27.69,Default,,0000,0000,0000,,But when you are a designer, you do have control over the variables,
Dialogue: 0,0:07:27.69,0:07:30.72,Default,,0000,0000,0000,,and that’s when it is really important to ascertain cause.
Dialogue: 0,0:07:30.72,0:07:35.95,Default,,0000,0000,0000,,Here’s an example of a study that came out right when the iPhone was released,
Dialogue: 0,0:07:35.95,0:07:41.04,Default,,0000,0000,0000,,done by a research firm User Centric, and I’m going to read from this news article here.
Dialogue: 0,0:07:41.04,0:07:43.50,Default,,0000,0000,0000,,Research firm User Centric has released a study
Dialogue: 0,0:07:43.50,0:07:48.73,Default,,0000,0000,0000,,that tries to gauge how effective the iPhone’s unusual onscreen keyboard is.
Dialogue: 0,0:07:48.73,0:07:51.07,Default,,0000,0000,0000,,The goal is certainly a noble one
Dialogue: 0,0:07:51.07,0:07:56.34,Default,,0000,0000,0000,,but I cannot say the survey’s approach results in data that makes much sense.
Dialogue: 0,0:07:56.34,0:07:59.86,Default,,0000,0000,0000,,User Centric brought in twenty owners of other phones.
Dialogue: 0,0:07:59.86,0:08:05.12,Default,,0000,0000,0000,,Half had qwerty keyboards, half had ordinary numeric phones, with keypads.
Dialogue: 0,0:08:05.12,0:08:08.09,Default,,0000,0000,0000,,None were familiar with the iPhone.
Dialogue: 0,0:08:08.09,0:08:13.68,Default,,0000,0000,0000,,The research involved having the test subjects enter six sample test messages with the phones
Dialogue: 0,0:08:13.68,0:08:17.34,Default,,0000,0000,0000,,that they already had, and six with the iPhone.
Dialogue: 0,0:08:17.34,0:08:20.82,Default,,0000,0000,0000,,The end result was that the iPhone newbies took twice as long
Dialogue: 0,0:08:20.82,0:08:26.78,Default,,0000,0000,0000,,to enter text with an iPhone as they did with their own phones and made lots more typos.
Dialogue: 0,0:08:26.78,0:08:31.62,Default,,0000,0000,0000,,So let’s critique this study and talk about its benefits and drawbacks.
Dialogue: 0,0:08:31.62,0:08:34.02,Default,,0000,0000,0000,,Here’s the webpage directly from User Centric.
Dialogue: 0,0:08:34.02,0:08:37.62,Default,,0000,0000,0000,,What’s our manipulation in this study?
Dialogue: 0,0:08:37.62,0:08:41.78,Default,,0000,0000,0000,,Well the manipulation is going to be the input style.
Dialogue: 0,0:08:41.78,0:08:45.08,Default,,0000,0000,0000,,How about the measure in the study?
Dialogue: 0,0:08:45.08,0:08:48.63,Default,,0000,0000,0000,,It’s going to be the words per minute.
Dialogue: 0,0:08:48.63,0:08:56.31,Default,,0000,0000,0000,,And there’s absolutely value in being able to measure the initial usability of the iPhone.
Dialogue: 0,0:08:56.31,0:09:00.37,Default,,0000,0000,0000,,For several reasons, one is if you’re introducing new technology,
Dialogue: 0,0:09:00.37,0:09:03.68,Default,,0000,0000,0000,,it’s beneficial if people are able to get up to speed pretty quickly.
Dialogue: 0,0:09:03.68,0:09:09.33,Default,,0000,0000,0000,,However it’s important to realize that this comparison is intrinsically unfair
Dialogue: 0,0:09:09.33,0:09:14.94,Default,,0000,0000,0000,,because the users of the previous cell phones were experts at that input modality
Dialogue: 0,0:09:14.94,0:09:18.70,Default,,0000,0000,0000,,and the people who are using the iphone are novices in that modality.
Dialogue: 0,0:09:18.70,0:09:24.04,Default,,0000,0000,0000,,And so it seems quite likely that the iPhone users, once they become actual users,
Dialogue: 0,0:09:24.04,0:09:29.48,Default,,0000,0000,0000,,are going to get better over time and so if you’re not used to something the first time you try it,
Dialogue: 0,0:09:29.48,0:09:35.06,Default,,0000,0000,0000,,that may not be a deal killer, and it’s certainly not an apples-to-apples comparison.
Dialogue: 0,0:09:35.06,0:09:40.01,Default,,0000,0000,0000,,Another thing that we don’t get out of this article is “Is this difference significant?”
Dialogue: 0,0:09:40.01,0:09:46.96,Default,,0000,0000,0000,,So we read that each person who typed six messages in each of two conditions
Dialogue: 0,0:09:46.96,0:09:52.00,Default,,0000,0000,0000,,and so they did their own device and the iPhone, or vice versa.
Dialogue: 0,0:09:52.00,0:10:00.00,Default,,0000,0000,0000,,Six messages each and that the iPhone users were half the speed of the…
Dialogue: 0,0:10:00.00,0:10:08.81,Default,,0000,0000,0000,,or rather the people typing with the iPhone were half as fast as when they got to type with a mini qwerty
Dialogue: 0,0:10:08.81,0:10:12.57,Default,,0000,0000,0000,,at the device that they were accustomed to.
Dialogue: 0,0:10:12.57,0:10:17.13,Default,,0000,0000,0000,,So while this may tell us something about the initial usability of the iPhone,
Dialogue: 0,0:10:17.13,0:10:23.01,Default,,0000,0000,0000,,in terms of the long-term usability, you know, I don’t think we get so much out of this here.
Dialogue: 0,0:10:23.01,0:10:29.82,Default,,0000,0000,0000,,If you weren’t s atisfied by that initial data, you’re in good company: neither were the authors of that study.
Dialogue: 0,0:10:29.82,0:10:35.45,Default,,0000,0000,0000,,So they went back a month later and they ran another study where they brought in 40 new people to the lab
Dialogue: 0,0:10:35.45,0:10:39.95,Default,,0000,0000,0000,,who were either iPhone users, qwerty users, or nine key users.
Dialogue: 0,0:10:39.95,0:10:42.87,Default,,0000,0000,0000,,And now it’s more of an apples-to-apples comparison
Dialogue: 0,0:10:42.87,0:10:48.99,Default,,0000,0000,0000,,in that they are going to test people that are relatively experts in these three different modalities —
Dialogue: 0,0:10:48.99,0:10:55.31,Default,,0000,0000,0000,,after about a month on the iPhone you’re {\i1}probably{\i0} starting to asymptote in terms of your performance.
Dialogue: 0,0:10:55.31,0:11:02.88,Default,,0000,0000,0000,,Definitely it gets better over time, even past a month; but, you know, a month starts to get more reasonable.
Dialogue: 0,0:11:02.88,0:11:12.01,Default,,0000,0000,0000,,And what they found was that iPhone users and qwerty users were about the same in terms of speed,
Dialogue: 0,0:11:12.01,0:11:16.92,Default,,0000,0000,0000,,and that the numeric keypad users were much slower.
Dialogue: 0,0:11:16.92,0:11:21.74,Default,,0000,0000,0000,,So once again our manipulation is going to be input style and we’re going to measure speed.
Dialogue: 0,0:11:21.74,0:11:24.56,Default,,0000,0000,0000,,This time we’re also going to measure error rate.
Dialogue: 0,0:11:24.56,0:11:30.42,Default,,0000,0000,0000,,And what we see is that iPhone users and qwerty users are essentially the same speed.
Dialogue: 0,0:11:30.42,0:11:36.54,Default,,0000,0000,0000,,However, the iPhone users make many more errors.
Dialogue: 0,0:11:36.54,0:11:40.15,Default,,0000,0000,0000,,Now, one thing I should point out about the study is
Dialogue: 0,0:11:40.15,0:11:46.78,Default,,0000,0000,0000,,that each of the different devices was used by a different group of people.
Dialogue: 0,0:11:46.78,0:11:51.60,Default,,0000,0000,0000,,And it was done this way so that each device was used by somebody
Dialogue: 0,0:11:51.60,0:11:55.88,Default,,0000,0000,0000,,who is comfortable and had experience with working with that device.
Dialogue: 0,0:11:55.88,0:12:00.52,Default,,0000,0000,0000,,And so, we removed the worry that you had newbies working on these devices.
Dialogue: 0,0:12:00.52,0:12:04.60,Default,,0000,0000,0000,,However, especially in 2007, there may have been significant differences
Dialogue: 0,0:12:04.60,0:12:11.31,Default,,0000,0000,0000,,in who the people were who were using the early adopters of the 2007 iPhone
Dialogue: 0,0:12:11.31,0:12:17.05,Default,,0000,0000,0000,,or maybe business users were particularly drawn to the qwerty devices or people who had better things
Dialogue: 0,0:12:17.05,0:12:22.46,Default,,0000,0000,0000,,to do with their time than send e-mail on their telephone or using the nine key devices.
Dialogue: 0,0:12:22.46,0:12:26.64,Default,,0000,0000,0000,,And so, while this comparison is better than the previous one,
Dialogue: 0,0:12:26.64,0:12:31.50,Default,,0000,0000,0000,,the potential for variation between the user populations is still problematic.
Dialogue: 0,0:12:31.50,0:12:36.84,Default,,0000,0000,0000,,If what you’d like to be able to claim is something about the intrinsic properties of the device,
Dialogue: 0,0:12:36.84,0:12:42.21,Default,,0000,0000,0000,,it may at least in part have to do with the users.
Dialogue: 0,0:12:42.21,0:12:45.44,Default,,0000,0000,0000,,So, what are some st rategies for fairer comparison?
Dialogue: 0,0:12:45.44,0:12:50.25,Default,,0000,0000,0000,,To brainstorm a couple of options one thing that you can do is insert your approach in to your production setting
Dialogue: 0,0:12:50.25,0:12:52.69,Default,,0000,0000,0000,,and this may seem like a lot of work —
Dialogue: 0,0:12:52.69,0:12:56.54,Default,,0000,0000,0000,,sometimes it is but in the age of the web this is a lot easier than it used to be.
Dialogue: 0,0:12:56.54,0:13:03.13,Default,,0000,0000,0000,,And it’s possible even if you don’t have access to the server of the service that you’re comparing against.
Dialogue: 0,0:13:03.13,0:13:06.56,Default,,0000,0000,0000,,You can use things like a proxy server or client-side scripting
Dialogue: 0,0:13:06.56,0:13:11.57,Default,,0000,0000,0000,,to be able to put your own technique in and have an apples-to-apples comparison.
Dialogue: 0,0:13:11.57,0:13:16.58,Default,,0000,0000,0000,,A second strategy for neutralizing the environment difference between a production version
Dialogue: 0,0:13:16.58,0:13:25.69,Default,,0000,0000,0000,,and your new approach is to make a version of the production thing in the same style as your new approach.
Dialogue: 0,0:13:25.69,0:13:30.90,Default,,0000,0000,0000,,That also makes them equivalent in terms of their implementation fidelity.
Dialogue: 0,0:13:30.90,0:13:34.00,Default,,0000,0000,0000,,A third strategy and one that’s used commonly in research,
Dialogue: 0,0:13:34.00,0:13:39.42,Default,,0000,0000,0000,,is to scale things down so you’re looking at just a piece of the system at a particular point in time.
Dialogue: 0,0:13:39.42,0:13:42.71,Default,,0000,0000,0000,,That way you don’t have to worry about implementing a whole big, giant thing.
Dialogue: 0,0:13:42.71,0:13:48.19,Default,,0000,0000,0000,,You can just focus on one small piece and have that comparison be fair.
Dialogue: 0,0:13:48.19,0:13:52.78,Default,,0000,0000,0000,,And the fourth strategy is that when expertise is relevant,
Dialogue: 0,0:13:52.78,0:13:55.86,Default,,0000,0000,0000,,train people up — give them the practice that they need —,
Dialogue: 0,0:13:55.86,0:14:00.74,Default,,0000,0000,0000,,so that they can start at least hitting that asymptote in terms of performance
Dialogue: 0,0:14:00.74,0:14:04.99,Default,,0000,0000,0000,,and you can get a better read than what they would be as newbies.
Dialogue: 0,0:14:04.99,0:14:11.80,Default,,0000,0000,0000,,So now to close out this lecture, if somebody asks you the question “Is interface x better than interface y?”
Dialogue: 0,0:14:11.80,0:14:15.26,Default,,0000,0000,0000,,you know that we’re off to a good start because we have a comparison.
Dialogue: 0,0:14:15.26,0:14:18.54,Default,,0000,0000,0000,,However, you also know to be worried: What does “better” mean?
Dialogue: 0,0:14:18.54,0:14:25.96,Default,,0000,0000,0000,,And often, in a complex system, you’re going to have several measures. That’s totally cool.
Dialogue: 0,0:14:25.96,0:14:30.58,Default,,0000,0000,0000,,There’s a lot of value in being explicit though about what it is you mean by better —
Dialogue: 0,0:14:30.58,0:14:33.72,Default,,0000,0000,0000,,What are you trying to accomplish? What are you trying to [im]prove?
Dialogue: 0,0:14:33.72,0:14:38.00,Default,,0000,0000,0000,,And if anybody ever tells you that their interface is {\i1}always{\i0} better,
Dialogue: 0,0:14:38.00,0:14:44.30,Default,,0000,0000,0000,,don’t believe them because nearly all of the time the answer is going to be “it depends.”
Dialogue: 0,0:14:44.30,0:14:48.44,Default,,0000,0000,0000,,And the interesting question is “What does it depend on?”
Dialogue: 0,0:14:48.44,0:14:53.00,Default,,0000,0000,0000,,Most interfaces are good for some things and not for others.
Dialogue: 0,0:14:53.00,0:14:57.97,Default,,0000,0000,0000,,For example if you have a tablet computer where all of the screen is devoted to display,
Dialogue: 0,0:14:57.97,0:15:04.20,Default,,0000,0000,0000,,that is going to be great for reading, for web browsing, for that kind of activity, looking at pictures.
Dialogue: 0,0:15:04.20,0:15:06.37,Default,,0000,0000,0000,,Not so good if you want to type a novel.
Dialogue: 0,0:15:06.37,0:15:09.14,Default,,0000,0000,0000,,So here, we’ve introduced controlled comparison
Dialogue: 0,0:15:09.14,0:15:13.78,Default,,0000,0000,0000,,as a way of finding the smoking gun, as a way of inferring cause.
Dialogue: 0,0:15:13.78,0:15:17.31,Default,,0000,0000,0000,,And often for, when you have only two conditions,
Dialogue: 0,0:15:17.31,0:15:21.00,Default,,0000,0000,0000,,we’re going to talk about that as being a minimal pairs design.
Dialogue: 0,0:15:21.00,0:15:24.92,Default,,0000,0000,0000,,As a practicing designer, the reason to care about what’s causal
Dialogue: 0,0:15:24.92,0:15:29.60,Default,,0000,0000,0000,,is that it gives you the material to make a better decision going forward.
Dialogue: 0,0:15:29.60,0:15:32.20,Default,,0000,0000,0000,,A lot of studies violate this constraint.
Dialogue: 0,0:15:32.20,0:15:39.71,Default,,0000,0000,0000,,And, that gets dangerous because it doesn’t, it prevents you from being able to make sound decisions.
Dialogue: 0,0:15:39.71,0:15:43.80,Default,,0000,0000,0000,,I hope that the tools that we’ve talked about today and in the next several lectures
Dialogue: 0,0:15:43.80,0:15:48.82,Default,,0000,0000,0000,,will help you become a wise skeptic like our friend in this XKCD comic.
Dialogue: 0,0:15:48.82,0:15:53.00,Default,,0000,0000,0000,,I’ll see you next time.