Lecture 1.3: Evaluating Designs (12:15)

其實還沒翻完，但是不知道為什麼他外面會寫 100%（汗）

在很多方面來說，『與他人一起評估設計』這點

是在互動設計中最有創意，最具挑戰性，卻也是時常不被重視的一環

測試人們與你的設計互動，可以讓你得到洞察心得

得到新的想法，變更設計，明智的決策，以及修正錯誤

我認為設計是如此有趣的領域的一個原因，是它的真實與客觀的關係。

我發現設計如此令人著迷，因為我們在回答問題時說：

「我們怎麼測量會成功？」這樣的問題

而不只是「這是個人喜好」或「就是跟著感覺走」

同時，這個答案是複雜、開放、主觀的

需要「不只是知道使用7或3這樣的數字」的智慧

在這堂課中，我們要學的其中一件事情

就是用各種不同的方法來得到各種不同的知識

為何要與別人一起評估設計？為什麼要了解人們是如何使用互動系統的？

我想主要原因之一是「『一個使用者介面有多好』用說的難說服別人」

除非你實際找個使用者來測試，

因為客戶，設計師和工程師通常在使用者介面這個領域已經了解得太多

或者在參與設計和建造使用者介面時，已經產生盲點了

同時，他們可能不夠了解使用者真正想做什麼

雖然經驗及理論可以提供協助，仍然很難預測使用者真實會做的行為

可能想知道：「人們看的出來這玩意兒怎麼用嗎？」

「使用這介面時，使用者是會大罵髒話還是ㄎㄎ笑呢？」「這個設計如何和另一個設計比較呢？」

「如果我改動這個介面，這個變更是如何影響人們的行為呢？」

「會出現哪些新的設計？」「設計如何隨時間演進呢？」

這些都是很棒的關於介面的問題，而且要使用不同的方法來得到答案

不同方法擁有用途很廣的工具箱的價值，

類似行動通訊和社交軟體的領域裡特別寶貴的，在那裡人們使用習慣，特別是前後的差異而且，

也從人們如何使用軟體上網之類的事也會隨時間有明顯演進。

也隨時間明顯演進，其它其它的人們如何使用軟體的回響

為了讓大家稍微了解一下, 我想快速的講幾個HCI研究裡實際的類型

我提供的例子幾乎是常見的，

因為這最容易分享,

如果你有相關的例子，歡迎貼在論壇

我保持使用者介面案例收集，

並且我和其它的學生將期待看見你可以帶來什麼。

一個值得學的方法，是一個設計的使用者經驗

把人帶進你的實驗室或辦公室，試用看看

我們通常稱之為「易用性研究（usability studies）」

這「看著別人使用我設計的介面」是一個在HCI領域中常見的方法

這種傳統的以使用者為中心設計基本的策略是

反覆帶人們進入實驗室或辦公室除非時間到。才會放出來。

而且，若你預算充足，準備一間單向鏡的房間

而讓開發小組在鏡子的另一面

在一個精簡的環境，這可能只是帶人進來你宿舍辦公室。

透過這個過程，你會學習到很多很多。

每一次，我還是一個學生，朋友，或同事

看著受測者使用一個新的互動系統

我們發現，設計師的盲點，引起系統的怪行為、錯誤、錯誤的假設

然而，這種方法有一些重大的缺點。

特別是，實驗屬性不是如此貼近真實世界

在現實世界中，人們可能有不同的任務，目標，動機，物理特性

相對於你單純的辦公室或實驗室

你猜想人在使用這樣的使用者介面時，更複雜的情況

比如在一個公車站牌排隊等候的情況

第二，有"Please me"（刻意，不自然）的實驗偏見，

就在當你帶了人來測試你的介面

他們知道他們是來測試你開發的科技

並且所以他們可能努力測試或特別用心的測試

比起他們一直沒有被實驗室的條件限制

和那些看著他們受測的人一起待著

第三，最基本的形式在於你只想用一個使用者介面，沒有比較的對象。

所以，當你追縱人的開心或生氣或喜悅的微笑

你不知道他們是否可以更開心或更不生氣或更多微笑

如果你從來沒有使用過不同的使用者介面

最後，需要將人帶到實際的地方

這通常比人們想得要來得輕鬆很多很多

它可以是心理負擔，即使沒有別的
It can be a psychological burden, even if nothing else.

從人們獲得回饋的一個非常不同方式，是做一項調查

asking about different street light designs.

Surveys are great because you can quickly get feedback from a large number of responses.

And it’s relatively easy to compare multiple alternatives.

You can also automatically tally the results.

You don’t even need to build anything; you can just show screen shots or mock-ups.

One of the things that I’ve learned the hard way, though,

is the difference between what people say they’re going to do and what they actually do.

Ask people how often they exercise and you’ll probably get a much more optimistic answer

than how often they really do exercise.

The same holds for the street light example here.

Try to imagine what a number of different street light designs might be

is really different than actually observing them on the street

and having them become part of normal everyday life.

Still, it can be valuable to get feedback.

Another type of responder strategy is focus groups.

In a focus group, you’ll gather together a small group of people to discuss a design or idea.

The fact that focus groups involve a group of people is a double-edged sword.

On one hand, you can get people to tease out of their colleagues things that they might not have thought

to say on their own; on the other hand, for a variety of psychological reasons, people may be inclined

to say polite things or generate answers completely on the spot

that are totally uncorrelated with what they believe or what they would actually do.

Focus groups can be a particularly problematic method when you are looking at trying to gather data

about taboo topics or about cultural biases.

With those caveats — right now we’re just making a laundry list, and —

I think that focus groups, like almost any other method, can play an important role in your toolbelt.

Our third category of techniques is to get feedback from experts.

For example, in this class we’re going to do a bunch of peer critique for your weekly project assignments.

In addition to having users try your interface,

it can be important to eat your own dog food and use the tools that you built yourself.

When you are getting feedback from experts, it can often be helpful to have some kind of structured format,

much like the rubrics you’ll see in your project assignments.

And, for getting feedback on user interfaces, one common approach to this structured feedback

is called heuristic evaluation, and you’ll learn how to do that in this class;

it’s pioneered by Jacob Nielson.

Our next genre is comparative experiments:

taking two or more distinct options and comparing their performance to each other.

These comparisons can take place in lots of different ways:

They can be in the lab; they can be in the field; they can be online.

These experiments can be more-or-less controlled,

and they can take place over shorter or longer durations.

What you’re trying to learn here is which option is the more effective,

and, more often, what are the active ingredients,

what are the variables that matter in creating the user experience that you seek.

Here’s an example: My former PhD student Joel Brandt, and his colleague at Adobe,

ran a number of studies comparing help interfaces for programmers.

In particular they compared a more traditional search-style user interface for finding programming help

with a search interface that integrated programming help directly into your environment.

By running these comparisons they were able to see how programmers’ behaviour differed

based on the changing help user interface.

Comparative experiments have an advantage over surveys

in that you get to see the actual behaviour as opposed to self report,

and they can be better than usability studies because you’re comparing multiple alternatives.

This enables you to see what works better or worse, or at least what works different.

I find that comparative feedback is also often much more actionable.

However, if you are running controlled experiments online,

you don’t get to see much about the person on the other side of the screen.

And if you are inviting people into your office or lab,

the behaviour you’re measuring might not be very realistic.

If realistic longitudinal behaviour is what you’re after, participant observation may be the approach for you.

This approach is just what it sounds like: observing what people actually do in their actual work environment.

And this more long-term evaluation can be important for uncovering things

that you might not see in shorter term, more controlled scenarios.

For example, my colleagues Bob Sutton and Andrew Hargadon studied brainstorming.

The prior literature on brainstorming had focused mostly on questions like

“Do people come up with more ideas?”

What Bob and Andrew realized by going into the field

was that brainstorming served a number of other functions also,

like, for example, brainstorming provides a way for members of the design team

to demonstrate their creativity to their peers;

it allows them to pass along knowledge that then can be reused in other projects;

and it creates a fun, exciting environment that people like to work in and that clients like to participate in.

In a real ecosystem, all of these things are important,

in addition to just having the ideas that people come up with.

Nearly all experiments seek to build a theory on some level — I don’t mean anything fancy by this,

just that we take some things to be more relevant, and other things less relevant.

We might, for example, assume

that the ordering of search results may play an important role in what people click on,

but that the batting average of the Detroit Tigers doesn’t,

unless, of course, somebody’s searching for baseball.

If you have a theory that sufficiently, formal mathematically that you may make predictions,

then you can compare alternative interfaces using that model, without having to bring people in.

And we’ll go over that in this class a little bit, with respect to input models.

This makes it possible to try out a number of alternatives really fast.

Consequently, when people use simulations,

it’s often in conjunction with something like Monte Carlo optimization.

One example of this can be found in the ShapeWriter system,

where Shuman Zhai and colleagues figured out how to build a keyboard

where people could enter an entire word in a single stroke.

They were able to do this with the benefit of formal models and optimization-based approaches.

Simulation has mostly been used for input techniques

because people’s motor performance is probably the most well-quantified area of HCI.

And, while we won’t get much to it in this intro course,

simulation can also be used for higher-level cognitive tasks;

for example, Pete Pirolli and colleagues at PARC

had built impressive models of people’s web-searching behaviour.

These models enable them to estimate, for example, which links somebody is most likely to click on

by looking at the relevant link texts.

That’s our whirlwind tour of a number of empirical methods that this class will introduce.

You’ll want to pick the right method for the right task, and here’s some issues to consider:

If you did it again, would you get the same thing?

Another is generalizability and realism — Does this hold for people other than 18-year-old

upper-middle-class students who are doing this for course credit or a gift certificate?

Is this behaviour also what you’d see in the real world, or only in a more stilted lab environment?

Comparisons are important, because they can tell you

how the user experience would change with different interface choices,

as opposed to just a “people liked it” study.

It’s also important to think about how to achieve how these insights efficiently,

and not chew up a lot of resources, especially when your goal is practical.

My experience as a designer, researcher, teacher, consultant, advisor and mentor has taught me

that evaluating designs with people is both easier and more valuable than many people expect,

and there’s an incredible lightbulb moment that happens

when you actually get designs in front of people and see how they use them.

So, to sum up this video, I’d like to ask what could be the most important question:

“What do you want to learn?”