Abstract

There has been a surge in artificial intelligence (AI) technologies co-opted by or designed for people with visual disabilities. Researchers and engineers have pushed technical boundaries in areas such as computer vision, natural language processing, location inference, and wearable computing. But what do people with visual disabilities imagine as their own technological future? To explore this question, we developed and carried out tactile ideation workshops with participants in the UK and India. Our participants generated a large and diverse set of ideas, most focusing on ways to meet needs related to social interaction. In some cases, this was a matter of recognizing people. In other cases, they wanted to be able to participate in social situations without foregrounding their disability. It was striking that this finding was consistent across UK and India despite substantial cultural and infrastructural differences. In this paper, we describe a new technique for working with people with visual disabilities to imagine new technologies that are tuned to their needs and aspirations. Based on our experience with these workshops, we provide a set of social dimensions to consider in the design of new AI technologies: social participation, social navigation, social maintenance, and social independence. We offer these social dimensions as a starting point to forefront users’ social needs and desires as a more deliberate consideration for assistive technology design.

Keywords

Visually impaired, blind, accessibility, design, ideation, multicultural, AI, artificial intelligence

Introduction

There is a strong industrial push to create (artificial) intelligent agents that utilize speech and computational vision to enable new experiences. While intended for the mainstream, it is people with visual disabilities who have become especially proficient power users of conversational agents . More recently, there have been more explicit explorations of how computational vision might enable the agent experience for people with visual disabilities. We see products that: find and read text , identify objects and people , , as well as describe images on social media .

As artificial intelligence matures, it becomes increasingly important to understand the kinds of things that people with visual disabilities would like to have as part of their tech toolkit. While there is a growing literature on what people find challenging now (e.g., ,), we wanted to prompt those with visual disabilities to imagine what artificial intelligence might offer in the future. Such a future may address practical problems that users currently face, or include a set of new abilities that we have not yet considered.

Helping people imagine novel ideas is often done through structured ideation methods . However, typically, these methods rely on overt visual activities such as using ideation cards to prompt ideas as well as subtler visual activities such as recording the outcome of an exercise with post-it notes. Moreover, in group work - even in sharing physical models - collaboration can be highly visual. To work with people with visual disabilities, it is clear new ideation tasks are needed that do not rely on vision. In this paper, we describe a set of novel ideation tasks that we adapted to use with a diverse group of people with visual disabilities.

As people with visual disabilities are a very diverse group, we wanted to reflect that diversity in our participants. In particular, while much research and development in assistive technology has focused on resource-rich environments with advanced infrastructure, about 90% of the world’s 285 million people with visual disabilities live in low-income settings . As we explore how intelligent agents can enable people with visual disabilities, we wanted to consider how differences in context, culture, and resource availability would affect the ideas generated. To this end, we used our ideation methods in two contrasting contexts, UK and India.

The central focus of this paper is a synthesis of the ideas generated using adapted, tactile ideation techniques with visually disabled participants in workshops held in the UK and India. We found that participants, despite different cultural contexts, focused on intelligent technologies that enabled them to interact more easily with others. Even mundane challenges were couched very directly in the social context in which they were raised. These findings suggest that we need to take careful consideration of the social dimensions of the lived experiences of people with visual disabilities.

In doing so, we might extend the common emphasis on wholly practical or functional challenges this user group faces, such as identifying money or navigating a floorplan. A design space might thus be opened up that focuses on the subtler, but equally important set of challenges introduced by participating in rich and varied social settings, that have yet to receive substantial attention.

This paper makes three specific contributions:

  1. The concept and realization of tactile ideation workshops specifically developed for people with visual disabilities;
  2. A cross-cultural comparison of ideas generated by people with visual disabilities in the UK and India, illustrating a consistent desire for social experience;
  3. An articulation of a set of social dimensions to further a more deliberate design consideration for users’ social needs and desires in assistive technology design.

Related Work

We begin this section with a brief overview of the types of systems being developed for people with visual disabilities in recent years. We then draw upon a large, diverse literature on ideation, capturing relevant key ideas that can be utilized for tactile ideation, and summarize related literature on designing with people who have a visual disability. As these literatures are diverse and spread across academic fields and industry, we do not attempt to cover them exhaustively, but highlight elements that are particularly relevant to the findings within this paper.

Systems Research in Visual Disability

There is a considerable literature on developing systems to make life easier for those who with visual disabilities. Recent papers have focused on: the creation and use of tactile graphics; improvements to screen readers, such as concurrent audio or access to charts ; reading out visual information with finger-mounted cameras ; 3D printed tactile maps ; supporting code navigation ; and not least, blind photography . The majority of these systems, while diverse, are motivated by access issues, providing support for actions and activities available to people with sight.

The most heavily researched area has been that of navigation and orientation. These range from spotting zebra crossings to the use of guide drones . Other examples include: finding bus stops ; traversing open spaces ; navigating in buildings , and indoor navigation more generally . In addition to this work, there are many technical contributions supporting system development for navigation such as the use of computer vision . We are also beginning to see navigation technologies reach a large number of users through industrial efforts, such as Microsoft Soundscape, a 3D spatialized audio navigation system, and American Printing House for the Blind’s Nearby Explorer app .

A newer area of concentration is object and image identification. Early explorations allowed users to crowd-source picture and object recognition . More recently, research has identified the challenges of recognizing images on social networking sights , and addressed them by designing an automatic captioning service . Others have looked at object identification more directly through a proposal for a personalized object detector ; studies understanding image capture for object identification ; as well as design experiments to understand object identification for people with low vision . With these studies, we see design proposals or initial uses of artificial intelligence for practical tasks.

There is now a growing literature that focuses on the lived experience of people with visual disabilities rather than the technology per se. Research has pointed out that assistive technology can impede social interaction , the authors introducing the term social accessibility to prompt designers to think beyond the assistance a device provides to its practicality in social settings . Zolyomi et al. take this one step further to consider the social dimensions of adopting a sight assistive technology . These authors pull out several examples in which people chose to access visual cues from the system to support social participation through understanding the surroundings or a conversational reference.

The social dimensions of the lived experience, however, have only received limited attention by system builders. One group of researchers have explored a social assistant . Made from a camera and vibrating belt, the system indicates the location and distance of an interaction partner and their facial expression. This work illustrates how system and person co-adapt. Other research explores the capture of emotional valence and head nodding, and delivers the determined responses verbally. They illustrate the challenge of generating categorical responses that may be context dependent . There is also work (building on affect recognition) that focuses on communicating gaze direction through tactile feedback. However, so far, the system is only beginning to be tested with visually disabled users .

Ideation Methods

Ideation is the creative process of generating new ideas. While there are many methods, Graham and Bachmann delineate nine approaches . Some of these approaches are to solve specific problems, such as a known accessibility issues. Others are intended to create entities or experiences not yet known. There are two types of methods in this latter category that we would particularly like to highlight as relevant here: 1) derivative ideas that involve changing an existing entity; and 2) symbiotic ideas which come from combining multiple ideas into a singular entity. These two approaches, particularly suited to engendering new ideas, have been embodied in a range of different techniques.

Ideation cards offer a common technique used in designing interactive experiences with technology. Such cards help participants reflect on specific aspects of a design or combine unexpected ideas. Golembewski, for example, has shown how designers might create their own cards, helping them mix people, place, and objects in serendipitous ways . There are a number of card sets available around specific topics, such as humanistic aspects of design or legal and ethical aspects in technology design . Woelfel provides an overview of existing tools .

Another ideation technique, and one that is more tactile, is Lego Serious Play (LSP). LSP is a facilitated workshop in which participants respond to tasks by building symbolic and metaphorical models using LEGO and produce accompanying narratives. There is an emphasis on concrete expression of experiences and ideas that are otherwise abstract. The practical challenges of implementing this method have been nicely documented for imagining health futures . This work describes how to build the confidence of participants in the ideation process, gradually scaling up the exercises. It also talks about the importance of mixing individual and group tasks to enable the sharing and building on others’ ideas. While tactile, LSP relies heavily on sight to build and narrate the concept.

There are many other ideation activities and games that enable idea generation through derivation, symbiosis, and spontaneity . Other methods that we have drawn upon include: show-and-tell, object brainstorming, and critique. Show-and-tell is an activity in which everyone brings an object and describes how it represents an activity, enabling any of the three idea generation types. Similarly, object brainstorming utilizes objects chosen at random as a source for inspiration. Critique of current systems is a mechanism to provide insights into design issues by populations who do not currently use a technology, e.g. older people and banking . These methods provide the starting points for creating tactile ideation methods.

Design with People with Visual Disabilities

A range of methods have been used to give voice to visually disabled users in the design process. A common method is in situ interviewing or observation. For example, Branham and Kane interviewed blind individuals and their partners in their homes, exploring how they collaborate in creating accessible home spaces. Describing how "can't do" activities can move to "can do" activities with preparation help from a partner, this method enabled the authors to highlight the range of existing strategies that people already have to achieve an accessible home, and how the social dimensions must be accounted for in technology design.

User-centered design approaches shift the focus from understanding the user to encouraging the user to articulate their needs through a design process. Ye et al. , for example, use a wearable probe to provide a sense of the material and the practical experience of interacting with a wearable device through a speech-interface. The authors used the probe to help participants articulate their views on both form and function, giving users a voice in design problems scoped by the designer.

Participatory design attempts to integrate users into the design process itself, to capture tacit knowledge in the production of all aspects of the design from concept to features. We find examples of participatory design with people with visual disabilities in a range of contexts. Azenkot et al. for example used interviews and a method inspired by contextual inquiry to explore with blind participants how they would prefer interacting with a building service robot. A designer controlled and spoke for a robot guiding the participant down a hallway. Throughout, probing and clarifying questions were asked to gain insight into indoor navigation challenges and strategies as well as actionable design recommendations.

Brock describes the use of a multi-touch screen, a tactile paper map and audio output in the design of accessible interactive map prototypes for investigating haptic exploration strategies on spatial cognition with people with VI. Their approach involved brainstorming sessions, Wizard-of-OZ prototyping using raised-line maps and simulated speech output of proposed interaction concepts, and finally a comparative study to evaluate first system, assessing effectiveness and efficiency of their proposed solution and informing further improvements.

Designng a wearable navigation aid, Williams et al. employed low-fidelity prototyping with craft materials as an alternative to sketching. However, participants preferred talking in a question-answer style rather than utilizing the craft materials, leaving the facilitator to construct prototypes based on the discussions. The authors reflect that more structure would have helped people engage in this unfamiliar task. In a later activity, participants were asked to assemble a set of electronic components to design a device in response to a particular scenario. While this enabled detailed conversations about the device’s physical design, it did not help them generate new ideas.

Metatla et al. employed a range of methods in their PD process. Amongst others they found that hands-on experiences with technology demonstrations helped build understanding of the state of the art in accessible systems and a shared vocabulary for expressing non-visual design ideas. Similar to Williams et al [REF], in the creation of audio-haptic mock-ups, participants drifted away from the material and were found to only focus on verbal descriptions. Malleable digital prototypes in comparison were generative as they provided alternative ways to support a task. Finally, audio diaries provided a useful resource of extra feedback outside of design sessions.

Sahib et al. describe the use of scenarios as a basis for inviting dialogue with blind users to simulate how they would interact with a search interface. This enabled participants to provide formative feedback and critique proposed designs as well as to provide new ideas based on their experiences with assistive technologies.

Andrews also presents a host of methods used over a number of years to engage blind and partially sighted people in participatory design processes. These included: moodboards, foam models, cards, existing product feedback and storytelling. One of the key adaptations needed to make these activities work was the use of the designer as transcriber to questions posed; an approach that heavily relies on how the sighted person summarises and prioritises what is said, and that potentially risks taking the direct voice away from people with VI. Other issues identified included the use of low-fidelity prototyping to inspire open-ended conversation. The reliance on touch instead often encouraged detailed feedback, before moving to the general concept. While this emphasis is understandable, it ran counter to the purpose of the low-fidelity forms. This suggests the tactile and auditory experience of materials, rather than their visual form, might be considered in design activities.

Lastly, Ratto et al. detail their engagement in PD to design a better blind tennis ball. Blind tennis athletes and several hackers/engineers started with a discussion of the sport and problems faced. Following, they focused on prototyping with the labor divided between sighted people (building) and blind people (testing and commenting). To address this, the authors built tools for blind prototyping, including tactile overlays for circuit boards and a digital multimeter, highlighting the complex relationship between the materials being used and participation.

Method

Our primary methodological contribution is the design of tactile ideation tasks. Our proposal draws on two types of idea generation introduced above: derivative ideas and symbiotic ideas. Moreover, we have sought to adapt some of the existing approaches above to be orientated much more to non-visual interaction. In this section, we first present the concept of these tactile ideation tasks. We then discuss the two settings in which the workshops were carried out in UK and India. Finally, we describe how we synthesized the results of the ideation process for presentation in this paper.

Tactile Ideation Tasks

The workshop concept is built around the notion of a sixth sense, or a superpower, that the participants would like to have. A focus on a sixth sense was chosen to help people articulate a desire for a world augmented in some way while being technologically agnostic. More specifically, it was intended to question information need irrespective of technology, while respecting highly developed individual strategies and preferences to sense and form an understanding of the world. The workshop consists of two activity sessions.

Activity Set 1

The first session starts by asking people in small groups to describe, with the support of an object they are asked to bring, a sixth sense or super power that they would like to have. Objects (e.g. a pair of glasses) are passed around from participant to participant during description. The use of an object draws inspiration from the combination of two approaches: show and tell and object brainstorming (see above for explanations of these). The use of objects was a deliberate one to provide a tactile way to share and reference ideas.

This initial activity is then developed through asking each participant to choose a sixth sense that is not their own (referenced through the objects) and a talking button that speaks pre-recorded text. Talking buttons have place names on them, such as: at home, at the mall, at work, at the temple, etc. Participants are then asked to imagine how they would use their new sixth sense in that particular location. To keep their hands busy and create something that can be shared with the group, each participant is asked to create an accessory for their chosen sixth sense from a lump of clay given to them. Participants are then asked to discuss and hand their accessories to each other, before choosing one to give to the participants in another group.

The aim of this task, similar to ideation cards, is to juxtapose the unexpected and to stimulate creative thinking beyond a person’s own initial ideas. The use of clay draws upon the research from Lego Serious Play that proposes that making with the hands inspires different kinds of the ideas . The choice of making an accessory is intended to take the focus off having to find a form for the sixth sense. The need to choose a single concept from all of those made by the participants is intended to stimulate the discussion and prioritization of the benefits or disadvantages of a particular sixth sense. This approach was also envisioned as a means to examine the commonality of participants’ suggestions.

Finally, each group integrates the sixth sense and scenario received from the other group with an existing technology that they currently use to make a new technology. This can be either an entirely digital technology, such as an app, or a physical technology, such as a liquid level meter. The session finishes with both groups sharing their final concepts. This final activity provides an opportunity for the facilitator to explore current technology use. It also brings the sub-groups together to discuss all the generated ideas.

Activity Set 2

While the first set of activities is bottom up, with no constraints around the technology; the second set of activities is top down, ideating around a specific set of technologies, explicitly exploring opportunities for artificial intelligence, with a particular focus on computational vision. This set of activities starts with a discussion of what artificial intelligence technologies are capable of now and what is predicted they might be capable of in 10 years' time.

Participants are then asked to design a technology that they would like to use from a set of widgets given to them. In the first round, they are asked to choose one artificial intelligence (AI) widget and one output widget. The AI widgets were previously decided upon during a workshop with computational vision researchers, and include: a person recognizer, an object recognizer, an object aligner, and a room mapper. The output widgets include: speech, vibration, 3D audio, and tactile display. For example, participants may create a system that recognizes the alignment of two objects and provides feedback via vibration. These widgets are recorded on a talking button and given to participants in a shared 'craft box'.

Participants are further encouraged to use their own personal box of craft materials to illustrate a scenario in which their invented technology would be used. Included in the craft materials are: pipe cleaners, paper clips, clay, balloons, lego figures, a safety blanket, foam, and double-sided sticky pads. The making approach is in line with the theories from Lego Serious Play, with individual boxes aimed at reducing the problem of materials being too far away to reach or undiscoverable, a problem raised in the literature review.

After 15 minutes, people can add a second widget. The final designs, are then passed around on trays, described, and discussed across both groups.

UK Cohort

Six participants (2 women) were recruited through personal contacts of the lead researcher who is an active member of the local blind community. They were chosen to represent the diversity of the blind community. Participant ages ranged from 8 (represented by his mother) to 60 years old; vision levels ranged from none to ability to read adapted text; and both early and late blind were included. All participants were heavy technology users and could be seen as early adopters.

The ideation workshop took place at a research lab in Cambridge, UK. Participants worked in groups of three at small round tables for the first activity set. For the second, they sat in the same groups at larger rectangular tables, as more space was needed. The session was intended to last three hours, but intense discussions made it last more than four. Each group had a facilitator.

Participants in tactile ideation workshop in the UK
Tactile ideation workshop in the UK. (Left) Object-based show and tell; (Right) Craft-based scenario generation.
Women and men participating in tactile ideation workshop in India
Tactile ideation workshop in India. (Left) Women's breakout with facilitators; (Right) Men's breakout with a participant.

India Cohort

Eight blind and partially sighted individuals (4 women) were recruited from Enable India, a charity that teaches computer skills (e.g. keyboarding, and screen reader use) along with workplace skills (e.g. interacting in a sighted workforce). Participants ranged in age from late teens to thirties; sight levels ranged from adapted text to no sight; and included early and late blind users. All were learning to use technology to gain better jobs. Three were smartphone owners.

The workshops took place in a research lab in Bangalore, India. Participants were split into two rooms by gender, to enable a freer discussion in a culture where gender plays a strong mediating role. Each group sat around a small round table with a facilitator. While all participants spoke English, some were more confident than others. Local languages such as Kanada and Tamil were also spoken during the workshop, particularly in the women’s group. Several helpers sat around the outside of the table, helping with translation.

The ideation tasks were adapted based on the experience of the UK session as well as practical need. This workshop focused entirely on activity set 1 due to the resistance we saw in the UK workshop to using clay and craft materials (as discussed below). As we did not have direct contact with participants beforehand, we supplied various objects for Activity Set 1. These included: pipe cleaners, pebble magnets, survival blanket, stress ball, blue-tac, and paperclips. Participants used these to help think of a sixth sense. The places used in the place activity were adapted to be culturally relevant: Market, Temple, Relative’s house, and Work or School.

Data Synthesis

All of the workshops were video and audio recorded. The facilitators of each group also took notes during and after the session. Each workshop group had two people draw out the ideas which were then placed in a spreadsheet. Related context, such as use examples of proposed ideas or details about the participants’ background that helped interpretation, were also included. The authors worked together to cluster the data in meaningful ways. This was an iterative process informed by immersion in the literature as well as other related studies with people with visual disabilities. Permission was given for the use of photos in publications.

Findings

The two ideation sessions were both highly generative, spawning a large and varied number of ideas. Some of those ideas were direct descriptions of what a technology would do: “identify an official at an office or summon a guide” (Ip7). Others described how a technology might do something: “The accessory would transfer the patterns in the environment to me in a silent way, not involving vibration or audio. I want it to go directly to my brain.” (UKp5). Many of the ideas came as part of stories. Those stories communicated how a particular need would impact the participants; or, the participants integrated a variety of ideas generated through the iterative activities into a single proposed scenario. Below are two such examples, respectively.

A sixth-sense to tell me when I’ve offended someone as I can’t read people’s reactions anymore. You wouldn’t know if you queue jumped and someone was offended. People are likely to tell you when you are pleased, but may try to hide their frustration or anger. (UKp3)
The system already knows who your relatives are (because you interact all the time). Detect changes in furniture from last time visited. Who is present today? Is cousin there? Facial expression may indicate that family folks aren't that interested in having you around. (Ip6).

Most notable about the ideas generated is that the majority mention people either as objects of identification or as part of the story. If we break down our data into singular ideas and remove duplicates from the same group, we count 66 ideas of which 28 are from the UK. Four of the imagined sixth senses were not specific to people with a visual disability, focusing on “knowing the future” or “diagnosing illness through feeling the hand.” Forty of the remaining 62 ideas mentioned people in some way. While these are rough estimates given the difficulty of quantifying “an idea,” they do give a sense of how pervasive and important people are in the ideas that surfaced. We focus the remaining analysis on articulating the different ways people feature in our data set.

Identifying People

Many of the examples focused on identifying and locating known people. Some participants wanted to identify friends in a temple (Ip4), or know when their manager was passing by at work (UKp4). These two examples are illustrative of a range in which other strategies, such as voice or handshake recognition, could not be used because of social protocol. Two other situations were singled out as challenging for identifying people: networking events and chance meetings on the street. The noise and crowd of a networking event made navigation to sought-for people difficult. On the street, there was little in the way of context to help with sense-making and surmising who might be around.

When networking, it can be hard because you know people are in the room, but you don’t know where (UKp1).

As illustrated through these examples, identifying people had a number of purposes beyond engagement with a person: First, some of our participants pointed out with a wry sense of humor that identifying people also enables their avoidance. Avoiding people is something a person with a visual disability cannot easily do. Second, in an extended example from the India workshop, identifying which relatives were in the house when looking for “cousin” was desired (Ip8). This provided a social context to ascertain the socially appropriate manner for engaging with a cousin, even if the other people were not the intended focus of the visit.

Identifying people was not limited to a known person. It was also important to identify people routinely in the same environment. Participants spoke about how they often felt disconnected from the communities in which they lived and worked.

I live in a village. People know what I look like and they will often say hello to me, but I have no clue who they are. If they come and speak to me, they say, I’ve known you for 20 years. But I haven’t known you for 20 years, you’ve never spoken to me before. (UKp5)

There was almost a fascination with how sighted people could meet each other without directly interacting just by being in the same space. These examples stretch the idea of people we know by name to those we know by sight, or the familiar stranger.

Not least, there were several examples in which the person's role was more important than identity. In the most direct sense, there was a need to identify assistance when entering a building (Ip7). This could also extend to temporary roles, such as finding people who may be going to lunch (UKp4). The most general form of this was understanding who is around. For example, understanding where people are walking and praying in a temple enables navigating around them (Ip4). Equally, people ubiquitously wanted to know where people were absent, in order to find an empty seat.

Managing Social Interactions

Beyond the identification of people, many examples focused on the identification of social cues to enable the management of social interaction. Some participants emphasized the desire for the low-bandwidth communication provided by eye-contact (UKp5). Another participant pointed out that it provides a back channel for communicating with someone when others are present (UKp3). This is captured in the example below:

I want to be able to look at [blind son] across the room to let him know that he should stop what he is doing without drawing everyone’s attention by speaking aloud. (UKp6)

Others wanted a more sophisticated way to read reactions to modulate their own behavior, whether it be in a doctor-patient relationship or just with family.

A way to know how someone is responding when I’m breaking bad news [as a doctor] in a hospital context. (UKp2)
Relatives aren’t always that interested in having you around. It would be useful to gauge attention and interest from them in a conversation. (Ip6)

Interestingly, people were more concerned with getting negative cues rather than positive, pointing out that people are more forthcoming when they are pleased, but attempt to be neutral when displeased. This attempt at understatement made reading intent from audio cues alone far more challenging. Finally, people wanted to access non-verbal cues critical for interaction, such as an extended hand or a head nod. Non-verbal cues also extended to understanding attention.

I want to understand that the priest has extended his hand with an offering of Prasad. (Ip1)
Suppose we have gone to a vendor to buy some stuff and we are keeping on telling him something, but he is talking to the customer beside us. So we are not able to understand whether he is talking to *me* or the person beside me. (Ip5)

While one group in the UK mused over why people continued to use such non-verbal interactions with people they knew could not see, the groups in India did not question the phenomenon, assuming it was their responsibility to fit in socially.

Social Stories

There were also a large number of examples in which people featured prominently in stories, but were not necessarily the object of recognition. While the quotation below has seemingly little to do with people, unpacking it with the participants led to a discussion around conversational participation. When out-and-about, it was common to fall into a relationship in which the world was described by the sighted person and the description absorbed by the visually disabled person.

Here are a pair of glasses and they are magical, or technical, same thing. They can do pattern detection. I am very competitive and no matter what the situation my wife or colleagues can do better when it comes to matching patterns. (UKp4)

Conversations, when out-and-about, are often about visual similarity – “that is similar to the houses we saw in Sweden last year” – or visual difference – “that shop is now closed.” Our participants felt that the more visual cues they had, the more opportunity they had to initiate conversation.

On the surface, the next example may describe the mechanistic challenge of recognizing and distinguishing coins, yet the social story underling this instance rather highlights the desire to take part in activities without disrupting social norms.

At the temple, we find it hard to recognize the coins (1 Rupee or 2 Rupees) to offer during the prayers to the priest. It is also hard to distinguish notes to offer. With shops, we can always ask what note it is and exchange it with the right one if wrong. But at a temple, we are embarrassed to ask and exchange notes or coins. We want to be able to tell accurately. (Ip2, Ip3)

This example illustrates that the motivation for a simple recognition technology is influenced by the social setting in which it is required. While such technologies may be useful in a range of places, our examples highlighted how social spaces often reduce the availability of other strategies to gain information, making social participation more difficult. Social participation also took on a new shape with examples in which the imagined technology arose from wanting to avoid being taken advantage of.

I want a talking ATM. My friends or relatives help me now, but sometimes they take a tip. (Ip3)
The seller gives me something other than what I ask for. I tell him that this is not right, but he doesn’t believe me. I need something to prove that I'm right. (Ip1)

These examples show that social participation is not only a matter of desire, but also of necessity. The ability to demonstrate competence and “normality” is a key driver of informational need in some circumstances. While we only saw such examples in India, we know that there is related research that suggests a similar need in Western contexts, such as the demonstration of professionalism among blind people at work .

Socially motivated technology use also came from the need for social independence. Some of our examples highlight that a lack of information keeps young people from gaining the social independence they desire as illustrated in the quotation below.

My parents do this now, but I’d like my phone to be able to tell me about obstacles or steps (Ip8).

Most of the examples, like this one, asking for practical solutions did not reference people, but could be seen as socially motivated. While gaining social independence is perhaps the opposite of social participation, they are linked in that independence for mundane tasks enables effort to be put towards inherently social interactions unencumbered by need, creating an equity in interaction.

People across Cultures

The most striking aspect of our data is the similarity of ideas generated across the two workshops. The UK and India have substantial cultural and infrastructural differences. We had expected that this might lead to different ideation results, but this was largely not the case. The only differences were: the UK had ideas unrelated to visual disability; and India started the conversation with very direct day-to-day needs people wanted met, such as better walking directions. However, once settled into the activities, the ideas across the two localities became much more similar. The most compelling examples are found in those quotations which mirror human desires across cultures and locations.

If I'm talking to someone and they're not that interested in speaking to me, their facial expression will show: So you can finish the conversation quickly. She's just not that into you. (Ip5)
A way to read emotions during the [name] therapy groups; when girls are eyeing me. I can guess a lot less about what people think now that I cannot see their faces. (UKp1)

Discussion

We have presented the method and synthesis of findings from ideation workshops with visually disabled people in the UK and India in order to understand how this user group might imagine their own future with (artificial) intelligent agents. We were surprised to find that when taking an ideation approach agnostic to technology and current everyday needs, our participants focused on technologies that could help them in their social encounters. In some cases that was a matter of recognizing people. In others, it meant being able to participate in social situations without their disability being obtrusive. It was striking that this was consistent across the UK and India, despite substantial cultural and infrastructural differences.

In the discussion, we tease out what the social dimensions of the lived experience of this user group might look like in a design process. We also comment on the particularities of the tactile ideation workshop method and its role in forefronting social interaction in our findings. It must be said that the findings apply more broadly than the original topic of intelligent agents and can be addressed potentially by a wide range of technologies.

Enabling Social Experiences

The most striking aspect of our workshops was the strong, cross-cultural focus on the sociality of the lived experience. This stands in contrast to the political motivation of access and accessibility often referenced in the technology community. While access to education, work, or culture mediated through technology is critical, we should not forget that peoples’ lives are situated socially. In our workshops, the desire for nuanced communication and interaction with other people rose above the more practical challenges that our participants undoubtedly faced, such as getting to work.

This strong focus on social situations in the workshops was underpinned by a set of stories that suggested examples in which existing strategies were not sufficient. All our participants in the UK and most in India had strategies for getting to work and other daily activities. They were also skilled in using broader resources of context to infer social behavior . However, participants developed strategies that often broke down when there was a need to respect formal social structure. Indeed, participants felt that eliciting social information in many social settings foregrounded their disabilities in an undesired way.

Identifying people and their associated attributes was a prominent theme in our findings: knowing who is around, who is a familiar stranger, or who is in an official role. Our findings also highlight that sociality for our participants extended beyond recognizing people and their attributes. Participating socially often meant doing what others are doing, simulating a range of visual capabilities, such as object or text recognition. While this may be seen as simply an access issue, it is the social context which shapes why and how some of these needs might be met with technology. It may not be appropriate to use an expensive mobile phone in an Indian temple.

It is possible that social experiences featured strongly in our ideation workshops because of the inherently human implications of positive and negative social interaction for one’s sense of self . People are strongly motivated by loss aversion, and social awareness can provide a safety net for our behavior. Previous research has shown that avoiding deviations from social norms features strongly among uses of technology envisaged by some people with visual disabilities . Our findings suggest the same, prompting us to consider the design space of enabling social experiences in assistive intelligent technologies.

To support designers and technologists in thinking about what it might mean to enable social experiences, we identified social dimensions that were prevalent across our data.

Three of these dimensions span activities which enable meaningful social interaction: Social Participation, the process of participation in social interaction through shared grounding within (knowledge of) the general social context; Social Navigation, the process of identifying and entering into opportunities for social interaction; and Social Maintenance, the process of managing engaged interaction through knowledge of other participants’ social cues.

A fourth category, Social Independence, emerged as an important factor in enabling and motivating these activities (see Table 1). These dimensions are highlighted with the intention of broadening future thinking in ideation and design practice. By focusing on these actions and their meanings, we attempt to ensure that we design technologies that go beyond meeting people’s functional needs, to include those social needs that make us inherently human.

Categories of social activities and motivations
Axis Definition Example
Social Participation The ability to participate in a given social interaction. At the temple, we find it hard to recognize the coins to offer during the prayers to the priest. … But at a temple, we are embarrassed to ask and exchange notes or coins. We want to be able to tell accurately.

Visual cues of the environment provided the opportunity to initiate conversation.
Social Navigation The process of identifying and entering into opportunities for social interaction. When networking it can be hard because you know people are in the room, but you don’t know where.

Finding people who look like they are going to lunch.
Social Maintenance The process of managing interaction through knowledge of other participants’ social cues. I want to understand that the priest has extended his hand with an offering of Prasad.

A way to know how someone is responding when I’m breaking bad news [as a doctor] in a hospital context.
Social Independence The ability to be free from the constraints of social interaction through independent abilities. I want a talking ATM. My friends or relatives help me now, but sometimes they take a tip.

My parents do this now, but I’d like my phone to be able to tell me about obstacles or steps.

This design space, with compelling initial explorations (e.g. social interaction assistant ), must be approached with nuance. Literature, for example, has already alluded to the challenges of expressing continuous aspects of non-verbal interactions (e.g. facial expression) with the labelled classes machine learning systems can produce . Indeed, mapping the visual recognition of identified people in space to an audible representation raises a host of questions about how location, space and identify are co-constituted (see, for example, ). For those with little to no sight, there may be significantly different notions of people in space that are not easily aligned with visual modes of recognition. These insights remind us of the importance of social accessibility , both in the use of technology and priority in designing it.

A Reflection on Method

The ideas generated in the workshops undertaken are without doubt shaped by the methods that we used. A key element of the method was to focus on encouraging participants to imagine a sixth sense rather than a technology per se. Interestingly, participants in the UK did not contain their ideas to visual disability; many wanted to predict the future. In India, the participants started out focused on daily challenges they experienced ignoring the idea of a sixth sense. Gradually, through the layering of exercises that built on and changed previous ideas, our participants started to reach beyond what they thought was possible, to what they would really like.

It was striking that in the UK, in which the sixth sense activities were followed by activities imagining a new technology, this lead to dramatically different ideas. In the first set of activities, all the participants focused on ways of connecting with others. In the second , all participants built navigational and mapping tools. We can surmise that this dramatic change may illustrate the difficulty people have in imagining the form of new technologies or how they’d work. It may have been also an issue of the difficulties of prototyping without vision. One participant said: “I’m only doing maps because they are more fun to make.” Regardless, it is clear that methods that draw people away from solving daily problems open up the space for imagining future technologies.

A substantial part of the workshop relied on physical objects as a means to support ideation and communication between participants. Objects worked well in both the UK and India as “fiddle” things: ways to keep the hands busy and not feel compelled to talk as ideas formed. In the UK, they also worked well for sharing concepts and helping people keep track of ideas. In India, some of the participants treated the objects quite literally: the safety blanket was like a parachute to get off a plane; blue-tac could be used to make art work with one’s children for school. The use of objects as prompts for lateral thinking seemed to be an unfamiliar idea to some. This likely has less to do with the use of objects, but rather familiarity with design-led methods for those with particular educational backgrounds .

Prototyping, the creation of new objects as a means to explore or present ideas, did not work well. It was enjoyed by those with partial-sight, but those with less vision found it difficult. While people enjoyed playing with the materials, wrapping a fluffy pipe cleaner around the neck or playing with clay as putty, it was difficult to put things together in a coherent scene. The spatial understanding needed taxed people in a way that did not encourage ideation. Making the materials easily available in personal boxes and providing bounded trays for the work was not enough. The clay also made people’s hands sticky, a problem if you use your hands to make sense of the world. It could be interesting to explore the adaptation of methods, such as invisible design, that elicit discussion through ambiguous film without ever showing the design .

We present in this paper the ideas generated through the ideation process; however, there was also a lot of insightful side talk. The sessions naturally encouraged people to volunteer information about their current technology use. We got, for example, several excellent comparisons between available technologies. Participants talked at length about the appropriate form-factor of devices. Not least, participants, both in India and UK, were very forthcoming about their thoughts and choices related to living with a visual disability. This openness built as the session went on, with some of the most poignant discussions at the end. We felt that tactile engagement worked well as a means for empathetic engagement to conduct enquiry into people’s lives without intruding .

Conclusions

As artificial intelligence matures, it becomes increasingly important to understand the kinds of things that people with visual disabilities would like to have as part of their tech toolkit. In this paper, through findings of tactile ideation workshops in both the UK and India, we highlight an underexplored space for imagining technologies for people with visual disabilities that forefronts the inherent sociality in which they live. As designers, technologists, and researchers work to imagine how intelligent technologies can partner with people to increase capabilities , we encourage a more deliberate focus on users’ social needs and desires.

References

D Adams, S Kurniawan, C Herrera, V Kang, and N Friedman. 2016. Blind Photographers and VizSnap: A Long-Term Study. In Proc ASSETS ’16. D Ahmetovic, C Gleason, C Ruan, K Kitani, H Takagi, and C Asakawa. 2016. NavCog: a navigational cognitive assistant for the blind. In Proc MobileHCI ’16. D Ahmetovic, R Manduchi, J M Coughlan, and S Mascetti. 2015. Zebra Crossing Spotter: Automatic Population of Spatial Databases for Increased Safety of Blind Travelers. In Proc ASSETS ’15. American Printing House for the Blind. Nearby Explorer. http://louis.aph.org/product/Nearby-Explorer,142793.aspx?FormatFilter=8 C Andrews. 2014. Accessible Participatory Design: Engaging and Including Visually Impaired Participants. In Inclusive Designing. Springer. M Avila, M Funk, and N Henze. 2015. DroneNavigator: Using Drones for Navigating Visually Impaired Persons. In Proc ASSETS ’15. S Azenkot, C Feng, and M Cakmak. 2016. Enabling building service robots to guide blind people: A participatory design approach. In Proc Human Robot Interaction HRI ’16. C M Baker, L R Milne, and R E Ladner. 2015. StructJumper. In Proc CHI ’15. J P Bigham, C Jayant, H Ji, G Little, A Miller, R C Miller, R Miller, A Tatarowicz, B White, S White, and T Yeh. 2010. VizWiz: nearly real-time answers to visual questions. In Proc UIST ’10. S Blessenohl, C Morrison, A Criminisi, and J Shotton. 2015. Improving Indoor Mobility of the Visually Impaired with Depth-Based Spatial Sound. In ICCVW ’15 IEEE Int Conf on Computer Vision Workshops. J Bornschein, D Prescher, and G Weber. 2015. Collaborative Creation of Digital Tactile Graphics. 117–126. In Proc ASSETS ’15. S M Branham and S K Kane. 2015. The Invisible Work of Accessibility: How Blind Employees Manage Accessibility in Mixed-Ability Workplaces. In Proc ASSETS ’15. S M Branham and S K Kane. 2015. Collaborative Accessibility : How Blind and Sighted Companions Co-Create Accessible Home Spaces. In Proc CHI ’15. P Briggs, M Blythe, J Vines, S Lindsay, P Dunphy, J Nicholson, D Green, J Kitson, A Monk and P Olivier. 2012. Invisible design: exploring insights and ideas through ambiguous film scenarios. In DIS ’12. A M Brock. 2013. Touch the map!: designing interactive maps for visually impaired people. SIGACCESS Access. Comput. 105: 9-14. J S Brown and P Duguid. 1994. Borderline issues: Social and material aspects of design. Human–Computer Interaction 9, 1: 3–36. M Campbell, C Bennett, C Bonnar, and A Borning. 2014. Where’s my bus stop?: supporting independence of blind transit riders with StopInfo. In Proc ASSETS ’14. A Fiannaca, I Apostolopoulous, and E Folmer. 2014. Headlock: a wearable navigation aid that helps blind cane users traverse large open spaces. In Proc ASSETS ’14. B Friedman and D Hendry. 2012. The envisioning cards: a toolkit for catalyzing humanistic and technical imaginations. In Proc CHI ’12. G Fusco and V S Morash. 2015. The Tactile Graphics Helper: Providing Audio Clarification for Tactile Graphics Using Machine Vision. In Proc ASSETS ’15. M Golembewski and M Selby. 2010. Ideation decks: a card-based design ideation tool. In DIS ’10. D Graham and T Bachmann. 2004. Ideation: The Birth and Death of Ideas. Wiley. D Gray. 2010. Gamestorming: A Playbook for Innovators, Rulebreakers, and Changemakers. J Guerreiro and D Gonçalves. 2015. Faster Text-to-Speeches: Enhancing Blind People’s Information Scanning with Faster Concurrent Speech. In Proc ASSETS ’15. Horus. https://horus.tech/horus/?l=en_us D Jain. 2014. Path-guided indoor navigation for the visually impaired using minimal building retrofitting. In Proc ASSETS ’14. H Kacorri, K M Kitani, J P Bigham and C Asakawa. 2017. People with Visual Impairment Training Personal Object Recognizers: Feasibility and Challenges. In Proc CHI’17. N J Kemp. 1981. Social psychological aspects of blindness: A review. Current Psyc Rev 1, 1: 69. S Krishna, D Colbry, J Black, V Balasubramanian and S Panchanathan. 2008. A systematic requirements analysis and development of an assistive device to enhance the social interaction of people who are blind or visually impaired. In Wkshp on Comp Vis Apps for the Vis Impaired. E Luger and A Sellen. 2016. Like Having a Really Bad PA: The Gulf between User Expectation and Experience of Conversational Agents. In Proc CHI ’14. E Luger, L Urquhart, T Rodden, and M Golembewski. 2015. Playing the Legal Card: Using Ideation Cards to Raise Data Protection Issues within the Design Process. In Proc CHI ’15. R Manduchi and J M Coughlan. 2014. The last meter: blind visual guidance to a target. In Proc CHI ’14. D Massey. 2013. Space, place and gender. John Wiley. O Metatla, N Bryan-Kinns, T Stockman, and F Martin. 2015. Designing with and for people living with visual impairments: Audio-tactile mock-ups, audio diaries and participatory prototyping. In CoDesign 11(1), 35-48. Microsoft. 2016. https://blogs.msdn.microsoft.com/accessibility/2016/04/07/seeing-ai/ Microsoft. 2014. https://blogs.windows.com/devices/2014/11/14/microsoft-research-3d-soundscape-technology-helps-visually-impaired/#ODbbB3H3y22YqHsD.97 M R Morris, A Perkins, C Yao, S Bahram, J P Bigham, and S K Kane. 2016. “With most of it being pictures now, I rarely use it”: Understanding Twitter’s Evolving Accessibility to Blind Users. In Proc CHI ’16. L Murray, P Hands, R Goucher, and J Ye. 2016. Capturing social cues with imaging glasses. In Proc Ubicomp ’16. Orcam. http://www.orcam.com/ S Panchanathan, S Chakraborty and T McDaniel. 2016. Social Interaction Assistant: A Person-Centered Approach to Enrich Social Interactions for Individuals With Visual Impairments. IEEE J Sel Top in Signal Processing, 10, 5. S Qiu, M Rauterberg, and J Hu. 2016. Tactile Band: Accessing Gaze Signals from the Sighted in Face-to-Face Communication. In Proc TEI ’16. Rasmussen Consulting. 2012. The Science Behind the LEGO SERIOUS PLAY method. Retrieved from http://seriousplayground.squarespace.com/storage/The Science Behind the LEGO SERIOUS PLAY Method.pdf M Ratto, I Record, G Coons, and M Julien. 2014. Blind tennis: extreme users and participatory design. In Proc PDC ’14, 2, 41–44 K Shinohara and J O Wobbrock. 2016. Self-Conscious or Self-Confident? A Diary Study Conceptualizing the Social Accessibility of Assistive Technology. ACM Transactions on Accessible Computing (TACCESS) 8, 2: 5. N G Sahib, T Stockman, A Tombros, and O Metatla. 2013. Participatory design with blind users: a scenario-based approach. In Proc INTERACT ’13. K Shinohara and J O Wobbrock. 2011. In the shadow of misperception. In Proc CHI ’11. L Stearns, R Du, U Oh, Y Wang, L Findlater, R Chellappa, and J E Froehlich. 2014. The design and preliminary evaluation of a finger-mounted camera and feedback system to enable reading of printed text for the blind. In Euro Conf on Computer Vision. D Swann. 2011. NHS at Home: Using Lego Serious Play to Capture Service Narratives andEnvision Future Healthcare Products. INCLUDE 2011 Proceedings. http://eprints.hud.ac.uk/13355/1/SwannpdfF358_1689.pdf R Talhouk, S Mesmar, A Thieme, M Balaam, P Olivier, C Akik, H Ghattas. 2016. Syrian refugees and digital health in Lebanon: Opportunities for improving antenatal health. In Proc ASSETS ’16. B Taylor, A Dey, D Siewiorek and A Smailagic. 2016. Customizable 3D Printed Tactile Maps as Interactive Overlays. In Proc ASSETS ’16. A Taylor. 2017. Becoming more capable. In Platypus, the CASTAC Blog. Ret April 27, 2017 from http://blog.castac.org/2017/04/becoming-more-capable J Vines, M Blythe, S Lindsay, P Dunphy, A Monk, and P Olivier. 2012. Questionable concepts: critique as resource for designing with eighty somethings. In Proc ASSETS ’12. J Wallace, PC Wright, J McCarthy, DP Green, J Thomas, and P Olivier. 2013. A design-led inquiry into personhood in dementia. In Proc CHI ’13. M A Williams, E Buehler, A Hurst, and S K Kane. 2015. What not to wearable: using participatory workshops to explore wearable device form factors for blind users. In W4A ’15, Web for All Conference. M A Williams, C Galbraith, S K Kane, and A Hurst. 2014. “just let the cane hit it.” In Proc ASSETS ’16. C Wölfel and T Merritt. 2013. Method card design dimensions: a survey of card-based design tools. In Proc INTERACT ’13. World Health Organization. 2013. Factsheet on Visual impairment and blindness. Retrieved from http://www.who.int/mediacentre/factsheets/fs282/en/ S Wu, J Wieland, O Farivar, and J Schiller. 2017. Automatic Alt-text: Computer-generated Image Descriptions for Blind Users on a Social Networking Service. In Proc CSCW ’17. H Ye, M Malu, U Oh, and L Findlater. 2014. Current and future mobile and wearable device use by people with visual impairments. In Proc CHI ’14. Y Zhao, S Szpiro, J Knighten, and S Azenkot. 2016. CueSee: exploring visual cues for people with low vision to facilitate a visual search task. In Proc Ubicomp ’16. A Zolyomi, A Shukla, and J Snyder. 2016. Social Dimensions of Technology-Mediated Sight. In Proc ASSETS ’16. H Zou and J Treviranus. 2015. ChartMaster: A Tool for Interacting with Stock Market Charts using a Screen Reader. In Proc ASSETS ’15.