Skip to content

“Digital Assistant” Devices Can Monitor Your Showering Habits.

December 16, 2017

When one thinks of “digital assistants,” such as Amazon Echo and Google Assistant, one might automatically think of these devices as being used in private, in one’s home. However, these devices are also being promoted as useful in the classroom (e.g., see here and here and here).

Although one might view digital assistants as merely helpful, hands-off operators of one’s other digital devices, there’s more to the story of who, exactly, is being “helped.”

It seems that those digital assistants are also created to gather information on their users, for the benefit of the corporate world.

In December 2017, Consumer Watchdog posted this summary of patents filed by Google and Amazon related to their digital assistants’ information-gathering capabilities. Consider the following, keeping in mind the privacy concerns (and associated liabilities for teachers and schools) that such capabilities pose in the classroom setting:

A new study reveals that Amazon and Google have filed patent applications for a number of technologies that would dramatically expand surveillance of consumers’ private lives. These patent applications show how technology companies use home data to draw disturbing inferences about households, and how the companies might use that data for financial gain. The technologies detailed in the patent applications include:

  • A system for deriving sentiments and behaviors from ambient speech, even when a user has not addressed the device with its “wakeword.” In 2014, as Amazon was preparing to roll out its Echo smart speaker, the company patented an algorithm that could listen to human speech, including phone conversations, for statements of interest such as “I love skiing.” The algorithm would then process the statements into key words that could be used to target advertising. The patent describes transmitting the keywords to Amazon servers as text, which could allow the company to spy on conversations while technically keeping its promise to only store and analyze audio recordings that a user intends to share.
  • Multiple systems for identifying speakers in a conversation and building interest profiles for each one. Both Google and Amazon offer users the option of creating acoustic “voice profiles” for voice-activated smart devices in their homes. These profiles can help the devices tailor services to the person speaking. Patents show that both Amazon and Google could also use voice profiles to associate behaviors with individual members of the household, in order to better target ads.
  • A method for inferring users’ showering habits and targeting advertising based on that and other data. Dozens of patent applications for Google’s smart home devices detail scenarios in which Google may share data from smart home devices with third parties, including businesses, who can then use the data to make inferences about users’ sleeping, cooking, entertainment, and showering schedules. These inferences, Google says, “may help third-parties benefit consumers by providing them with interesting information, products and services as well as with providing them with targeted advertisements.”
  • A system for recommending products based on furnishings observed by a smart home security camera. A particularly troubling patent application describes how Google could use video feeds from smart devices to determine users’ behaviors and characteristics, including “gender, age, fashion-taste, style, mood, known languages, preferred activities, and so forth.” The patent application describes smart devices that target advertisements based on the title of the book by a user’s bedside, the presence of a guitar or basketball in the room, and the face of a famous actor on a user’s t-shirt.
  • A methodology for “inferring child mischief” using audio and movement sensors. Another Google patent describes a smart home system that monitors the activity of every member of a household and reports back to a designated “policy holder.” One application uses voice fingerprinting to identify the presence of children and motion sensors to detect activity, and reports inferred mischief when children are both active and quiet. This patent also mentions advertising applications.
  • Systems for inserting paid content into the responses provided by digital assistants. Both Amazon and Google have patented methods for serving ads to users through their smart devices. An Amazon discusses a system of “intelligent sponsorship based on knowledge generated by reference to the human being entity using the system.” A Google patent shows how advertisers could bid to have the company’s digital assistant feature their products in its voice-based search results.

While Amazon and Google may be focused on the commercial implications of their inventions, these technologies also have troubling legal and ethical implications. Smart devices with cameras and microphones inevitably capture video or audio of people who did not consent to being recorded. Third parties can also access data from smart devices; a recent study found that hackers could easily access insecure wifi-enabled “smart” toys and talk to children. Law enforcement has also begin to subpoena data from smart devices in criminal investigations.

The 15-page Consumer Watchdog summary also includes some enlightening information on the background and capabilities of the Amazon Echo and Google Assistant and is well worth a full read. However, here is a final bit that caught my particular attention:

Many other patents for Google’s smart home products contain boilerplate describing an “advertising/communication paradigm […] that estimates characteristics (e.g., demographic information), desires and/or products of interest of a user based on device usage.”

One patent application, ironically named “Privacy-Aware Personalized Content for the Smart Home,” goes even further:

“In some embodiments, the robot may use its sensors to measure sounds and appearance of people and/or objects in the room and send audio and/or visual features that can be used to assess properties of the objects and/or people. For example, for people, the audio and/or visual features may be used to ascertain gender, age, fashion, taste, style, mood, known languages, preferred activities, and so forth.”

The application goes on to describe marketing sports camp to a 15-year-old boy holding a basketball in the living room, combining browser search history with an image on a user’s t-shirt to infer an interest in the actor Will Smith, and recommending a TV program based on the book on a user’s bedside table. …

Google’s smart home patent applications also describe the company’s interest in users’ health data. Several applications describe creating an audio signature for the master bathroom that will record the number of times that a toilet is flushed. Another application describes monitoring a user’s nighttime breathing patterns to detect a sleep disorder, and storing the information in a user’s “sleep profile.” While these patent applications do not describe systems for advertising to users based on this information, Google has demonstrated a willingness to advertise to users based on health data derived from other products. For example, a 2015 patent describes a scheme to advertise vacations to individuals whose heart rates indicate that they are under stress.

Google’s vast repository of information about its users has made the company the single biggest recipient of digital advertising revenues.

Information is power, and digital assistants are the power harnesses delivering constant, detailed, highly personal information into the hands of those who want it simply to make a buck.

And whereas it might seem neat to have a talking, computerized assistant overseeing your other electronics, even helping you in your classroom, keep in mind that the device is also monitoring you, gathering massive information about you, even in places where you believe your privacy is a given.

shower head


Want to read about the history of charter schools and vouchers?

School Choice: The End of Public Education? 

school choice cover  (Click image to enlarge)

Schneider is a southern Louisiana native, career teacher, trained researcher, and author of two other books: A Chronicle of Echoes: Who’s Who In the Implosion of American Public Education and Common Core Dilemma: Who Owns Our Schools?. You should buy these books. They’re great. No, really.

both books

Don’t care to buy from Amazon? Purchase my books from Powell’s City of Books instead.

  1. Laura H. Chapman permalink

    Spooky surveillance, and moving so fast it is hard to keep up.
    Thanks for the link to the Consumer Report summary.
    Here is a “business intelligence” website that reports on patents for more companies than are covered by the Consumer Reports. There is a steep fee for full access: the amount assumes that users are venture capitalists. There is enough information to launch your own research, especially about patents. Once you have a patent number it is still easy to get the Patent Office report and all of the drawings/diagrams.
    This is one of several websites tracking investments by venture capitalists, including some of the proliferating financial products known as social impact bonds (pay for success contracts) and the emerging use of blockchain technologies. When you are not busy ( smily face) check out how personalized learning badges and certificates help to make students into marketable “capital.”

  2. A few questions although you probably won’t hold yourself out as an all knowing tech guru, you have certainly done your homework and are trustworthy –

    Can one disable the device (or the app) when it is not being used for specific purposes (music, information etc as opposed to controlling or monitoring other household devices) to avoid its constant data collection?

    I understand there is no camera within the device itself (Echo Dot or Alexa). It has to be connected to a home surveillance system to video record, correct?

    Although a teacher may not have one of these devices in her/his classroom, it’s my understanding that there are numerous educational computer software programs that are being used that also have these data collection capabilities. Even if the teacher/students are not using such a program at any given moment, if the computer is on, that software is active – correct?

    • Lee, I do believe the digital assistant does not record people and must rely on the technology to which it is connected. I do not believe it is possible to completely neutralize it, especially if it is always on in a standby mode.

  3. Thanks for this update, Mercedes. If you haven’t yet checked out the #surveillancecapitalism or #longviewed hashtags, consider taking a look. They’ve got some good folks calling out these kinds of things. Your analyses would fit well.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s