Carnegie Mellon researchers use Twitch to collect sounds for AI research

Carnegie Mellon researchers designed a live-streaming video game to collect audio from players that’ll populate a database for AI research. The team’s game — Rolling Rhapsody — is specifically designed to be played on Twitch, and it tasks streamers with rolling a ball across a map to collect “treasure” while viewers record sound from their homes via an app.

It’s researchers’ belief that recordings of domestic sounds like thudding from a bedroom door or a coughing fit could be used to create a range of useful technologies. For instance, Google drew on audio from thousands of its own meetings and YouTube videos to train the noise-canceling algorithm in Google Meet. Meanwhile, a separate team of Carnegie Mellon researchers created a “sound-action-vision” corpus to anticipate where objects will move when subjected to physical force.

Rolling Rhapsody recently completed a fifth playtest with popular Twitch streamers. In coordination with the researchers, the streamers prompted viewers to record and upload sounds from Android and iOS companion apps. Those sounds were played back during the stream each time the streamer collected a piece of treasure, so that everyone watching could listen in unison.

“Imagine being able to collect 500 microwave sounds in 10 minutes by reaching just one person — Twitch makes this possible,” the team explained in a blog post on the project’s website. “It had more than 15 million daily active users and between 2.2 and 3.2 million monthly broadcasters in 2018. For each streamer we reach, we get many viewers because the amplifier effect takes place … It enables the ability to collect thousands of sounds from the viewers.”

Rolling RhapsodyRolling Rhapsody

Above: The Rolling Rhapsody companion app.

The hope is to one day make the sounds publicly available without compromising contributors’ identities. All Rolling Rhapsody players and viewers must opt in and provide consent to upload sounds, and the game gives them opportunities to redact sounds that accidentally capture something personal. They can also delete submissions, choose to store sounds locally, and withdraw their consent at any time

Rolling Rhapsody is by no means perfect — it requires players at home to label recorded sounds, and about 50% of them provide the wrong label. But work on it continues with a broader field test scheduled for later this summer.

“We can use this as a proof of concept for a new kind of game experience that can result in ethical data collection from the home. We can collect data in a way that’s fun and feels good for everybody involved,” lead researcher Jessica Hammer said in a statement.  “This research doesn’t have to be limited to gathering audio data for the home. A simple extension is gathering other kinds of audio data. Then you can use the same game, just change the kinds of challenges you give the players.”

Rolling Rhapsody, which is supported by sponsorships from Philips Healthcare and Bosch, is a part of Polyphonic, a larger Carnegie Mellon initiative that includes an app for sound labeling and validation and an interface where researchers can view and download sounds.

Huawei has the same app problem that doomed Windows Phone

Huawei is now the world’s biggest smartphone maker in terms of shipments, according to Canalys data, which suggests the Chinese tech firm surpassed rival Samsung in Q2 2020, albeit mostly due to its domestic sales in China. This is the first time in nine years that a company other than Samsung or Apple has led the market.

Huawei was quick to pounce on the news, particularly given the turbulent 14 months it has endured after the U.S. issued an embargo forcing it to stop using Google’s flavor of Android in its new handsets.

But despite Huawei’s glee at surpassing Samsung, its future in the global smartphone arena looks bleak, due in large part to the app restrictions enforced by the U.S. ban. The crux of Huawei’s problem, as most people by now know, is that buyers of its newer phones can’t access many big-name apps without having to jump through gargantuan hoops. In many ways, Huawei’s predicament is similar to that Windows Phone faced a decade ago.

Like Windows Phone


Above: Windows Phone 8 launch (2012)

Image Credit: Sean Ludwig/VentureBeat

At the time of its launch, Microsoft’s mobile operating system was often packaged with some of the best hardware the market had to offer, and Nokia’s devices in particular offered superb cameras. But Windows Phone just couldn’t ditch the “lack of apps” tag around its neck, though Microsoft did spend considerable resources convincing developers to build for Windows Phone, even holding developer competitions. In the end, Microsoft managed to persuade some companies to build for its platform, but the apps were often not as full-featured as their iOS and Android counterparts, and a fair chunk of them were rarely updated.

Similarly, Huawei devices are widely lauded for features like AI-infused chips and quality cameras. And the company is also trying to incentivize developers to join its platform, recently launching a $1 million contest that builds on its $1 billion Shining Star developer program.

In the year or so since the U.S. embargo came into effect, Huawei has doubled down on its development of Huawei Mobile Services (HMS), its own version of Google Mobile Services (GMS) designed to replace Google’s ecosystem and provide developers with tools to create apps for the Huawei platform. In a press briefing earlier this week, Jaime Gonzalo, VP of Mobile Services Europe for Huawei’s consumer business group, touted progress the company has made in the last year, revealing that it now has 1.6 million developers on board, an increase of 76% year-on-year, and more than 80,000 apps that now use elements of Huawei’s HMS.

But such growth metrics don’t really matter to most people. All they care about is whether they can readily access all the tools and services they want, something Huawei has attempted to address.

“In reality, there are 3 million apps out there, so this [81,000 apps] is less,” Gonzalo said. “[But] the thing is, it’s not about volume of apps, it’s about the pertinence and the services that these provide. So we evaluated our users, how many apps they use on their phones, and we found that no matter which country, the average number of apps is about 80 … what is the point of offering 3 million apps if many of them are clones of one another or bloatware?”

Although Huawei’s case for quality over quantity is valid, the real problem is that a vast chunk of the world’s most popular apps are simply not available through its AppGallery app store.

Delayed problem

In the immediate aftermath of the U.S. embargo, most Huawei phone users saw no difference in their app selection because the restriction only affects newer Huawei and Honor-branded handsets. Huawei has often downplayed the changes during recent product launches.

Without access to Google Play Services, anyone buying a new Huawei or Honor device today cannot easily access WhatsApp, Instagram, Uber, Uber Eats, Airbnb, Facebook, Google Maps, YouTube, Netflix, Twitter, Tinder, Dropbox, Slack, Amazon Prime, Spotify, eBay, Strava, or many others.

In certain cases, it is possible to access some of the aforementioned services through a browser, though often with less functionality. Huawei also recently introduced a new service called Petal Search, which enables users to search for the .APK files of Android apps either through the companies themselves (e.g. Facebook makes its Android apps available to download directly from its own website) or from third-party .APK libraries.

But this solution is unwieldy and deviates wildly from people’s expectations. Moreover, it’s not easy to provide timely, automatic updates when companies refresh their apps — a core function served by Google and Huawei’s respective mobile services.

Above: Huawei’s Petal Search links to .APK files of popular apps

There aren’t any barriers stopping companies — in the U.S. or otherwise — from adding their apps to Huawei’s AppGallery. Amazon, Microsoft, and Snap have already embraced AppGallery, which points to one of the key differences between the situation Huawei faces today and the one Microsoft faced with Windows Phone. Huawei devices are still based on Android, which makes it much easier for developers to get their apps into its ecosystem.

But Huawei is now racing the clock. The majority of its devices are currently in the hands of millions of people globally who can still access Google’s Android. Now the company is trying to get its house in order before those consumers begin upgrading to new devices. As people transition from older Huawei or Honor handsets to the fresh crop of devices, many won’t realize they’re buying into a completely different version of Android and will be left bitterly disappointed.

A decade ago, many less tech-savvy consumers were similarly impacted when they bought new Windows Phones for the camera only to discover they couldn’t access most of the apps their friends had on Android or iOS.

Online retailers are taking note this time. Some are posting prominent warnings next to new Huawei devices to ward off a backlash from confused consumers who can’t figure out how to install WhatsApp, Instagram, or Uber.

Above: Huawei smartphone for sale on U.K. website

Such websites also provide detailed guides for customers, explaining the various processes they can go through to bridge at least part of this Google-sized gap. But people don’t like workarounds and they don’t like friction, which is where Huawei’s rivals stand to benefit.

Indeed, Huawei may have surpassed Samsung as the top dog in global smartphone shipments, but Samsung could be the big winner when Huawei’s customers look for brands unencumbered by app restrictions. Huawei can probably achieve some success in markets that are less reliant on apps and services from Google and other U.S. companies, like Russia, but the company’s chances of remaining a global powerhouse are slim if it can’t make core apps easily available to its users.

Sure, there will always be some users willing to try alternatives to market-leading apps, but not at any meaningful scale. In the end, it doesn’t matter how good your on-device AI is, or how amazing your camera is. That is the painful lesson Microsoft and Nokia learned with Windows Phone all those years ago.

Google’s TF-Coder tool automates machine learning model design

Researchers at Google Brain, one of Google’s AI research divisions, developed an automated tool for programming in machine learning frameworks like TensorFlow. They say it achieves better-than-human performance on some challenging development tasks, taking seconds to solve problems that take human programmers minutes to hours.

Emerging AI techniques have resulted in breakthroughs across computer vision, audio processing, natural language processing, and robotics. Playing an important role are machine learning frameworks like TensorFlow, Facebook’s PyTorch, and MXNet, which enable researchers to develop and refine new models. But while these frameworks have eased the iterating and training of AI models, they have a steep learning curve because the paradigm of computing over tensors is quite different from traditional programming. (Tensors are algebraic objects that describe relationships between sets of things related to a vector space, and they’re a convenient data format in machine learning.) Most models require various tensor manipulations for data processing or cleaning, custom loss functions, and accuracy metrics that must implemented within the constraints of a framework.

The researchers’ TF-Coder tool aims to synthesize tensor manipulation programs from input and output examples and natural language descriptions. Per-operation weights allow TF-Coder to enumerate over TensorFlow expressions in order of increasing complexity, while a novel type- and value-based filtering system handles constraints imposed by the TensorFlow library. A separate framework combines predictions from multiple independent machine learning models that choose operations to prioritize during operations searches, conditioned on features of the input and output tensors and the natural language description of a task. This helps tailor the searches to fit the particular synthesis task at hand.



TF-Coder considers 134 tensor-manipulation operations of the 500 in TensorFlow including reshapes, filters, aggregations, maps, indexing, slicing, grouping, sorting, and mathematical operations. It’s able to handle problems involving compositions of four or five different operations and data structures of 10 or more components, which have little room for error as the shapes and data types must be compatible throughout.

The coauthors say that in experiments, TF-Coder achieved “superhuman” performance on a range of real problems from question-and-answer site StackOverflow. Evaluated on 70 real-world tensor transformation tasks from StackOverflow and from a production environment, TF-Coder successfully synthesized solutions to 63 tasks in 17 seconds on average and lead to “significantly” faster synthesis times (35.4% faster on average) compared with not using models. Remarkably, TF-Coder also produced solutions that the coathors claim were “simpler” and “more elegant” than those written by TensorFlow experts — two solutions required fewer operations than the best handwritten solutions.

“We believe that TF-Coder can help both machine learning beginners and experienced practitioners in writing tricky tensor transformation programs that are common in deep learning pipelines,” the coauthors wrote a preprint paper describing TF-Coder. “Perhaps the most important lesson to be learned from this work is simply the fact that a well-optimized enumerative search can successfully solve real-world tensor manipulation problems within seconds, even on problems that human programmers struggle to solve within minutes.”

Shadowlands looks to move World of Warcraft forward by learning from its past

World of Warcraft is gearing up for its next expansion, Shadowlands. After a lukewarm response from many players toward Battle for Azeroth, Shadowlands is looking to rebuild some goodwill between Blizzard Entertainment and its community.

Part of this involves looking at what exactly it is that so many disliked about Battle for Azeroth, which included complaints about unpolished experiences like Island Expeditions and Warfronts, a restrictive and annoying system of acquiring abilities through gear, and the difficulty of playing as more than one character for endgame content.

Earlier this month, I played the alpha test of Shadowlands and talked with the game’s technical director, Frank Kowalkowski. I wanted to know how Shadowlands is looking to stand out from the last few World of Warcraft expansions, and ask about new features like Torghast, a roguelike-inspired experience which has players exploring an unending tower.

This is an edited transcript of our interview.

After the battle

GamesBeat: What lessons did Blizzard learn from the last expansion, Battle for Azeroth (BfA)?

Frank Kowalkowski: There’s quite a few of them. One of them we’re undergoing right now, which is making sure that — we’re going to beta next week, and we feel like we have a first pass on all of our endgame systems ready to go and to show to players on the beta and our own development team, and we can begin receiving feedback. We feel like maybe in BfA, some of that played out well in the beta, with Azerite abilities and the neckpiece. We’re trying to get a bit of a head start on some of that to get feedback as soon as possible so we can continue to tune up until we release Shadowlands. The second one, which is again based a lot on feedback throughout BfA and based on the positive reception we got for both the Queen Azshara content drop and the Lost Visions of N’Zoth, was player choice, player agency. We’re doubling down on that. The covenant is the obvious example of that, but you’ll also get to choose crafting your legendary. We’re also going to give you more mains to be able to populate your weekly chest than we did in BfA as well.

GamesBeat: Covenants are a major feature in Shadowlands. You pick one of four, and that determines certain abilities and other features that you unlock. Are players going to be able to pick a covenant they want because it’s their favorite? Or will players be forced to pick certain ones because they are the strongest for their class.

Kowalkowski: We’re going to do a lot of tuning to make sure that we don’t have a, yeah, if you’re a demon hunter you have to pick this covenant. The covenant is about more than just the signature ability and the one power for your class. There are soulbinds that factor into the covenants, and other systems like Torghast that you’re going to be able to leverage to customize your character. It’s more than just the ability. That’s the one that’s up front because it’s the first thing players will see and experience, but there’s a lot of depth to the covenant system. We think it’s going to be up to each player to decide how they pick a covenant. You can pick it based on your play style, the types of things you want to do, but we also want to give you the option to pick a covenant for aesthetic or personal choice reasons. You identify with — I play a death knight, so maybe I identify with the Necrolords. I should be able to make that choice and not feel bad for doing that. There should be rewards and tuning such that I feel good about my choice.

A screenshot of the sky in the Maw in Warcraft: ShadowlandsA screenshot of the sky in the Maw in Warcraft: Shadowlands

Above: A player gazes at the sky in the Maw zone in Warcraft: Shadowlands.

Image Credit: Heather Newman

GamesBeat: For the last few expansions, players have earned new spells and abilities outside of traditional sources, like leveling and the talents system. Why is Blizzard so set on tying new features to unlocking abilities.

Kowalkowski: Again, choice. If I’m a death knight and I get an ability that’s cool because I hit level 60, that helps me a bit with my class, but if I’m looking at wanting to customize my character, you don’t necessarily get that if we just hand it to you. We want to give you abilities that are centralized around the theme and the fantasy of your class. What we have with essences or with what we’re doing in Shadowlands with the covenant abilities or your legendaries, we’re giving you the option to craft your character. That’s where we feel we gain a lot of ability for players to determine their own path and feel responsible for how they want to craft their character. It’s another choice, just like picking your race was. That gave you a few different abilities. Just as much as — choice is scattered throughout WoW. It starts the first time you log into the character select screen and pick Horde or Alliance. It’s about making sure that we’re giving that to players.

GamesBeat: One criticism of BfA was it wasn’t very alt-friendly. It was difficult to keep up with more than one character. Is Shadowlands going to be more accommodating of this?

Kowalkowski: That’s our goal. When you look, again — we learned a lot of lessons throughout BfA. We’ve put some of them into the current content that’s out there, the catch-up mechanisms. Making catch-up mechanisms is something we want to carry forward. When you make an alt in Shadowlands, for instance, you can pick a covenant right away if you want to begin developing that covenant. You don’t have to wait until maximum level. A lot of the renown building and things like that will have catch-up mechanisms built in, so you don’t necessarily have to follow the same progression path as your main character.

GamesBeat: There are some systems that have been in place since the Warlords of Draenor expansion, like the mission table. How are some of these older, more familiar endgame systems being shaken up in Shadowlands?

Kowalkowski: We created those as evergreen systems. If it’s been around for more than a couple of expansions and players expect it moving forward, we think of that as something we need to look at and refresh and do something different with. With world quests we wanted to incorporate them more into the covenant system. We did that through a system called callings, where a calling may send you out to do some activities in a zone, and if you choose to do world quests in that zone, that’s something you can do. World quests will still be out there and around, but they won’t be feeding necessarily into the emissary system, where you log into the game, hit the end key, and decide what you’ll go out and do that day, if the reward is worth it. We’re packaging that with the covenant system. The followers table is something where we’ve looked at it over the years. In BfA it felt very much like it was just match the symbol to the symbol. There was no strategy to it. Do I have those symbols available to do the thing, and if so, hit the button. Or keep adding people until I get to 100 percent. We wanted to rethink that, to give players the ability to have a bit more fun with it, and be able to try something different with the ability to put your own followers and troops and things into more of an interactive style of combat. When we talk about — we feel like that’s the next evolution in the follower system, rather than just trying to do the same thing we did before.

Above: Bastion is one of the new zones in Shadowlands.

Image Credit: Blizzard

Dead men tell tales

GamesBeat: How long ago did you decide to make an expansion about the afterlife? 

Kowalkowski: It really revolves around how we think about the future of World of Warcraft. We do plan fairly far ahead. History shows us that World of Warcraft has been around for a while, and will hopefully still be around for a while, and we should plan for that. Knowing we were going to the Shadowlands, we’ve known that for many years, before we ever began developing levels for it. If you want to talk exact years, it’s probably been … a while. A decade, at least.

GamesBeat: One of the things that struck me the most from playing a bit of the alpha is the new selection screen for making a character. Why was now the time to rework some of these basic elements?

Kowalkowski: Almost every expansion, we look at the player’s connection with their character. The ability for players to express themselves through their character. Sometimes it’s like Warlords, where we uprezzed all the character models. Sometimes it can be a new class, like Legion. Sometimes it can be the allied race option we took with Battle for Azeroth. Giving players new ways to build something in the world that they can connect with. It has to fit thematically as well. This time we looked at character customization. Can we be more — can we give players better options to build an avatar that’s more along the lines of something that they can connect with? We haven’t done that in many years. While we provided new races, we haven’t given the existing races a lot of means to be able to change or to craft themselves to be more representative of what players want to play. When we think about wanting to do these big endeavors, we take every expansion with that theme. In Shadowlands this is where we wanted to focus our efforts.

GamesBeat: You have Torghast in Shadowlands. In the past the big pillars of content have been PvP, raiding, dungeons. Is Torghast its own pillar in Shadowlands?

Kowalkowski: Certainly in Shadowlands players are going to want to run Torghast. First and foremost because it’s fun. Every time I’ve run Torghast it’s a different experience. I can take the same character, same spec through and have a different experience many times in a row. It’s not just because the hallways and the creatures look different. It has to do with the anima powers that I’m earning. The first few anima powers may decide the next few I’m going to take. It’s a diverse system that stands on its own. It’s very fun, but it’s also going to be a primary mechanism for crafting your legendary. Players are going to want to do that. In terms of being its own pillar, players are going to want to do this content. They’re going to be — there’s a lot of reasons to do it. For the duration of Shadowlands, Torghast is going to be part of the player’s experience.

GamesBeat: Renown is something that’s important for endgame progression. Can you explain how that works?

Kowalkowski: Azurite power was something that you could — again, I mentioned checking the world quests. There are times when I would look at all the Azerite power ones and go out there and do that, because I wanted to fill that bar and then immediately fill the bar again. Renown is going to be something that you go earn in a fixed amount each week. It’s not necessarily something that you’re going to be trying to min-max because there’s a set amount that you’re able to accomplish unless you’re on catch-up mode. That’s one difference. Similarly, we didn’t want to bind your covenant advancement or your sanctum advancement to reputation, for similar reasons.

GamesBeat: You’ve been working on the classes quite a lot during the alpha. You’ve made some big changes to some already during testing. Are there a few specs right now that are still on the radar as you’re in this beta period?

Kowalkowski: Certainly we have the capacity to keep an eye out and hear player feedback, and also look at the data to understand and make sure that — if we see something that’s an anomaly, like death knight’s blood boil is insta-killing things, we have the capability to track those. Even if we’re not hearing it directly from player feedback. We’ll continue to tune all the way up until we’re ready to release the game.

A screenshot of the sky in the Maw in Warcraft: ShadowlandsA screenshot of the sky in the Maw in Warcraft: Shadowlands

Above: A player gazes at the sky in the Maw zone in Shadowlands.

Image Credit: Heather Newman

Shadows rise

GamesBeat: Shadowlands is set for this fall. Is there a reason you don’t have a concrete release date yet?

Kowalkowski: We know in our heads what the date is. We’re building toward that and we feel confident about that. But we also want to make sure that we don’t release the date because the date’s there. We want to make sure that the date we release is because we feel the game is good and the experience is going to be good for everyone. We want to give ourselves the flexibility, if we need an extra week on either side of the date we have in our heads, we have that. We want to make sure that we land when we think the game is good.

GamesBeat: This is the first expansion coming out while Classic is available at the same time. Has that changed the design philosophy at all, or the mechanics of when you’re releasing content? 

Kowalkowski: We have two flavors of WoW, and it’s awesome. We have one flavor for the players that embrace the classic philosophy, and we also have the main line game that’s been — that came from Classic, but has evolved and incorporated and adapted to the community of players over the last 16 years. We do view them as two separate ecosystems. That’s why your characters don’t move between the two. We do feel like we’re supporting two games. It’s awesome, quite frankly.

GamesBeat: Is implementing the level squish difficult? The max level is changing from 120 to 60. Was that more difficult to do than the item level squish?

Kowalkowski: Absolutely. So many things in the game revolve around level. World of Warcraft, from a technical standpoint, one of the reasons I love being an engineer on this game is that there’s a never-ending — we are facing challenges that nobody else faces in gaming. This was one of them. We’ve had this game around for 16 years and 120 levels. We needed to take a look at that and think, if we’re going to continue to keep doing World of Warcraft, and we will continue making World of Warcraft, it’s more daunting the more levels we add. Levels become less and less meaningful. How can we make levels meaningful again? We wanted to do this level squish. You can imagine that without having planned that from day one, a lot of data in the game has levels sitting there. It’s a hard coded value. We had very talented engineers and designers attack the problem and figure out how we could do something similar to what we did with item squish to levels so that this doesn’t become a persistent issue every time we might want to think about doing it. It’s been a fun technical challenge, but it’s probably an order of magnitude more difficult than item was, because that just affected items. This affects creatures, zones, quests, you name it.

GamesBeat: Is it especially difficult to develop for World of Warcraft because it is a 16-year-old game? 

Kowalkowski: We keep the game fairly well updated from a technology standpoint. We’re very proud of the technology that we’ve been able to develop over the years. There’s very little code that’s survived from day one of World of Warcraft, because we constantly — we’re evolving it. It’s not just game features that are evolving. Technology is evolving. We’ve rewritten huge aspects of the game’s rendering code and gameplay code. On the server side we’ve addressed a lot of things that — World of Warcraft was a single-server closed ecosystem back in the day, and now we’re this more regional game where players are able to come together across servers a lot more easily. We continue to evolve the technology on this game. It’s an amazing challenge, and it’s one where we solve problems that very few other people in gaming have to tackle. It’s fun. It’s awesome. I’m very thankful that I have a team of very smart people to do this.