How the Super Bowl ad featuring David Letterman, Oprah Winfrey and Jay Leno came together.

Youth social networking researcher dana boyd has observed that many people presume the way they use social networks is the way everyone uses them. "I interviewed gay men who thought Friendster was a gay dating site because all they saw were other gay men," she says. "I interviewed teens who believed that everyone on MySpace was Christian because all of the profiles they saw contained biblical quotes. We all live in our own worlds with people who share our values and, with networked media, it's often hard to see beyond that."

Now picture our perspective leaving our own experiences, zooming out and up until we can see how all the different groups are interacting on a worldwide social network. That bird's-eye view could be both beautiful and horrible if the resolution was clear enough. That's what a Ramen-eating, ex-Apple engineer named Pete Warden is about to release to the public this week.

Sponsor

This Wednesday, Warden will make friend, fan page and name data from hundreds of millions of Facebook users available to the academic research community. It's a move that Facebook has to have seen coming, a move that many in the data-centric community have been calling on the company itself to do for years, and an event that's been complicated by Facebook's recent privacy policy changes, which have muddied the waters of right and wrong but rendered even more data available for outside analysis.

If what people call Web 2.0 was all about creating new technologies that made it easy for everyday people to publish their thoughts, social connections and activities, then the next stage of innovation online may be services like recommendations, self and group awareness, and other features made possible by software developers building on top of the huge mass of data that Web 2.0 made public. It's a very exciting future, and Warden is about to fire one of the earliest big shots in that direction.

Nerds in Space: Social Graph Analysis For Solving Large-Group Problems

Pete Warden studied Computer Vision in college in the UK, then got into game development. After moving to LA, he spent 6 years building graphics drivers for the original Playstation and the XBox. Then he started his own independent business, where thankfully he open-sourced much of his work (something he's still doing today).

When he found out that starting his own business wasn't going to work with his immigration status, he was very fortunate to have also caught Apple's eye with the software he had been releasing to the public. Apple bought his company in order to bring him on board. The proceeds of that small sale are now sustaining his next project after leaving the company.

After spending 5 years at Apple struggling to navigate the maze of people and connections and types of expertise in order to get the information he needed, Warden decided to go independent and build a company that solved exactly that kind of problem. "I can't think of a better big company to work for, but it was still a big company," he says. "It was hard to find the right people to talk to, whether for particular expertise or for contacts at external companies." And so Pete Warden left Apple to build a company that would use social graph analysis to solve problems like that. He called the company Mailana.

We've written here a number of times about Mailana's tool to analyze the social graph of any Twitter user. Enter the username of someone on Twitter and Mailana will show you which 20 other people they have exchanged the largest number of reciprocal public @ replies with. Find someone interesting or important? Mailana's Twitter analyzer will tell you who they most regularly interact with. See, for example, The Inner Circles of 10 Geek Rockstars on Twitter.

Pulling Down the Facebook Social Graph

Now Warden is about to unveil a much larger project along the same vein. For the past 6 months he's been crawling public profile pages on Facebook. He now has more than 215 million of them indexed and updated about once a month. When he began he was using webcrawling service 80legs but in time he had to build his own crawling infrastructure.

When I talked to him this afternoon, he had already begun uploading 100 GB of user data onto his server to make available for academic research starting on Wednesday. Warden says he's removed identifying profile URLs but kept names, locations, Fan page lists and partial friends lists. All those fields of data are just waiting to be analyzed and cross referenced. That's one very rich resource.

Yesterday Warden posted some of his own initial observations from the data on his personal blog. Those included:

• God is in almost every state in the Southern US the #1 most popular Fan page among Facebook users. Among people in the LA, San Francisco and Nevada regions? "God hardly makes an appearance on the fan pages, but sports aren't that popular either," Warden writes. "Michael Jackson is a particular favorite, and San Francisco puts Barack Obama in the top spot." In the Oregon and Idaho region? Starbucks is #1.
• In the Mormon-influenced areas of Utah and Eastern Idaho, the most popular Fan pages are The Book of Mormon, Glen Beck and the vampire book Twilight, which was authored by a Mormon.
• The bulk of Warden's posted analysis yesterday was about location networks. People in the Western US tend to have Facebook friends all over the country, people in the Southern US tend to mostly be friends with people who have remained in the same area.

Taking a Deeper Look

These observations are interesting, but they are only the beginning of what's possible. Name, location, friends and interests are great data points to analyze. Warden has written a program that will estimate gender as well, based on names. All these data points can be cross-referenced with outside data, too. Members of Facebook's own staff did this kind of analysis when they compared last names on the site to US census data about the likelihood of people with particular last names reporting particular racial backgrounds in order to estimate the changes in Facebook's racial composition over time.

"I'm mostly thinking 'What do I try first?'," Warden says. "There's so many interesting ways to slice the data - especially as I'm starting to get changes over time. I'm also trying to map out political networks in aggregate; how polarized the fans of particular politicians are - so how likely a Sarah Palin fan is to have any friends who are fans of Obama, and how that varies with location too. One of my favorite results is that Texans are more likely to be fans of the Dallas Cowboys than God."

Warden says he hasn't talked to anyone from Facebook since he started crawling the site, but he did get an email from someone on the security team asking him to take down instructions he'd posted exposing a security hole that made harvesting peoples' email addresses easy. So the company is paying attention. "I'd love to see them put me out of business by putting decent data out there," Warden says. He says his Amazon Web Services bill was over $5k last month.

Why is he indexing all this content and why is he going to hand it over to the academic world later this week? "I am fascinated by how we can build tools to understand our world and connect people based on all the data we're just littering the internet with," Warden says.

"Nobody thinks about how much valuable information they're generating just by friending people and fanning pages. It's like we're constantly voting in a hundred different ways every day. And I'm starry-eyed believer that we'll be able to change the world for the better using that neglected information. It's like an x-ray for the whole country - we can see all sorts of hidden details of who we're friends with, where we live, what we like."

For a great example of the kind of social impact that data analysis can make, Warden points to some of the fascinating ways that GIS data is illuminating the intersection of race and public services. Data has shed light on social injustices for decades and measurable information about the interactions of hundreds of millions of people every day on Facebook offers opportunities to discover both good and bad news about the contemporary human condition.

Warden says he's not yet been able to interest any investors in his ideas for businesses based on this data, so his girlfriend Liz Baumann, a former insurance actuary, stepped in to help and is now running much of the crawling. He says he's now focused on "working on ways of presenting all this information in a form that answers questions for people willing to pay." His first experiment along those lines is the very interesting FanPageAnalytics.com.

What does Pete Warden hope for from this week's public release of all this Facebook data? "Hopefully I'll get to see a bunch of interesting [academic research] papers come out of it, worst case. And I'd like to be the guy people turn to when they need stuff like this."

Already well-respected among a fringe group of bleeding-edge geeks, we hope that Warden's work on social graph analysis will end up impacting a far larger number of people than may ever know his name.

Discuss

If you want a Blu-ray player or a new car, you’re pretty much stuck with what big companies like Sony and GM have to offer. It might not be exactly what you want, but it’s available and it’s cheap.

Until recently, the tools of production — from design to manufacturing — were inaccessible to the masses. Just because someone tinkering in their garage could design a better product didn’t mean the inventor could suddenly scale up the operation into production.

Wired Editor in Chief Chris Anderson thinks this is about to change. In “Atoms Are the New Bits” from the February issue of Wired, he argues that the proliferation of open-source, DIY design in recent years signals the dawn of a new industrial revolution.

In his previous bestseller The Long Tail, Anderson discussed the long tail of bits — the idea that open source software serving niche markets collectively takes up a larger market share than large, best-selling products. In his latest Wired story, Anderson argues that — thanks to the democratization of the tools of manufacturing — product design is moving toward the long tail of stuff.

But can a legion of small-scale product designers armed with great ideas collectively churn out more wares than Apple and Sony? In this week’s Storyboard podcast, Anderson discusses his new “mind grenade” with Wired Executive Editor Thomas Goetz.

Photograph: Dan Winters

A major new winter storm is headed east over the U.S. today, and threatens to dump a foot or more of snow on Philadelphia, New York City, and surrounding regions Tuesday and Wednesday. Philadelphia is still digging out from its second top-ten snowstorm of recorded history to hit the city this winter, and the streets are going to begin looking like canyons if this week's snowstorm adds a significant amount of snow to the incredible 28.5" that fell during "Snowmageddo...<br /><a href="http://www.wunderground.com/blog/JeffMasters/comment.html?entrynum=1427">Read More</a>

Amazon’s recent announcement of dramatically higher royalty rates for authors and book publishers, a move designed to level the playing field with Apple’s iPad tablet, seems to be having some effect: another author has signed an exclusive book deal for the Amazon Kindle. In this case, Gavin de Becker — author of several books about security — has agreed to release expanded and updated editions of two of his books, “The Gift of Fear” and “Just 2 Seconds.” A news release says that while both books have been available as print copies for some time, this is the first time “The Gift of Fear” has been available electronically, and both will be exclusive to Amazon’s Kindle Store for one year.

This deal appears to be very similar to a Kindle exclusive announced by author Stephen Covey — creator of the “The 7 Habits of Highly Successful People” line of books — in December, which saw the author transfer the rights to two of his books from Simon & Schuster to an electronic publisher in order to do the deal with Amazon. As part of that arrangement, Covey was to get about 50 percent of the proceeds from the sales of Kindle versions of the books, in contrast to the usual 25 percent that most publishers provide. Amazon will have exclusive e-book rights for a year.

Last month, Amazon signed another high-profile author to an ebook exclusive: best-selling Brazilian writer Paolo Coelho agreed to give the company and the Kindle exclusive rights to Portugese versions of 17 of his popular novels. None of Coelho’s books have been available in e-book format before. A recent report also confirmed that British author Ian McEwan signed an exclusive deal last year for electronic publishing rights to five of his books in return for 50 percent of the royalties.

One wonders whether any of these authors will want to renegotiate their exclusive deals, now that Amazon is offering authors and publishers 70 percent royalties instead of just 50 percent. In any case, the upward pressure on royalty rates is a welcome sign of increased competition in book publishing, thanks to both Amazon and Apple, and that is something many authors will no doubt be pleased to see.

Meanwhile, Amazon continues its fight to keep e-book prices low, although it appears to be steadily losing ground in that battle. After Macmillan refused to lower its prices to the $9.99 that Amazon was demanding, the electronic retailer yanked the publishers books from its shelves (both print and electronic), but was later forced to capitulate. Since then, two other publishers have also renegotiated higher prices with the company. Amazon has since put print versions of Macmillan books back on its virtual shelves, but according to a recent report, it is still not stocking Kindle versions.

Related content from GigaOM Pro:

Why Closed Platforms Might Not Be So Bad

The average American is expected to spend nearly $1,000 on services like cable, Internet and online video games.

By Charles Petersen

The Accidental Billionaires: The Founding of Facebook, A Tale of Sex, Money, Genius, and Betrayal
by Ben Mezrich

Stealing MySpace: The Battle to Control the Most Popular Website in America
by Julia Angwin

Facebook, the most popular social networking Web site in the world, was founded in a Harvard dorm room in the winter of 2004. Like Microsoft, that other famous technology company started by a Harvard dropout, Facebook was not particularly original. A quarter-century earlier, Bill Gates, asked by IBM to provide the basic programming for its new personal computer, simply bought a program from another company and renamed it. Mark Zuckerberg, the primary founder of Facebook, who dropped out of college six months after starting the site, took most of his ideas from existing social networks such as Friendster and MySpace. But while Microsoft could as easily have originated at MIT or Caltech, it was no accident that Facebook came from Harvard.

The US government filed its Statement of Interest regarding the revised Google settlement yesterday with the District Court in New York. While the statement was signed by an attorney from the Antitrust Division of the Justice Department, several agencies including the Copyright Office reportedly contributed to it.

As you may recall, the judge has only 2 choices: he can approve the settlement, or send it back to the parties for revision. He cannot modify it himself.

The US government statement advises the judge that the public interest would be best served by sending the settlement back, and points out that the revised version still suffers from the "same core problem" that afflicted the first version: "an attempt to use the class action mechanism to implement forward-looking business arrangements that go far beyond the dispute before the Court in this litigation."

The press reports that I've seen take the government's statement as an emphatic thumbs down.

The judge has scheduled a hearing for February 18 in his Manhattan courtroom.

It is very unlikely that the judge will approve this version of the settlement. Also, he may once again decide postpone a full-fledged fairness hearing-although the many objectors, large and small, are eager to have their day in court. Because the parties withdrew the proposed settlement before the originally scheduled fairness hearing occurred in October 2009, the judge has not yet formally considered the many objections filed to date on the revised settlement and filed in anticipation of the fairness hearing cancelled last October.

Bottom line for the long term: even if the judge sends the settlement back, and even if the parties agree to deadlines as short as the deadlines for this presumably ill-fated revision, there is no resolution in sight for the litigation.

Whether the case is tried or the settlement discussions continue, the legal end point will not be the trial judgment or settlement approval issued by the district court judge. The end point will be the disposition of the final appeal from that district court judgment or approved settlement, and that disposition is years away.

It's hard to imagine what relevance the final legal disposition would have then, as public and private innovators are not sitting on their hands, waiting for the judge to sort this out.

AMEE, the US/UK-based startup that aims to build the largest engine for computing greenhouse gas emissions, has secured a $5.5m series B financing lead by Amadeus Capital Partners alongside existing investors, including O’Reilly AlphaTech Ventures and Union Square Ventures. AMEE will use the funding to expand its geographic reach and platform.

The prize AMEE is aiming for, known in the sector as “enterprise carbon management”, is expected to reach $4 billion by 2017 because of government and consumer pressure to address climate change. AMEE’s engine is now being used by companies offering carbon accounting or business intelligence software, as well as governments, multi-nationals and SMEs.

The problem AMEE is addressing is that there are currently, multiple standards and hundreds of thousands of individual emission factors used to determine something’s carbon footprint.

So AMEE has codified the major greenhouse gas standards, their computation models, and emission factors into an engine that is available to clients its API. This is exposed as a RESTful HTTP-based web service with XML, JSON and Atom interfaces. Customers so far include SAS and the UK Government’s Department of Energy and Climate Change (DECC). AMEE’s search engine also enables discovery of greenhouse gas standards and emission factors.

In July last year it boosted its board with Toby Coppel, former chief strategy officer of Yahoo!

It has also created RealTimecCarbon, a joint project by Dynamic Demand, AMEE and Demand Logic, which lets you see the real-time carbon intensity of electricity so consumers can avoid consuming at times of high emissions. This could eventualy be linked to appliances, buildings and factories so they could automatically manage demand according to the carbon being released.

Pat Burtis, Investment Manager at Amadeus Capital Partners, joins the AMEE Board of Directors.

That's the rumor (BI via WSJ). The idea is to let Gmail become your portal into status update.  

It won't work, period, unless it connects to Facebook and Twitter. And so far, as I've pointed out before, Google won't do that, at least, not yet, and and certainly not in the way it should be done.

Google is simply not understood by consumers to be a place where they can connect with friends and colleagues. If it intends to become that, it has some DNA mutation in its future.

This one should be interesting.  

The meeting would mark the first time in the long health care debate that leaders from both sides would be allowed to air their ideas publicly.

Are you a budding Web entrepreneur who would like some pointers or advice from seasoned company founders? MayField Fund and First Round Capital are sponsoring a raffle to give away mentoring sessions with the founders of Digg (Jay Adelson), Flickr (Caterina Fake), Mint (Aaron Patzer), Ning (Gina Bianchini), Slide (Max Levchin), and Zynga (Mark Pincus).

The raffle will take place at a private event in Silicon Valley with space for 100 attendees on March 1. But you can win a ticket for the event by applying here. The event and raffle are free, but the 100 attendees in the running will be selected beforehand by partners at Mayfield and First Round.

Winners of the raffle do not get to become Best Friends Forever with the founders. But they will get one intense mentoring session each.

The way the Senate works is no longer consistent with a functioning government, and senators should change the rules to end obstructionism.

Because today's Public Editor column takes the unusual step of recommending that a respected Times journalist be reassigned, I thought it only fair to offer Bill Keller, the executive editor, an opportunity to respond in full. Here is what Keller had to say.

Gmail is set to become Google’s next major push into social media. According to The Wall Street Journal, the popular webmail service will soon launch a new feature for sharing content and status updates with friends. [Update: We think Google might announce these features on Tuesday]

As WSJ points out, Gmail users can already update their statuses — sort of — through Gmail’s chat feature. Currently, this feature is more akin to the traditional IM “away message.” However, with this new social push, Gmail will offer a timeline-view of your friends’ status updates, just like on Facebook and Twitter.

Those updates might come from both Gmail and third-party services. According to WSJ, Google-owned YouTube and Picasa will be integrated into the stream. The huge question then is whether or not the new feature will include updates from Twitter and Facebook.

If so, the new features could be thought of more like a TweetDeck or Seesmic, looking to provide an aggregate view of your friends’ social media activities along with the ability to push status updates to the services you use from inside of Gmail. If not, it could be thought of as a major competitor to Twitter and Facebook as Gmail looks to covert its millions of e-mail users into adherents to a whole new breed of social media service.

An issue with the latter, however, is that Gmail has historically added people to your contacts based on e-mail interactions. Hence, this contact list often varies significantly from your friends on social sites where relationships need to be made explicitly.

In other words, your Gmail contacts aren’t necessarily the same people you want to share status updates, photos and videos with. This is an issue that shouldn’t be overlooked in evaluating the new features Google is soon to unveil.

Tags: facebook, gmail, Google, trending, twitter

Perhaps yesterday’s tweet by Google CEO Eric Schmidt put it best. Hell has indeed frozen over. Google has run its first major television ad, during the Super Bowl, no less. The rumor that Google would run a commercial during today’s Super Bowl 2010 proved true. Google aired a spot from its online video Search Stories series, [...]

....

The bank’s demands for billions of dollars from the insurer bled it of cash, which the government later provided.

As more and more of our friends and favorite organizations start publishing updates online, being able to organize them well is becoming even more important. Niche-popular desktop social media stream-reader Tweetdeck issued a software update this morning and the most striking change is in its handling of user groups. It's beautiful. The new Tweetdeck is faster, more flexible and easier to navigate.

Groups, we have argued, are the secret weapon of the social web. Here are five ways that the new Tweetdeck gets groups right and that Facebook, the world's dominant social media stream reader by a long-shot, could learn from what Tweetdeck is doing. That would drastically improve Facebook's own user experience.

Sponsor

Internet startup investor John Borthwick of Betaworks has told us that he invested in Tweetdeck specifically because its column metaphor represented a drastic break from the page-based metaphor of the rest of the web and the Instant Messaging metaphor of most other Twitter clients. That's how Tweetdeck works: it lets you put your friends and contacts on Twitter, Facebook, MySpace and LinkedIn into grouped columns across your screen. It's a powerful system and the clear leader in the ecosystem of interfaces built around Twitter. Competitor Seesmic has a similar offering and is also based on columns for groups.

These applications may be more overwhelming than many mainstream users are looking for, but the principles could be adapted to Facebook's own interface in some very interesting ways.

The Problem With Facebook's Group Support

Groups of users, which Facebook calls Lists, are extremely helpful in prioritizing messages by their source. They enable users to subscribe to more sources of information in total without fear they will miss high-priority content. Groups help contextualize messages in a stream and with good search support they can help you target queries and unearth the information you're looking for within a limited space of trusted, topical sources of information. Last month, Facebook suggested its users subscribe to news organizations on the social network and put those updates in a special list called News, for example.

Unfortunately, Facebook has never treated Groups with the respect that they deserve. The newest redesign pushes friends Lists a click removed from the front page of the site, even. (It took me three clicks from the home page to see the view pictured on the right, for example.) The company is instead focused on serving up content from favored sources using the new News Feed (vs Live Feed) algorithm. This algorithm says that the more you've interacted with a source of information in the past, the more likely you are to want to read that person's updates in the future. News Feed is a self-reenforcing paradigm that simplifies and narrows a user's universe by taking editorial control out of their hands and putting it in the hands of a black-box formula.

How could Facebook better handle groups? Let's take a look at how Tweetdeck does it.

Tweetdeck's Superior Handling of Groups

• Buttons to navigate directly

  The new Tweetdeck update today added a series of buttons at the bottom of the application for your group columns. Hover over them and you'll see the column name. Click on that button and you'll be brought immediately to that column, a much nicer experience than the awkward old scrolling. The hover display does show some information that's probably not useful to anyone (API calls remaining) but putting rapid group navigation in a good place in the app makes a big difference. It would also be nice if those buttons were configurable in some way - like the red one or #3 is for my @ replies column. But Tweetdeck now makes it much, much easier to navigate between groups than Facebook does. Facebook almost seems to be discouraging use of groups by burying them several clicks below top level options.

  
  
  
• Different Notification Priorities

  Tweetdeck makes it easy to set up different levels of notification for different groups. For example, my "high-priority" Twitter group sends full messages as a pop-up in the corner of my screen when it updates, but my column for people I actually know in real life on Facebook (like my family members) throws a pop-up and an audio notification in Tweetdeck when there's something new. That's awesome.

  Facebook is all about the notifications, why not let me get a special notification when someone in a particular group has posted an update?

  
  

• Shareable Groups, Suggestions etc.

  For years Twitter has been fundamentally public and Facebook private, but for better or for worse that's changing on Facebook. Isn't it time for Facebook to offer bundles of pages or people that you can follow all at once? Twitter and Tweetdeck make this easy with lists, sharable lists and the Tweetdeck directory of recommended lists to follow. Facebook users would love something like that! My college newspaper staff as one big list of Facebook users to follow? My football team? Suzy Bright's curated list of top sex writers to follow on Facebook? People would eat that kind of stuff up.
  

• Keyword Filters

  Last week I was working when the season premier of Lost came on TV. I'm likely to watch it later on DVD. Tweetdeck let me add a filter to all of my groups to hide any posts that included the word Lost! Sick of hearing about the iPad? No problem! Tweetdeck does a great job of building value on top of these groups of contacts: filter for, filter out keywords, analyze a group for its most-used words. There are lots of possibilities. Facebook users would probably like these same options.
  

• A Group for New Friends

  Tweetdeck has a great feature that lets you create a column to display the bios of people who have just started following you. From there you can click once to follow them back. Facebook could easily do something like that, bring new friends and/or recommended friends up higher into a very visible place in the interface. Facebook wants users to go beyond connecting with people they already know in real life (sorry, users, that's what Facebook has decided) so why not create a "group" that's made up of people who are fans of the same pages you are or are otherwise recommended for connecting with?

There are all kinds of ways that Facebook could offer meaningful support for user groups and turn the News Feed into a more powerful tool, with more control for users and more value in the long run. Tweetdeck is doing a pretty darned good job of exactly that.

Discuss

Through the use of various tests, a handful of dating Web sites are competing to impose some structure on the quest for love.