LinkedIn for iPhone 3.2: Where Are They Now?

Keeping track of what your connections are doing professionally is one of the most important things we offer to our members at LinkedIn.   We’re excited to announce LinkedIn for iPhone 3.2, because we’ve made it easier than ever to stay up-to-date and find the most important changes happening in your professional network.

Download: LinkedIn for iPhone v3.2

It’s All About the Profile

We’ve made a number of changes to how we show profile updates from your connections to make it easier to quickly find the most important changes.  From either All Updates or Profiles, you can now see more information about what was changed and make a better decision on how important it is to you professionally or personally.

From a profile update, you can tap on your connection’s name to go to the top of their profile, or tap on any section to jump right to that section.  We’ve also made it easier to find the changes by highlighting them in our new profile design.

In addition to letting you see what sections have changed when a connection updates their profile, we’ll also let you know when they add a Twitter account, change their address or phone number, or add their birthday.

Who Do You Know?

One of the most commonly used features in our current iPhone app is viewing who you have in common with another LinkedIn member.  It’s a great way to build out your own network, and it’s also useful when you’re meeting someone new and need to know more about them to help start the conversation.

You can now see how many people you have in common right from the main profile screen.  We’ve also created a single view that lets you quickly switch between viewing who you have in common and viewing their full connections list.

Optimized for Retina Display

The new version of LinkedIn for iPhone is sporting all new icons and images, taking advantage of the amazing new Retina display on the iPhone 4.

What’s Next

The iPhone team here at LinkedIn is growing, and we have 2 amazing interns this summer (see if you can find them in the screenshots above).  We have a lot of great new things on the horizon for October and November, and are very excited about the new experiences we’ll be bringing you this year.

If you want to keep up with what we’re doing, and have a place to give us feedback and talk about your ideas for LinkedIn on the iPhone, please join our LinkedIn for iPhone group.

Download: LinkedIn for iPhone v3.2

Share: Email | LinkedIn | Digg | Twitter

New and improved moderation features for LinkedIn Groups

Since its roll out at the end of June, the new groups interface has already significantly increased member participation, measured by user comments and visits to Groups. We’ve also been listening to user feedback from many of our active groups on LinkedIn. In the spirit of continuous improvement, we’re excited to present the first set of improvements to how Groups can be moderated that we’ve incorporated after hearing from you.

1. Delete threads directly. One of the biggest concerns group managers have had is around spam and how best to manage it effectively. Starting this week, we’ve made it very easy for moderators to actively fight spam by deleting inappropriate content from the “More” tab on any LinkedIn groups’ thread.

2. Set up email notifications for new LinkedIn groups posts. If you want to clean out spam the minute it’s posted, as a group manager you can now get email alerts for every new post and delete spam straight from these emails.  Subscribe to these alerts by going to More > My Settings > Activity and check the box “Send me an email for each new discussion.”

3. Delete inappropriate posts right from your email updates. We’re also making it easier than ever for group moderators to delete inappropriate posts with one-click right from the email alerts of the discussions you follow in your group. In a coming release, managers and moderators can also remove, block, and delete not just one email thread but all contributions of the offending spammer right from these emails.

4. Find and encourage group members to help you moderate. You don’t have to take on the work of moderating your group alone. Deputize trusted members to help moderate so they can utilize all the great moderation features too.

5. Encourage your members to flag inappropriate content. We also wanted to make it easier for all members of your group to help you weed out spam and inappropriate content. All posts and comments flagged as inappropriate by group members will be added to the group managers’ new moderation queue. In one swift move, you can delete or clear flags from posts as well as blacklist spammers with the powerful “remove member, block & delete contributions” button.

6. Automatically delete posts by setting a limit on the number of flags. You can also automate the process of bringing down spam in the queue by defining the number of flags a post or comment needs to reach before it gets automatically deleted (like magic!). The default is 10, and you can adjust this setting to a higher or lower number as needed in Manage > Edit Group Settings.

This is just the first in a series of improvements we’ve planned for Groups to improve the quality of your conversations, by giving both moderators and group members a way to easily manage it.

We look forward to hearing your feedback, so please feel free to leave a comment or @linkedin us on Twitter.

Share: Email | LinkedIn | Digg | Twitter

Share your favorite HuffPost news on LinkedIn

Ed. note: This belongs to a series of posts on how professional web sites and services are integrating LinkedIn functionality using the LinkedIn API. This post is from Paul Berry, CTO at the Huffington Post (HuffPost) where he develops products and infrastructure.

I’m excited to announce that starting today, LinkedIn users can share their favorite HuffPost business news articles with their LinkedIn network. As the leading social news site, HuffPost enables communities to share news with their favorite social networking sites. Now, we’ll be extending HuffPost Social News to include the professional community on LinkedIn.

Here’s a recent video interview with our CEO, Eric Hippeau, on our social news sharing announcements. Learn more on how to sync your HuffPost and LinkedIn accounts, after the jump.

HuffingtonPost integrates with LinkedIn

We believe this integration will generate tremendous engagement and interest from our business section readers, enabling easy sharing on LinkedIn of favorite HuffPost business and travel stories.

Here’s a 3-step process to start sharing your favorite HuffPost business news items on LinkedIn:

1. Sync your HuffPost and LinkedIn accounts

The easiest way to get started would be to sync your LinkedIn and HuffPost account (if you have one). Look for the LinkedIn button on the HuffPost homepage header.

Clicking through the logo will guide you through the Sign-up process that’ll let you sync your two accounts

2. Find travel or business articles you’d like to share with your professional network

Once your accounts are synced, go back to perusing your favorite HuffPost articles and you’ll notice that each article provides sharing functionality with your favorite social networking sites.

3. Share relevant breaking news with your business community on LinkedIn

When you find the appropriate breaking news you’d like to share, click through the share button that’ll open up the share functionality page. Now, you can update this share page with relevant comments before it goes out to your LinkedIn network updates home page.

So, what are you waiting for? Get started by binding your two accounts and start sharing business and travel news from HuffPost!

Ed. Note: If you’re a publisher, who’d like to integrate LinkedIn Sharing functionality to your site, email publisher@linkedin.com.

Share: Email | LinkedIn | Digg | Twitter

Connecting with business contacts is just a Bump away

Ed. note: This belongs to a series of posts on how professional web sites and services are integrating LinkedIn functionality using the LinkedIn API. Jake Mintz, co-founder of Bump technologies, describes how the most recent version of their mobile app now lets you connect with people with just a bump of your mobile phones.

We created the Bump app to make it easier to connect and share with people in the real world. It couldn’t be easier, just bump two phones together and they connect. Millions of people are now bumping their contact information and photos to each other.

But connecting with a new acquaintance on social networks is becoming as or more important than sharing your contact information with them. That is why we are so excited to tell you about a two big new features in Bump 2.0: bump to connect on LinkedIn and bump to find mutual connections on LinkedIn.

Bump to connect on LinkedIn

Instead of waiting until you get to your computer to invite someone to connect with you on LinkedIn you can now do it more easily in person. LinkedIn invitations can be bumped by themselves, or with your contact information.

Bump to find mutual connections on LinkedIn

Have you ever met someone and thought that you must know some of the same people, but couldn’t figure out whom? Now, with just a bump, you can see if you have any of the same connections on LinkedIn. Discovering that you do know the same people provides valuable context and richer conversation when meeting someone.


We’re incredibly excited to work with LinkedIn to bring professional networks into the real world.

Bump 2.0 for iPhone is available now for free on the iTunes App Store!

Share: Email | LinkedIn | Digg | Twitter

Creative Professionals can now display their portfolios on LinkedIn

Ed. note: This belongs to a series of posts on how professional web sites and services are integrating LinkedIn functionality using the LinkedIn API. This post is from Scott Belsky, CEO and co-founder of one of the leading platforms that helps creative professionals display their work – Behance.

We are thrilled to introduce Creative Portfolio Display, an invaluable application for creative professionals within LinkedIn and beyond.

For too long, creative professionals in industries such as advertising, fashion, design, architecture, film, photography, and digital media have lacked tools to efficiently manage their portfolios and showcase their talent. Personal websites don’t adequately broadcast your work for discovery by industry peers and top companies and neither is it tied to your professional identity on the web. Plus, keeping your work samples up to date on multiple websites is inefficient, resulting in portfolios becoming outdated and forgotten.

With LinkedIn Creative Portfolio Display application, we aim to empower creative professionals by creating a one-stop solution for maintaining their work portfolio and broadcasting it to millions.

Introducing Creative Portfolio Display

With Creative Portfolio Display, LinkedIn members can create a free, multimedia work portfolio that is displayed on their LinkedIn profiles as well as other websites and galleries around the web. This new LinkedIn application is powered by the Behance Network, the leading online platform for creative professionals.

Here are some key features:

  1. From one central portfolio, your work can be displayed within LinkedIn, Behance.net, AIGA, MTV, and other networks/galleries around the web.
  2. The Behance Network is a free service, allowing you to host an unlimited number of multimedia projects that include still images, video reels, text, and/or audio samples.
  3. Creative Portfolio Display seamlessly syncs with your free Behance.net account; Create a new portfolio project on Behance.net, and it instantly appears on LinkedIn and our other partner platforms.

Through Behance’s Creative Portfolio Display application, LinkedIn becomes a tremendous opportunity for creative talent to get found and showcase their work to millions of visitors.

If you are a creative professional on LinkedIn, we invite you to join the Behance Network and sync your portfolio with LinkedIn. If you are already a member of the Behance Network, simply install the Creative Portfolio Display application on your LinkedIn Profile.

We welcome your feedback as we continue to improve LinkedIn Creative Portfolio Display and other Behance Network services.

Install The Creative Portfolio Display Application on LinkedIn

Share: Email | LinkedIn | Digg | Twitter

Legal Updates that matter to you, from lawyers on LinkedIn

Ed. note: This belongs to a series of posts on how web sites and services are integrating LinkedIn functionality using the LinkedIn API. This post is from Aviva Cuyler, founder and CEO at JD Supra. JD Supra helps legal professionals showcase their expertise by bringing their content in front of a targeted audience.

Last week saw the launch of JD Supra’s Legal Updates, an InApp that brings timely legal news and commentary to everyone on LinkedIn. Legal Updates publishes a feed of legal content matched to your profession or industry, and allows you to connect directly to any lawyer whose work you’ve read and insights you value.

We created Legal Updates to provide a networking tool for two groups:

  1. Lawyers who understand the importance of showcasing expertise by sharing useful content online; and
  2. All professionals on LinkedIn who count on valuable business intelligence and insight (in this case, legal insight) to stay ahead in their field.

Add Legal Updates to your LinkedIn homepage

1. A custom (and customizable) feed of important legal updates:

When you add Legal Updates to your LinkedIn homepage, we automatically match the feed of articles, alerts, and other documents to your industry or profession. The feed is also entirely customizable. Follow legal topics that matter to you (real estate, insurance, IP law, tech law, tax, employment law, etc.), or follow specific sources of content (the lawyers and law firms writing and publishing this information).

Additionally, throughout Legal Updates you can click to follow a particular subject or add a lawyer or law firm to your homepage feed.

2. A browsable, searchable archive of legal information directly within LinkedIn

Legal Updates also brings a rich repository of legal updates (articles, analysis, briefs, filings, and more) directly to LinkedIn. You can browse topics covering numerous industries, or search for information on a specific issue. Leading lawyers and law firms from around the world add their legal content daily.

3. For lawyers, powerful distribution, meaningful connections

Legal Updates allows lawyers and other legal professionals to upload and share their content directly within LinkedIn. Any lawyer can include a portfolio of legal content on their LinkedIn profile, be found in the Legal Updates archive, and distribute their work to professionals in the fields they serve.

Moreover, for any lawyer who has connected their JD Supra and LinkedIn accounts, we include an “Add to Network” link in their documents. Any professional reading Legal Updates can add those lawyers whose work they admire or find influential to their LinkedIn network.

Business intelligence just got richer on you LinkedIn homepage!

Add Legal Updates now

Share: Email | LinkedIn | Digg | Twitter

How can social media help your small business grow

As part of our continuing series featuring our employees, we’re turning you over to Dan Yoo. Although many of our employees would describe themselves as entrepreneurial, Dan channels his passion as a foodie into investing in a brick and mortar business — a restaurant in the Financial District of San Francisco called Stone Korean Kitchen. Here he talks about how social sharing helped him increase sales in a measurable way.

When I started at LinkedIn about a year ago, I expected the work to be challenging and exciting. I was right. In addition, thinking about and working with social technologies has had a huge impact on how I and my fellow investors (including my colleague, Robby Kwok) promote Stone Korean Kitchen, a project we started with a third partner who runs the business full-time. We actively engage through our company page on LinkedIn and similar business pages on Yelp and Facebook. Also, we broadcast news and promotions through Twitter, Foursquare, Facebook and Yelp to drive business to the restaurant. These social channels also help us find out what our customers think about us, and we use the feedback to address problems.

Perhaps the best example of the effect social media has had on our business is our most recent experience with social sharing through Groupon. Groupon is a service that enables companies to offer great discounts in bulk. The coupon only goes into effect if enough people purchase it. We were lucky enough to be featured as a San Francisco Groupon, and we sold 2,600 coupons as a result. This gave a huge boost to our business — our post-groupon revenue was, on average, 63% higher than our pre-Groupon revenue over the same amount of time.

You can see the initial spike in revenue in the graph below. That’s to be expected after distributing a coupon. What we found even more interesting was the “new normal” that resulted. Even after the bump from Groupon, our revenue has leveled off to almost 50% higher than before.

There are a few contributing factors to this, we believe. One is simply repeat customers, but the other is all of our social media efforts combining to create a viral spread of knowledge about Stone Korean Kitchen. After the Groupon bump, we got double the number of Yelp reviews each month than we had in the months prior to it. More Yelp reviews gave us added legitimacy and further boosted our ability to draw in customers. It was the online equivalent to walking by a crowded, bustling restaurant. If that many people are enjoying it, they must be on to something.

While it’s a lot of work to keep up with all of the activity in two such different worlds, I like how they interact with each other. I can bring my own expertise in high tech to the world of small business, and as an entrepreneur I am more aware of how much social technology increasingly now matters to small businesses around the world.

Also check out TechCrunch’s coverage of small businesses that features Dan and Robby’s social sharing experiments as a small business.

Share: Email | LinkedIn | Digg | Twitter

Over 6 million LinkedIn members in India and counting…

Ed. note: Jose Mallabo’s last Q&A blog post was on the topic of LinkedIn and Social Networking in India. Since then we’ve hired a country manager in India and we continue to grow at a rapid clip in that part of the world. We also recently added our 6 millionth LinkedIn member there, which seemed like perfect timing for a blog Q&A with Hari V. Krishnan, our country manager in India.

JM: You opened the India office about 6 months ago. What’s the biggest lesson you’ve learned?

HK: With a country this large and diverse, we need to constantly invest in understanding our users better – this ranges from our end users to our advertisers and recruiters (for e.g.). That’s job 1, 2 and 3. Our team has also spent a lot of time in India this past year understanding the market, our potential partners and hearing feedback from our members.

JM: India just passed 6 million members and remains one of the fastest growing member bases in the LinkedIn network. What’s driving this?

HK: Culturally, professionals in India have always valued a strong professional network. In a hyper-competitive global economy, no one wants to miss an opportunity to build that strong network and I think people are seeing the possibilities with social networking sites like ours. Don’t get me wrong, we have a lot of work to do to make LinkedIn indispensable to the Indian work force – but we’re headed in the right direction.

JM: How many Indian members will it take to hit a tipping point?

HK: Jeff likes to say that our ultimate goal is to connect all of the world’s professionals. My job is to connect the millions of professionals from India and help them be successful.

JM: Are you opening a technology center in Bangalore?

HK: (Laughs) That’s been the formula for dotcoms, hasn’t it?

We’re focused in understanding our end users’ needs better, as well as developing both our marketing solutions and hiring solutions businesses in India. We rely on our product and engineering teams in the Bay Area for developing the platform and maybe some day we’ll have our own tech teams in India. But it’s not our immediate focus.

JM: What about launching a site in Hindi?

HK: Local language is a critical part of our localization efforts around the world. We’ve recently launched in Portuguese and Italian – and both have spurred strong adoption by members.

That said, the language of business in most Indian markets is English. That’s the primary market demand and we need to listen to it.

JM: The Indian work force is among the youngest in the world. Do you have any advice to the early jobbers just cutting their teeth?

HK: Building long-term business connections and relationships starts on day 1 of your career and can be the key to sustained success. Professional networking sites such as LinkedIn affords you not only the most effective way to find a job but also be great at that job while building your professional brand in the process. I always urge users to check out our learning center for more tips.

JM: What can we expect from you and LinkedIn India towards the end of the year?

HK: We’ve added some great talent to the team in India over the last several months.  With this team in place driving our business lines and managing our partners, you can expect us to get more involved with our user community at a grassroots level.  We hope to do more meet ups with our members and customers in India at all key business centers in Mumbai, Delhi, Bangalore and beyond.

JM: Really?

HK: Really. I’ve got my credit card ready.  I look forward to checking out all the cool coffee shops around the country with fellow LinkedIn members!

Share: Email | LinkedIn | Digg | Twitter

How Philips engages with the global healthcare community on LinkedIn

Ed. note: Last week in London, Philips’ Global Director of Online Hans Notenboom spoke with LinkedIn’s Kevin Eyres at UK’s largest marketing event, Marketing Week Live. Their presentation offered a sneak peak into how Philips uses LinkedIn as a platform to engage with their audience in the healthcare sector in innovative new ways. Here, Kevin and Hans discuss some of the key themes and background to the partnership.

Kevin Eyres (KE): Tell us a bit more about the marketing strategy at Philips

Hans Notenboom (HN): As a company, Philips aims to look beyond simply what our technologies can do, and think about how we can improve people’s lives through innovation. Part of this is listening to our customers and the wider community to better understand the challenges they face and where we can improve what we do to stay relevant. A major part of our business is healthcare technology and the more we looked at building a platform to engage with professionals in this industry, the more we realised this community already existing on LinkedIn.

KE: How does LinkedIn benefit your audience in the healthcare industry?

HN: Healthcare professionals around the world discuss the latest technological developments with their peers and colleagues every day. With LinkedIn, we can bring these thousands of conversations happening worldwide onto one platform. Doctors from India to the US to Germany can now share insights and experience in an environment where we play the facilitator in an unobtrusive way. This is vital for ensuring the conversation stays focused and flows easily, delivering real benefits to our audience.

KE: What were the key decisions in getting to this stage?

HN: With the “Innovations in Health” LinkedIn group, we’ve taken a significant step in a new direction for our business-to-business marketing in moving part of that engagement off of our own website. Part of the drive behind that is the realisation that in the future 95% of the online engagement with our customers will happen off our own website. Ignoring that fact would have been a mistake and we now have an opportunity to have conversations with the five million healthcare professionals on LinkedIn in an environment that they are comfortable in and trust.

KE: What role does the issue of trust play in this for Philips?

HN: When it comes to any part of business, but especially something like healthcare, trust is paramount. LinkedIn is your professional identity, meaning the contribution you make is highly likely to be thoughtful and insightful. This is the environment we wanted to create to ensure its usefulness for members. If you don’t add value with these engagement efforts then you won’t get people coming back, and this is a long term investment for us. We’re still in the early stages but already seeing a great level of engagement among the group’s 11,000 plus members and this is increasing daily.

KE: What would be your top three tips for brands looking to engage their audiences through social media?

HN: Firstly, this is not a short term project. Social media needs to be an ongoing, integrated element of your communications with the outside world and your marketing strategy to be genuine. Secondly, context is king. Think about the type of message you want to get to your audience and the right time to do this. Since we’re focused in the professional space, Linkedin was the best choice. For other brands social networks might be more appropriate. Finally, moderation is very important. We took the decision to use well-known names in the sector to help us with this and it’s helped keep the conversations relevant and valuable for the members.

KE: Any early insights you can share?

HN: The group is already proving successful in bringing the right professionals together to have in-depth and valuable conversations about their respective fields. The group numbers and engagement have far exceeded our forecasts and we’re excited about the new elements we’ll be bringing in over the next weeks and months.

Note: Innovations in Health is the first of a series of groups Philips will be hosting on LinkedIn, each relevant to a specific sector. For more information on the opportunities for brands to engage the global professional community on LinkedIn, please visit our dedicated Marketing Solution site.

Share: Email | LinkedIn | Digg | Twitter

LinkedIn, Apache Pig, and Open Source

Code Alert! This is a part of our continuing series on Engineering at LinkedIn. If this isn’t your cup of Java, check back tomorrow for regular LinkedIn programming. In the meanwhile, check out some of our latest product features, tips and tricks, or user stories. - Ed.

One of the reasons I joined LinkedIn Analytics is its commitment to open source. At LinkedIn, we love open source. We’re committed to contributing to Hadoop and Pig and giving back to the open source community through projects like Azkaban and Voldemort. We are determined to provide the open source community with the complete and painless data cycle that we enjoy – to enable even casual hadoop users to analyze data from their application at scale, to mine it for value and store it easily and reliably so that it can drive use and close the data loop. Look for new open source tools and projects from LinkedIn Analytics in the coming months that will help make this possible!

Hadoop drives many of our most powerful features at LinkedIn. About half of our Hadoop jobs are submitted by Apache Pig. This means that along with Azkaban and Voldemort, Pig is a large part of LinkedIn’s data cycle – the process behind features like People You May Know and Who Viewed My Profile.

LinkedIn and Apache Pig

I have used Pig intensively for about a year. During that time, I have come to love Pig for what it enables me to do: easily manipulate my data at scale, to turn raw data into data products. If Perl is the duct tape of the internet, and Hadoop is the kernel of the data center as computer, then Pig is the duct tape of Big Data. Pig lets me easily flow my data in parallel with simple commands. It lets me flow my data through dynamic languages like Python if I want to use SciPy, through simple UDFs in Java if I want to use a function repeatedly and share it with others, and ILLUSTRATE lets me check the output of my lengthy batch jobs and their custom functions without having to do a lengthy run of a long pipeline. Taken together, these features enable me to be productive.

I learned Pig not because I had a big data problem, but because I wanted to build a better interface for Hadoop (see: PigPen, WireIT, this demo video, this code). For a long time, I did not delve very deeply. There was no reason to do so: I didn’t have to know how to code in MapReduce – Pig ‘just worked.’ I issue SQLish commands in Pig Latin, and Pig parses these commands and creates and submits MapReduce jobs for me. This saves me from having to think too hard about the complexity of Java, MapReduce or Hadoop. I don’t like to think about anything but the problem I’m actually solving, and so while I have written Algebraic MapReduce jobs as Pig UDFs, I am unlikely to ever write a Java Hadoop job unless I absolutely have to.

Apache Pig is now fairly robust, but data-flows themselves can get complex fast. I’m pretty fluent in Pig Latin, but my code in any language rarely runs on the first try. With batch computing, running jobs repeatedly to debug them can take a long time and slow development to a crawl. One must often massage the Pig to command its will.

When I write Pig Latin code beyond a dozen lines, I check it in stages:

  • Write Pig Latin in TextMate (Saved in a git repo, otherwise I lose code)
  • Paste the code into the Grunt shell – Did it parse?
  • DESCRIBE the final output and each complex step – Did it still parse? Is the schema what I expected?
  • ILLUSTRATE the output – Does it still parse? Is the schema ok? Is the example data ok?
  • SAMPLE / LIMIT / DUMP the output – Does it still parse? Is the schema ok? Is the sampled/limited data sane?
  • STORE the final output and see if the job completes.
  • cat output_dir/part-00000 (followed by a quick ctrl-c to stop the flood) – Is the stored output on HDFS ok?

When you first tackle a complex task with Pig, that last step rarely happens on the first few tries. In time, you get more proficient.

As an incurious Pig user, I thought of Pig as a black box: a program with a command line. Nevertheless, I got to know the idiosyncrasies of each version as Pig matured from version 0.2 to 0.7 – unfixed bugs, unusual behaviors, and undocumented limitations. I never knew exactly why Pig behaved as it did, but I learned to get along with it.

Working on Pig

Several months ago I decided to work on the Pig project and I’m going after low hanging fruit the committers haven’t gotten around to and leaving the tough bits to them. Log analysis is a common use of Pig, and logs usually contain timestamps, so I want to add a Joda-Time DateTime data type to Pig.

But that is way too hard, so I’m going after boolean first. I checked out the code. I worked on it all weekend. I made a patch. I made many patches, actually. Time and again, I thought I was done, but I wasn’t. Booleans would load in grunt, so I thought it worked – but they wouldn’t store. I added physical storage code, so I could load and store. I emailed the LinkedIn Hadoop users list proclaiming victory… but it wouldn’t work on Hadoop. So I added Hadoop storage code, and it would load and store on Hadoop – but I couldn’t use operators to check for equality. I added code for ILLUSTRATE and it would illustrate, but I still couldn’t use booleans in a real job. This went on and on, and remains a work in progress.

During that weekend of long and frustrating hours of Pig hacking, the pattern became familiar. I was interacting with a different part of Pig each time I got a new kind of error. The hops from package to package in writing the patch corresponded to the stages of my long hours of stepwise data-flow checks in Grunt, as I had written Pig scripts most days over the course of the last year.

From a user’s perspective using the Grunt shell, this system seems like a cohesive entity – a single program – a complete (and somewhat irrational) Pig. It doesn’t seem that way anymore. Now that I’ve read the code, using Grunt is different. Knowing the way it all fits together at a high level – by tracing exceptions and seeing the package names of classes I’ve failed to implement because I didn’t know they existed or were required – I know that pig is actually segmented into many logical parts, independent arms that verify and process Pig Latin code independently and in different ways. The interface presented by grunt presents an illusion of wholeness that a deeper understanding of pig makes transparent – clear as illusion.

The Data Revolution

For me, understanding my work over the last year by understanding Pig was profound. It gave it more meaning, because strangely enough Pig has become a big part of my life. I’ve never much contributed to open source before, and I’m glad to be transitioning from a passive consumer of other people’s work to an active participant in an open source project. It is good to create openly, to give back. Open source is technical righteousness.

But more than that, this is an important time in computer science, and unlike many previous technical revolutions, this one is happening completely in the open. Like the integrated circuit before it, MapReduce is producing a paradigm shift that opens broad opportunities to produce new kinds of products from our massive collective backlog of data to help people in new and unprecedented ways. At LinkedIn we’ve amassed the world’s premiere data-set on the labor of professionals, and it is the mission of LinkedIn Analytics to leverage that deeply meaningful data to provide insight and value to our users. At LinkedIn Analytics data processing is both personal and meaningful, as the features we create enhance the working lives of tens of millions of people.

The Integrated Circuit solved the Tyranny of Numbers and unleashed Moore’s law, enabling a computerized, networked society. It did so with the considerable overhead of patent licensing and litigation. MapReduce is solving the Tyranny of Threads, enabling any company to process data at scale in parallel to extract real value from our most abundant and underutilized resource: information. It is doing it in the open, through free and open-source software, through the Apache Foundation, Hadoop and its sub-projects. We’ve gotten more efficient organizationally this time around.

If you love open source and you love big, meaningful data – we need you. Come join us. LinkedIn Analytics is hiring!

(Shout outs to Pete Skomoroch for acting as late-night editor, helping me dramatically improve this post!)

Share: Email | LinkedIn | Digg | Twitter

Close
E-mail It
Powered by ShareThis