Concurrent raises $900K to make Hadoop easier

Concurrent, the company providing the Cascading data workflow API, has raised a $900,000 seed round to capitalize on the newfound excitement around Hadoop. The funding came from Rembrandt Venture Partners, True Ventures (see disclosure below) and several angel investors.

Cascading, which Concurrent Founder and CEO Chris Wensel created, is an open-source API for creating and running data workflows atop Hadoop clusters. It’s an alternative to MapReduce, the standard framework for writing Hadoop applications, as well as Hive, the Facebook-created Apache project that provides data warehouse features for Hadoop environments. The Concurrent web site describes Cascading like this:

The processing API lets the developer quickly assemble complex distributed processes without having to “think” in MapReduce. And to efficiently schedule them based on their dependencies and other available meta-data.

Concurrent has been around since 2007, but only now is there enough activity around Hadoop and big data to justify putting much effort into building new products and hiring a team of engineers, said Wensel.

Certainly, Hadoop is at its pinnacle right now, with EMC, MapR and Hortonworks all making very public entrances into the distribution space lately to join incumbents such as Cloudera, IBM and Amazon Web Services (with Elastic MapReduce). Now that companies are comfortable with the prospect of Hadoop, and possibly using it to some degree, Wensel thinks they’re ready to start hearing about MapReduce alternatives.

Looking forward, Wensel thinks there’s an opportunity to expand Cascading support beyond Hadoop distributions (it’s currently certified for Apache Hadoop, MapR, EMC and Elastic MapReduce) and into new Hadoop-based “forks, derivatives and re-imaginings” that gain enough traction. Longer term, he sees an opportunity for a common API to support analytic workflows across a variety of distributed systems, Hadoop-based or not.

In the near future, though, Cascading users can look forward to version 2.0 in the fall, which includes a number of significant improvements, including the ability to use system memory for faster analysis of small datasets. He also said Concurrent plans to create products complementary to the Cascading framework that will help monitor monitor workflows and let users make better decisions by giving them more insights.

Although Concurrent’s seed funding is relatively small compared some of the other big data investments we’ve seen lately, it’s significant. I predicted in my second-quarter wrap-up for GigaOM Pro that we’ll start seeing more investment in higher-level Hadoop tools, and Cascading is one of them.

With the distribution layer locked down, there’s plenty of room for alternative data-processing frameworks such as Cascading and turnkey analytics products such as Zettaset, which just raised $3 million itself, to steal some of the spotlight and make it easier to take advantage of Hadoop’s parallel-processing prowess.

Disclosure: Concurrent is backed by True Ventures, a venture capital firm that is an investor in the parent company of this blog, Giga Omni Media. Om Malik, the founder of Giga Omni Media, is also a venture partner at True.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

View the original article here

This post was made using the Auto Blogging Software from WebMagnates.org This line will not appear when posts are made after activating the software to full version.

What it means if Yahoo Hadoop spinoff doesn’t do distribution

It looks like all the speculation about how Yahoo’s Hadoop spinoff company, Hortonworks, will affect Cloudera and other companies providing Hadoop-based products might have been overblown. During a phone call earlier this week, Hortonworks CEO Eric Baldeschwieler told me the company is still figuring out its strategy around offering a Hadoop distribution, which could be good news for presumed competitors such as Cloudera.

The ambivalence appears tied to the company’s narrow focus on improving Apache Hadoop and making it the go-to distribution. Baldeschwieler said that Hortonworks’ core business model will be around offering support and services, as well as helping drive Apache to “bridge the gap between what [Hadoop] is and what it can be.” The latter goal, of course, means working hard to improve the core Apache Hadoop distribution to make it more scalable, reliable and generally flexible.

If Hortonworks doesn’t offer a distribution, it might be because it doesn’t want to waste resources. It would have to build its own distribution and then work within Apache to get any improvements built into that code, resulting in a doubling up of effort and a somewhat unnatural split of allegiances given Hortonworks’ professed support for Apache Hadoop. This is the same issue Yahoo was trying to avoid earlier this year when it discontinued its own distribution and recommited all its efforts into Apache Hadoop. It looks now like that move was just setting the stage for the Hortonworks launch.

Already, Baldeschwieler said, a number of key features from Yahoo are slated to be included in upcoming Apache Hadoop releases. These features include a new MapReduce engine, federated storage for HDFS and a major improvement for how HBase interacts with HDFS. What all the work means, he explained, is that Apache Hadoop will be more stable, more scalable and more dynamic. In fact, he said, with the next scheduled release, developers will be able to use alternative processing frameworks beside Hadoop MapReduce.

Good news for some, bad for others

A Hortonworks focused entirely on Apache could be good news for Cloudera. In that case, it’s still very much in its current position of integrating and hardening the suite of Apache Hadoop products into its own open source distribution, then selling services and management software on top of it. The big difference will be that Apache Hadoop will look a lot more appealing because it will have Hortonworks providing expert service. But Cloudera doesn’t really have to change its story.

A service-focused Hortonworks might not be so good for companies such as MapR, which are pushing proprietary or semi-proprietary Hadoop distributions. The fewer distributions and the more focused they are around Apache Hadoop, the less appealing outliers might look to users concerned about being locked into their vendor. Baldeschwieler says he thinks the market will be big enough for value-added distributions like what MapR offers, but noted that Apache Hadoop has already proven itself within large enterprise and will continue to get better.

For example, he explained, Apache has been working hard to integrate some of the code that Facebook has introduced from its Hadoop deployment. At the time it announced its Hadoop distributions in May, EMC said its Community edition is based on Facebook’s code, but now Baldeschwieler has heard EMC is reconsidering that decision and might support the core Apache code instead. That hardly constitutes hard evidence, but it’s noteworthy because EMC is integrating MapR’s proprietary storage technology in its Enterprise edition release.

“What we don’t want to see happen,” Baldeschwieler said, “is the Hadoop market start to look like the Unix market in the ‘80s.” The more support there is around Apache Hadoop, he explained, the less chance there is for a Unix-like lost decade of competing distributions before Linux came around in the ’90s and became the center of the non-Windows universe. He thinks Apache Hadoop is and should be the Linux of big data.

Whatever path Hortonworks takes, though, Baldeschwieler thinks all the action around Hadoop will make it very difficult for alternative technologies, such as Microsoft Dryad and LexisNexis’s HPCC Systems to catch. “I think they’ve got their work cut out for them if they want to compete with the Hadoop community,” he said. Because even if the companies involved are at odds, they’re still a very big community.

Feature image courtesy of Flickr user miheco.

Related research and analysis from GigaOM Pro:
Subscriber content. Sign up for a free trial.

window.fbAsyncInit = function() {FB.init({appId: 180650338636285, status: true, cookie: true, xfbml: true});FB.api({method: 'links.getStats',urls: 'http://gigaom.com/cloud/what-it-means-if-yahoo-hadoop-spinoff-doesnt-do-distribution/'},function(response) {jQuery('#react-fb-count-button').html(response[0].commentsbox_count);});FB.Event.subscribe('comment.create', function(response) {var ajaxurl = 'http://gigaom.com/wp-admin/admin-ajax.php?action=new_fb_comment&post_id=';jQuery.get(ajaxurl + 379936);});};var e = document.createElement('script');e.type = 'text/javascript';e.src = document.location.protocol + '//connect.facebook.net/en_US/all.js';e.async = true;document.getElementById('fb-root').appendChild(e);

var _comscore = _comscore || []; _comscore.push({ c1: "2", c2: "6036014" }); (function() { var s = document.createElement("script"), el = document.getElementsByTagName("script")[0]; s.async = true; s.src = (document.location.protocol == "https:" ? "https://sb" : "http://b") + ".scorecardresearch.com/beacon.js"; el.parentNode.insertBefore(s, el); })();

Click to log in with: Not you? Remember me Submitting comment...
;(function($){$.fn.trackClick = function(){// track the clicktry {_gaq.push(['_trackEvent', this.parents('[id!=""]:first').get(0).id, 'clicked', (this.text() || this.children('img:first').attr('alt'))]);}catch (err) {}// wait a moment for the tracking to process, then follow the linksetTimeout('document.location = "' + $(this).attr('href') + '"', 200);};$('#brand-explorer a, #navigation a, .widget-wrap a').click(function () {$(this).trackClick();return false;}); })(jQuery);

View the original article here

This post was made using the Auto Blogging Software from WebMagnates.org This line will not appear when posts are made after activating the software to full version.