{"id":13317,"date":"2015-06-30T06:00:08","date_gmt":"2015-06-30T13:00:08","guid":{"rendered":"http:\/\/insidebigdata.com\/?p=13317"},"modified":"2015-06-29T15:48:10","modified_gmt":"2015-06-29T22:48:10","slug":"big-data-technology-for-scientific-research","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2015\/06\/30\/big-data-technology-for-scientific-research\/","title":{"rendered":"Big Data Technology for Scientific Research"},"content":{"rendered":"

This article is the third in\u00a0an editorial\u00a0series<\/a> with a goal to provide a road map for scientific researchers wishing to capitalize on the rapid growth of big data technology for collecting, transforming, analyzing, and visualizing large scientific data sets.
\n<\/em><\/p>\n

In the last article<\/a>, we took a look at the primary motivators for scientific researchers to engage the big data technology stack. The complete\u00a0insideBIGDATA Guide to Scientific Research <\/em>is available\u00a0for download from the insideBIGDATA White Paper Library<\/a>.<\/p>\n

\"insideBIGDATA_Guide_Research_feature\"<\/a>Big Data Technology for Scientific Research<\/strong><\/p>\n

The continued and rapid evolution of big data technology and services has formed a fertile foundation for scientific applications in the past several years. We\u2019ve seen big data hardware and software solutions promulgate access to analytics and methods of statistical learning like never before. And one thing researchers have learned along the way is that management of computing resources is one of the primary questions to be answered with big data. It is not just a case of determining the scale of resources needed for a project, but also how to configure them, all within the available budget. For example, running a large project on fewer machines might save on hardware costs but will result in a longer project timeline. In some cases, scientific big data are being stored in the cloud instead of on conventional hardware in a research lab. Instead of having to invest in the infrastructure of an on-premises High Performance Computing (HPC) cluster to analyze the data, some researchers are using HPC and big data methods in the cloud. The disadvantage of the cloud is that large data sets must be transferred to cloud storage.<\/p>\n

In this section we\u2019ll examine specific areas of big data technology that scientific researchers are deploying in order to see significant increases in their ability to manage the scientific data deluge.<\/p>\n

Colliding Worlds of HPC and Big Data<\/strong><\/p>\n

In order to address the needs of data-centric scientific research projects, the profiles of traditional HPC and big data are merging and becoming closely intertwined. The compute nature of HPC is finding significant benefit from big data analytics and its ability to process high volume, high velocity data sets. The current most effective software platform for big data analytics\u2014Hadoop \u2014has its classic architecture consisting of HDFS and MapReduce, running on commodity cluster nodes, and the HPC environment has a different architecture where compute is distinct from the storage solution. You\u2019d like to leverage your current investment in HPC by doing big data analytics on the architecture you already have. This is where two worlds come together.<\/p>\n

An area of scientific research that\u2019s benefiting from HPC and big data merging is genomics. The advancement of this area of research depends on the availability of HPC because of the compute resources needed to process genomic data sets. But these problems also require solving big data issues like analyzing the data, making sense of what\u2019s in the data, identifying what patterns emerge from the data, and more. Essentially, every aspect of what genomic researchers do is becoming an opportunity to capture, analyze and use big data. This is the same net effect seen from the perspective of many other scientific disciplines.<\/p>\n

Data Sources and Data Integration<\/strong><\/p>\n

Scientific researchers routinely collect extremely large data sets, primarily for computational analysis with an HPC\u00a0 system. These data sets can also be analyzed with big data tools to look for valuable insights with data visualization\u00a0 tools or advanced analysis algorithms. The difference between HPC and big data analytics is primarily that HPC is CPU bound, whereas big data analytical problems are IO bound. As the two environments continue to merge, researchers can apply big data analytics against a primarily HPC data set without moving the data set from the HPC environment to a Hadoop Cluster with HDFS.<\/p>\n

A paradigm shift in big data analytics relative to scientific research use cases, as well as other use cases where there is value from analyzing HPC data sets, necessitates a new direction away from the common HDFS architecture, and towards using MapReduce on the Lustre parallel file system. The Intel\u00ae Enterprise Edition for Lustre software unleashes the performance and scalability of the Lustre for HPC workloads, including big data applications becoming common within today\u2019s research labs. Dell built the reference architecture where Lustre is used in place of HDFS. Further, Dell layers analytics on top of this architecture as well.<\/p>\n

A key component to connecting the Hadoop and Lustre ecosystems is the Intel Hadoop Adapter for Lustre plug-in (Intel HAL). Intel HAL is bundled with the Intel Enterprise Edition for Lustre software. It allows the users to run MapReduce jobs directly on a Lustre file system. The immediate benefit is that Lustre is able to deliver faster, stable and easily managed storage for the MapReduce applications directly. A potential long term benefit using Lustre as the underlying Hadoop storage would be a higher raw capacity available when compared to HDFS due to the three time replication as well as the performance benefits of running Lustre on InfiniBand connectivity.<\/p>\n

Researchers are interested in multiple additional classes of data sets. Many of them may be on premise and consist of multiple types of data. Other data sources may come from publicly available sites, or purchased from one of the many services that provide data sources to labs. Dell\u2019s Data Integration Platform as a Service called Boomi could be used for such applications.<\/p>\n

Hadoop<\/strong><\/p>\n

The Hadoop distributed computing architecture increasingly is being deployed for scientific applications requiring big data capabilities, specifically managing, collecting and analyzing the data. Dell\u2122 Apache\u2122 Hadoop\u00ae solutions for big data provide an open source, end-to-end scalable infrastructure that allows you to:<\/p>\n