{"id":28308,"date":"2022-01-29T06:00:00","date_gmt":"2022-01-29T14:00:00","guid":{"rendered":"https:\/\/insidebigdata.com\/?p=28308"},"modified":"2022-01-31T09:04:57","modified_gmt":"2022-01-31T17:04:57","slug":"high-performance-computing-essentials-and-why-analytics-is-forever-grateful","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2022\/01\/29\/high-performance-computing-essentials-and-why-analytics-is-forever-grateful\/","title":{"rendered":"High-Performance Computing Essentials \u2026 And Why Analytics is Forever Grateful"},"content":{"rendered":"\n

The basics of High-Performance Computing (HPC) are fairly well known by those acquainted with even a modest amount of computer science. Instead of relying on a single machine for computations, HPC approaches involve multiple machines, which produce two notable effects that directly benefit analytics.<\/p>\n\n\n\n

First, the performance is much more prodigious, swifter, and equitable to the time frames for solving business problems with data today, even on fleeting analysis of real-time datasets. Moreover, because multiple machines are involved, the limitations around memory, compute, and storage that can slow just preparing data for analytics (to say nothing of running analytics) with traditional computing methods are all but eradicated.<\/p>\n\n\n\n

Still, the particulars of how HPC clusters are able to function in unison for a single job, and just what the longstanding repercussions for Artificial Intelligence, analytics, queries, and data mining are, are typically enigmatic to business users.<\/p>\n\n\n\n

According to Katana Graph<\/a> Principal Engineer Bo Wu, the architectural foundation for HPC methods involving a scalable knowledge graph engine includes \u201ca partitioner, a runtime [scheduler], and a communication engine.\u201d These three elements work together to not only make HPC a reality, but also to master computations so that AI or any other computational load is facile.<\/p>\n\n\n\n

The Partitioner<\/strong><\/p>\n\n\n\n

A partitioner\u2019s role in a scalable graph engine employing HPC architecture<\/a> is to shard the workloads\u2014in this case the actual graph\u2014onto different machines. Sharding is a technique for distributing different pieces (or shards) of a computational process onto various computers to reap the performance gains HPC supports. Sharding is crucial for dividing up datasets that will otherwise slow the activities on a single machine. \u201cIf you want to compute a large graph on a number of machines, you need to partition your graph into different machines so each machine gets a small portion of the overall graph, because the overall graph is too big to fit into the memory of a single machine,\u201d noted Keshav Pingali, Katana Graph CEO.<\/p>\n\n\n\n

For example, if organizations have assembled a large graph to detect instances of fraud in real time, partitioning that graph onto what can include potentially hundreds of machines drastically increases the performance of this application. Without HPC methods for this and other use cases, firms are stuck with enormous upfront costs for purchasing machines that have tremendous storage capacities. \u201cThey\u2019re restricted to a single machine, so\u2026users have to buy these enormously expensive machines with two and three terabytes, and even that\u2019s not enough,\u201d Pingali said.<\/p>\n\n\n\n

The<\/strong> Runtime Scheduler<\/strong><\/p>\n\n\n\n

The runtime capabilities Wu described are responsible for orchestrating the endeavors of the different machines so they uniformly work on the same job. For a scalable knowledge graph query engine, that work might include any aspect of graph AI from graph neural networks<\/a> to clustering, as well as several algorithms for mining, analytics, and queries. Once a dataset (like a graph) has been partitioned, \u201cin order to compute on each machine on the graph that has been loaded into it, there is a runtime system and a set of data structures that have been optimized for the use of graph computing,\u201d Pingali explained.<\/p>\n\n\n\n

The runtime system is responsible for orchestrating various facets of the computations, which might pertain to embedding techniques, for example, on a medical knowledge graph to determine which specific compounds can spur the development of a new drug. \u201cThe runtime schedules operations on the local machines and can [optimize] all the computation resources to maximize performance,\u201d Wu indicated.<\/p>\n\n\n\n

The Communication Engine<\/strong><\/p>\n\n\n\n

The communication engine plays a critical role in HPC methods by enabling different machines to express to one another what parts of a single job they are doing, have done, or require another machine to finish. To query large graphs to determine supply chain optimization and routing, for example, the individual machines in a HPC cluster must relay to one another \u201cthis is how much of the sub-graph I\u2019ve matched so far, or the rest of this sub-graph is on another host, or I need to communicate and match what I have so far to the other host and let that other host pick up where the previous one left off to match the pattern graph,\u201d Katana Graph CTO Chris Rossbach remarked.<\/p>\n\n\n\n

The most credible HPC undertakings rely on Message Passing Interface (MPI)<\/a> standards for communicating. \u201cThese support idiomatic communication patterns that most HPC applications need to use but, because they\u2019re implemented by experts at the library level and optimized in different library implementations, they can be super fast,\u201d Rossbach mentioned.<\/p>\n\n\n\n

Unbeatable Analytics<\/strong><\/p>\n\n\n\n

The aforesaid partitioner, runtime capabilities, and communication engine are integral elements for using HPC architecture for rapid computations on a scalable graph engine. In turn, HPC is the blueprint for the fastest analytics available to the enterprise, whether that involves AI or not. This architecture is necessary for the most performant computations\u2014at enterprise scale\u2014that organizations can get to accelerate their data-driven undertakings, as well as increase their respective bottom lines.  <\/p>\n\n\n\n

About the Author<\/strong><\/p>\n\n\n\n

<\/p>\n\n\n\n

\"\"<\/figure><\/div>\n\n\n\n

Jelani Harper is an editorial consultant servicing the information technology market. He specializes in data-driven applications focused on semantic technologies, data governance and analytics.<\/em><\/p>\n\n\n\n

Sign up for the free insideBIGDATA newsletter<\/a>.<\/em><\/p>\n\n\n\n

Join us on Twitter: @InsideBigData1 \u2013 https:\/\/twitter.com\/InsideBigData1<\/a><\/em><\/p>\n","protected":false},"excerpt":{"rendered":"

In this contributed article, editorial consultant Jelani Harper discusses how the basics of High-Performance Computing (HPC) are fairly well known by those acquainted with even a modest amount of computer science. Instead of relying on a single machine for computations, HPC approaches involve multiple machines, which produce two notable effects that directly benefit analytics.<\/p>\n","protected":false},"author":10513,"featured_media":22370,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"jetpack_post_was_ever_published":false,"footnotes":""},"categories":[65,115,87,180,258,56,97,1],"tags":[314,429,96],"acf":[],"yoast_head":"\nHigh-Performance Computing Essentials \u2026 And Why Analytics is Forever Grateful - insideBIGDATA<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/insidebigdata.com\/2022\/01\/29\/high-performance-computing-essentials-and-why-analytics-is-forever-grateful\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"High-Performance Computing Essentials \u2026 And Why Analytics is Forever Grateful - insideBIGDATA\" \/>\n<meta property=\"og:description\" content=\"In this contributed article, editorial consultant Jelani Harper discusses how the basics of High-Performance Computing (HPC) are fairly well known by those acquainted with even a modest amount of computer science. Instead of relying on a single machine for computations, HPC approaches involve multiple machines, which produce two notable effects that directly benefit analytics.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/insidebigdata.com\/2022\/01\/29\/high-performance-computing-essentials-and-why-analytics-is-forever-grateful\/\" \/>\n<meta property=\"og:site_name\" content=\"insideBIGDATA\" \/>\n<meta property=\"article:publisher\" content=\"http:\/\/www.facebook.com\/insidebigdata\" \/>\n<meta property=\"article:published_time\" content=\"2022-01-29T14:00:00+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2022-01-31T17:04:57+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/insidebigdata.com\/wp-content\/uploads\/2019\/03\/Analytics_SHUTTERSTOCK.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"300\" \/>\n\t<meta property=\"og:image:height\" content=\"215\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Editorial Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:site\" content=\"@insideBigData\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Editorial Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/insidebigdata.com\/2022\/01\/29\/high-performance-computing-essentials-and-why-analytics-is-forever-grateful\/\",\"url\":\"https:\/\/insidebigdata.com\/2022\/01\/29\/high-performance-computing-essentials-and-why-analytics-is-forever-grateful\/\",\"name\":\"High-Performance Computing Essentials \u2026 And Why Analytics is Forever Grateful - insideBIGDATA\",\"isPartOf\":{\"@id\":\"https:\/\/insidebigdata.com\/#website\"},\"datePublished\":\"2022-01-29T14:00:00+00:00\",\"dateModified\":\"2022-01-31T17:04:57+00:00\",\"author\":{\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\"},\"breadcrumb\":{\"@id\":\"https:\/\/insidebigdata.com\/2022\/01\/29\/high-performance-computing-essentials-and-why-analytics-is-forever-grateful\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/insidebigdata.com\/2022\/01\/29\/high-performance-computing-essentials-and-why-analytics-is-forever-grateful\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/insidebigdata.com\/2022\/01\/29\/high-performance-computing-essentials-and-why-analytics-is-forever-grateful\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/insidebigdata.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"High-Performance Computing Essentials \u2026 And Why Analytics is Forever Grateful\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/insidebigdata.com\/#website\",\"url\":\"https:\/\/insidebigdata.com\/\",\"name\":\"insideBIGDATA\",\"description\":\"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/insidebigdata.com\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-US\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9\",\"name\":\"Editorial Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/\",\"url\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"contentUrl\":\"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g\",\"caption\":\"Editorial Team\"},\"sameAs\":[\"http:\/\/www.insidebigdata.com\"],\"url\":\"https:\/\/insidebigdata.com\/author\/editorial\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"High-Performance Computing Essentials \u2026 And Why Analytics is Forever Grateful - insideBIGDATA","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/insidebigdata.com\/2022\/01\/29\/high-performance-computing-essentials-and-why-analytics-is-forever-grateful\/","og_locale":"en_US","og_type":"article","og_title":"High-Performance Computing Essentials \u2026 And Why Analytics is Forever Grateful - insideBIGDATA","og_description":"In this contributed article, editorial consultant Jelani Harper discusses how the basics of High-Performance Computing (HPC) are fairly well known by those acquainted with even a modest amount of computer science. Instead of relying on a single machine for computations, HPC approaches involve multiple machines, which produce two notable effects that directly benefit analytics.","og_url":"https:\/\/insidebigdata.com\/2022\/01\/29\/high-performance-computing-essentials-and-why-analytics-is-forever-grateful\/","og_site_name":"insideBIGDATA","article_publisher":"http:\/\/www.facebook.com\/insidebigdata","article_published_time":"2022-01-29T14:00:00+00:00","article_modified_time":"2022-01-31T17:04:57+00:00","og_image":[{"width":300,"height":215,"url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2019\/03\/Analytics_SHUTTERSTOCK.jpg","type":"image\/jpeg"}],"author":"Editorial Team","twitter_card":"summary_large_image","twitter_creator":"@insideBigData","twitter_site":"@insideBigData","twitter_misc":{"Written by":"Editorial Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/insidebigdata.com\/2022\/01\/29\/high-performance-computing-essentials-and-why-analytics-is-forever-grateful\/","url":"https:\/\/insidebigdata.com\/2022\/01\/29\/high-performance-computing-essentials-and-why-analytics-is-forever-grateful\/","name":"High-Performance Computing Essentials \u2026 And Why Analytics is Forever Grateful - insideBIGDATA","isPartOf":{"@id":"https:\/\/insidebigdata.com\/#website"},"datePublished":"2022-01-29T14:00:00+00:00","dateModified":"2022-01-31T17:04:57+00:00","author":{"@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9"},"breadcrumb":{"@id":"https:\/\/insidebigdata.com\/2022\/01\/29\/high-performance-computing-essentials-and-why-analytics-is-forever-grateful\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/insidebigdata.com\/2022\/01\/29\/high-performance-computing-essentials-and-why-analytics-is-forever-grateful\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/insidebigdata.com\/2022\/01\/29\/high-performance-computing-essentials-and-why-analytics-is-forever-grateful\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/insidebigdata.com\/"},{"@type":"ListItem","position":2,"name":"High-Performance Computing Essentials \u2026 And Why Analytics is Forever Grateful"}]},{"@type":"WebSite","@id":"https:\/\/insidebigdata.com\/#website","url":"https:\/\/insidebigdata.com\/","name":"insideBIGDATA","description":"Your Source for AI, Data Science, Deep Learning & Machine Learning Strategies","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/insidebigdata.com\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-US"},{"@type":"Person","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/2949e412c144601cdbcc803bd234e1b9","name":"Editorial Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/insidebigdata.com\/#\/schema\/person\/image\/","url":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","contentUrl":"https:\/\/secure.gravatar.com\/avatar\/e137ce7ea40e38bd4d25bb7860cfe3e4?s=96&d=mm&r=g","caption":"Editorial Team"},"sameAs":["http:\/\/www.insidebigdata.com"],"url":"https:\/\/insidebigdata.com\/author\/editorial\/"}]}},"jetpack_featured_media_url":"https:\/\/insidebigdata.com\/wp-content\/uploads\/2019\/03\/Analytics_SHUTTERSTOCK.jpg","jetpack_shortlink":"https:\/\/wp.me\/p9eA3j-7mA","jetpack-related-posts":[{"id":26807,"url":"https:\/\/insidebigdata.com\/2021\/08\/03\/insidebigdata-guide-for-higher-education-part-3\/","url_meta":{"origin":28308,"position":0},"title":"insideBIGDATA Guide for Higher Education \u2013 Part 3","date":"August 3, 2021","format":false,"excerpt":"[Sponsored Post] This insideBIGDATA technology guide co-sponsored by Dell Technologies and AMD, insideBIGDATA Guide for Higher Education, discusses how higher education leaders today want to improve student and institutional success through digital transformation. Dell Technologies is uniquely positioned to help at every step of the way, leveraging a robust portfolio\u2026","rel":"","context":"In "Academic"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2021\/07\/insideBIGDATA-Guide-for-Higher-Education-cover.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":5079,"url":"https:\/\/insidebigdata.com\/2013\/10\/22\/slidecast-sgi-speed-high-performance-analysis-big-data-innovations\/","url_meta":{"origin":28308,"position":1},"title":"Slidecast: SGI to Speed High Performance Analysis with Big Data Innovations","date":"October 22, 2013","format":false,"excerpt":"Drawing on extensive expertise and experience in HPC and managing high volume storage, SGI is introducing new solutions to perform Big Data analytics faster. Now the enterprise can achieve extreme capacity and scale needed for Big Data storage, and manage storage investments more cost effectively.","rel":"","context":"In "Big Data Hardware"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2013\/10\/sgi.png?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":14580,"url":"https:\/\/insidebigdata.com\/2016\/03\/02\/continuum-analytics-brings-serious-analytics-to-hadoop\/","url_meta":{"origin":28308,"position":2},"title":"Continuum Analytics Brings Serious Analytics to Hadoop","date":"March 2, 2016","format":false,"excerpt":"Continuum Analytics, the creator and driving force behind Anaconda, the leading modern open source analytics platform powered by Python, announced advancements in Anaconda bringing high performance advanced analytics to Hadoop.","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":19144,"url":"https:\/\/insidebigdata.com\/2017\/10\/13\/high-performance-computing-answering-big-data-dilemma\/","url_meta":{"origin":28308,"position":3},"title":"High Performance Computing: Answering the Big Data Dilemma","date":"October 13, 2017","format":false,"excerpt":"In this special guest feature, Jeff Reser, Global Product Marketing Manager of SUSE, suggests that as HPC\u2019s prowess in business expands, so does its ability to solve a variety of data management problems. Individuals struggling to tackle Big Data\u2019s most complex challenges should increasingly look at HPC to deliver the\u2026","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2017\/10\/reser_jeff.jpg?resize=350%2C200&ssl=1","width":350,"height":200},"classes":[]},{"id":13900,"url":"https:\/\/insidebigdata.com\/2015\/10\/20\/efficiency-big-data-meets-hpc-in-financial-services\/","url_meta":{"origin":28308,"position":4},"title":"Efficiency: Big Data Meets HPC in Financial Services","date":"October 20, 2015","format":false,"excerpt":"Converging High Performance Computing (HPC) and Lustre* parallel file systems with Hadoop\u2019s MapReduce for Big Data analytics can eliminate the need for Hadoop\u2019s infrastructure and speeding up the entire analysis. Convergence is a solution of interest for companies with HPC already in their infrastructure, such as the financial services Industry\u2026","rel":"","context":"In "Big Data"","img":{"alt_text":"","src":"","width":0,"height":0},"classes":[]},{"id":12988,"url":"https:\/\/insidebigdata.com\/2015\/04\/16\/ryfts-open-platform-supercharges-analytics-performance-by-eliminating-io-bottlenecks\/","url_meta":{"origin":28308,"position":5},"title":"Ryft’s Open Platform Supercharges Analytics Performance by Eliminating I\/O Bottlenecks","date":"April 16, 2015","format":false,"excerpt":"A new IDC Technology Spotlight \u201cRyft's Open Platform Aims to Supercharge Analytics Performance by Eliminating I\/O Bottlenecks\u201d sponsored by Ryft Systems Inc. examines the issues that are driving organizations to embrace HPDA. The paper looks at the role the new Ryft ONE open platform takes to supercharge performance on even\u2026","rel":"","context":"In "Analytics"","img":{"alt_text":"RYFT_table","src":"https:\/\/i0.wp.com\/insidebigdata.com\/wp-content\/uploads\/2015\/04\/RYFT_table.jpg?resize=350%2C200","width":350,"height":200},"classes":[]}],"_links":{"self":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/28308"}],"collection":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/users\/10513"}],"replies":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/comments?post=28308"}],"version-history":[{"count":0,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/posts\/28308\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media\/22370"}],"wp:attachment":[{"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/media?parent=28308"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/categories?post=28308"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/insidebigdata.com\/wp-json\/wp\/v2\/tags?post=28308"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}