{"id":19659,"date":"2017-12-26T08:30:32","date_gmt":"2017-12-26T16:30:32","guid":{"rendered":"https:\/\/insidebigdata.com\/?p=19659"},"modified":"2017-12-27T15:31:19","modified_gmt":"2017-12-27T23:31:19","slug":"heroic-data-engineer-lending-helping-hand-data-drowned-scientists","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2017\/12\/26\/heroic-data-engineer-lending-helping-hand-data-drowned-scientists\/","title":{"rendered":"The heroic Data Engineer – Lending a Helping Hand to Data Drowned Scientists"},"content":{"rendered":"

A recent Forbes article<\/a> on the 10 Predictions for AI, Big Data, and Analytics in 2018 states that Data engineer will become the hot new job title, displacing its sibling role of Data Scientist. Gil Press goes on to write that Indeed.com had 13% of data-related job postings for data engineers and less than 1% for data scientists.<\/p>\n

Intrigued, I looked at the job descriptions of Data engineer job postings by leading data-driven companies like Amazon and Facebook on LinkedIn. Strong Data Warehouse skills with a thorough knowledge of Data Extraction, Transformation, loading (ETL) processes and Data Pipeline construction expertise stood out as the essential and\u00a0 basic qualifications of an ideal Data Engineer.<\/p>\n

\"\"<\/p>\n

\"\"<\/p>\n

 <\/p>\n

Who is a Data Engineer Anyway?<\/strong><\/p>\n

If construction engineering deals with the designing, planning, construction, and management of physical infrastructures like buildings,roads, and tracks, data engineering applies the same intricacies to data.<\/p>\n

A <\/em><\/strong>data engineer<\/em><\/strong><\/a> plans, designs, constructs and maintains a reliable architecture for a steady flow of clean and structured data that is ready for further analysis and is viable for production environment.<\/em><\/strong><\/p>\n

Data engineering is gaining prominence because of the fact that\u00a0 organizations are choked by a deluge of data ranging from logs in multiple formats to valuable business information lying unstructured in the vast Internet.<\/p>\n

As data scientists and citizen data scientists with statistical and programming flair began proliferating, their common pain point lay in managing and maintaining the enormous volume of data. Data scientists who had to analyze and build models spent 80% of their time spotting and cleaning data<\/a>. That\u2019s when the need to fork their responsibilities came in, giving rise to a new breed of macho data engineers.<\/p>\n

A data engineer comes to the rescue by understanding the data needed for a business, identifying the relevant new data sources, extracting the data in usable formats, making sure the data is error free and loading them for data scientists and analysts to work on.<\/p>\n

The Data Engineering Tool-set<\/strong><\/p>\n

The work of a data engineer often overlaps with those of a data architect, a database administrator, and a software engineer implying a foretaste of each of their skill set is desirable. While a data architect or administrator is confined to their position of planning and maintenance of data infrastructure, a data engineer encompasses their titles to present a palatable form of data right from its origin to its final analysis exhibition.<\/p>\n

Suitable skills<\/a> for a data engineer would thus include:<\/p>\n