{"id":32613,"date":"2023-06-13T06:00:00","date_gmt":"2023-06-13T13:00:00","guid":{"rendered":"https:\/\/insidebigdata.com\/?p=32613"},"modified":"2023-06-09T10:56:53","modified_gmt":"2023-06-09T17:56:53","slug":"state-of-data-quality-report","status":"publish","type":"post","link":"https:\/\/insidebigdata.com\/2023\/06\/13\/state-of-data-quality-report\/","title":{"rendered":"State of Data Quality Report"},"content":{"rendered":"
\n
\"\"<\/figure><\/div>\n\n\n

Bigeye<\/a>, the data observability company, announced the results of its 2023 State of Data Quality<\/a> survey. The report sheds light on the most pervasive problems in data quality today.<\/p>\n\n\n\n

The report, which was researched and authored by Bigeye, consisted of answers from 100 survey respondents. At least 63 came from mid-to-large cloud data warehouse customers (with a spend of more than $500k per annum) who have some form of data monitoring in place, whether third-party or built in-house.<\/p>\n\n\n\n

First line of defense against data issues<\/strong><\/p>\n\n\n\n

Bigeye\u2019s survey found that data engineers are the first line of defense in managing data issues, followed closely by software engineers. The role of data engineer has moved on par with software engineering. Like software engineers, data engineers are in charge of a product – the data product – that increasingly demands software-like levels of process, maintenance, and code review.<\/p>\n\n\n\n

Desire for automation<\/strong><\/p>\n\n\n\n

Respondents who used third-party data monitoring solutions found about a 2x to 3x ROI over in-house solutions. They also noted that at full utilization, third-party data monitoring solved for two issues: fractured infrastructure, and anomalous data. They further reported that third-party data monitoring solutions had better test libraries, and a broader perspective on data problems. <\/p>\n\n\n\n

Data incident frequency<\/strong><\/p>\n\n\n\n

Research revealed that companies experience a median of five to ten data incidents over a period of three months. These incidents range from severe enough to impact the company’s bottom line, to reducing engineer productivity. These incidents take an average of 48 hours to troubleshoot.<\/p>\n\n\n\n

Organizations with more than five data incidents a month are essentially lurching from incident to incident, with little ability to trust data or invest in larger data infrastructure projects. They are largely performing reactive over proactive data quality work.<\/p>\n\n\n\n

Other important findings from the survey<\/strong><\/p>\n\n\n\n

There were other interesting insights revealed through survey results, including:<\/p>\n\n\n\n