Notes about Cutting-Edge Technologies and Everything

This month’s release delivers very useful features like Apache Arrow Flight with Python, full support for CDP 7.1, security enhancements for Oracle connections, a new support bundle and much more. This blog post highlights the following updates: Arrow Flight clients Query support bundle Kerberos support for Dremio-Oracle connections User/job metrics available in the UI Continue reading >>>

According to Gartner: “By 2022, organizations using multiple go-to-market approaches for digital commerce will outperform noncommerce organizations by 30 percentage points in sales growth." Covid-19 has forced many brands to accelerate their digital-first commerce strategies, sooner than they had planned, in an effort to keep up with customer demands and drive revenue.    Whether your brand was an early adopter, or is still struggling to implement a robust digital commerce strategy, the Gartner Predicts 2021:  COVID-19 Drives Accelerated Shift to Digital and Commerce Model Evolution report will help you prepare for 2021. We believe, in this report you will discover:  5 key digital commerce predictions for 2021 and beyond The market implications of these predictions  How you can embrace new market realities to propel your business Get the report >>>

DataOps techniques can provide a more agile and collaborative approach to building and managing data pipelines. The pandemic has accelerated the need for data and analytics leaders to deliver data and analytics insight faster, with higher quality and resiliency in the face of constant change. Organizations need to make better-informed and faster decisions with a focus on automation, real-time risk assessment and mitigation, continuous value delivery and agility. The point of DataOps is to change how people collaborate around data and how it is used in the organization  As a result, data and analytics leaders are increasingly applying DataOps techniques that provide a more agile and collaborative approach to building and managing data pipelines. What is DataOps? Gartner defines DataOps as a collaborative data management practice focused on improving the communication, integration and automation of data flows between data managers and data consumers across an organization. “The poi...

Solutions Review’s Data Integration Tools Vendors to Watch is an annual listing of solution providers we believe are worth monitoring. Companies are commonly included if they demonstrate a product roadmap aligning with our meta-analysis of the marketplace. Other criteria include recent and significant funding, talent acquisition, a disruptive or innovative new technology or product, or inclusion in a major analyst publication. Data integration tools vendors are increasingly being disrupted by cloud connectivity, self-service, and the encroachment of data management functionality. As data volumes grow, we expect to see a continued push by providers in this space to adopt core capabilities of horizontal technology sectors. Organizations are keen on adopting these changes as well, and continue to allocate resources toward the providers that can not only connect data lakes and Hadoop to their analytic frameworks, but cleanse, prepare, and govern data. The next generation of tools will offe...

Our aspiration to augment and improve every aspect of business and life with data, demands a paradigm shift in how we manage data at scale. While the technology advances of the past decade have addressed the scale of volume of data and data processing compute, they have failed to address scale in other dimensions: changes in the data landscape, proliferation of sources of data, diversity of data use cases and users, and speed of response to change. Data mesh addresses these dimensions, founded in four principles: domain-oriented decentralized data ownership and architecture, data as a product, self-serve data infrastructure as a platform, and federated computational governance. Each principle drives a new logical view of the technical architecture and organizational structure. The original writeup, How to Move Beyond a Monolithic Data Lake to a Distributed Data Mesh - which I encourage you to read before joining me back here - empathized with today’s pain points of architectural and or...

Adoption is the use of a new technology. Engagement is the amount of involvement with a technology. This small semantic difference is the key to unlocking the full potential of new applications. Some applications will have inherently higher engagement than others. For example, a frontline healthcare worker will have a high level of engagement with an electronic medical record because it contains essential information for treating patients. There are other technologies we introduce to make processes easier and faster, even if they are not required. For example, a data analyst may or may not choose to use a metadata management application to learn about the data they use every day. While using the application will make their work easier and faster, they can choose to do their work without it. Engagement is about utilization — increasing the likelihood that people will use the application. Continue reading >>>

Have we entered the Golden Age of Data? Modern enterprises are collecting, producing, and processing more data than ever before. According to a February 2020 IDG survey of data professionals, average corporate data volumes are increasing by 63% per month. 10% of respondents even reported that their data volumes double every month. Large companies are investing heavily to transform themselves into data-driven organizations that can quickly adapt to the fast pace of a modern economy. They gather huge amounts of data from customers and generate reams of data from transactions. They continuously process data in an attempt to personalize customer experiences, optimize business processes, and drive strategic decisions. The Real Challenge with Data In theory, breakthrough open-source technologies, such as Spark, Kafka, and Druid are supposed to help just about any organization benefit from massive amounts of customer and operational data just like they benefit Facebook, Apple, Google, Microso...

Metadata management is a core aspect of an organization’s ability to manage its data and information assets. The term “metadata” describes the various facets of an information asset that can improve its usability throughout its life cycle. Metadata and its uses go far beyond technical matters. Metadata is used as a reference for business-oriented and technical projects, and lays the foundations for describing, inventorying and understanding data for multiple use cases. Use-case examples include data governance, security and risk, data analysis and data value. The market for metadata management solutions is complex because these solutions are not all identical in scope or capability. Vendors include companies with one or more of the following functional capabilities in their stand-alone metadata management products (not all vendors offer all these capabilities, and not all vendor solutions offer these capabilities in one product): Metadata repositories — Used to document and manage meta...

Large-scale companies serve millions or even billions of people who depend on the services these companies provide for their everyday needs. To keep these services running and delivering meaningful experiences, the teams behind them need to find the most relevant and accurate information quickly so that they can make informed decisions and take action. Finding the right information can be hard for several reasons. The problem might be discovery — the relevant table might have an obscure or nondescript name, or different teams might have constructed overlapping data sets. Or, the problem could be one of confidence — the dashboard someone is looking at might have been superseded by another source six months ago.  Many companies, such as Airbnb, Lyft, Netflix, and Uber, have built their own custom solutions for this challenge. For us, it was important to make the data discovery process simple and fast. Funneling everything through data experts to locate the necessary data each time we...

Apache Pulsar's unique features such as tiered storage, stateless brokers, geo-aware replication, and multi-tenancy may be a reason to choose it over Apache Kafka. Today, many data architects, engineers, dev-ops, and business leaders are struggling to understand the pros and cons of Apache Pulsar and Apache Kafka. As someone who has worked with Kafka in the past, I wanted to compare these two technologies.  If you are looking for insights on when to use Pulsar, here are 10 advantages of the technology that might be the deciding factors for you. Continue reading >>>