Notes about Cutting-Edge Technologies and Everything

  Database management systems continue their move to the cloud — a move that is producing an increasingly complex landscape of vendors and offerings. This Magic Quadrant will help data and analytics leaders make the right choices in a complex and fast-evolving market. Strategic Planning Assumptions By 2025, cloud preference for data management will substantially reduce the vendor landscape while the growth in multicloud will increase the complexity for data governance and integration. By 2022, cloud database management system (DBMS) revenue will account for 50% of the total DBMS market revenue. These DBMSs reflect optimization strategies designed to support transactions and/or analytical processing for one or more of the following use cases:     Traditional and augmented transaction processing     Traditional and logical data warehouse     Data science exploration/deep learning     Stream/event processing   ...

  Strategic Planning Assumptions Through 2022, manual data management tasks will be reduced by 45% through the addition of machine learning and automated service-level management. By 2023, AI-enabled automation in data management and integration will reduce the need for IT specialists by 20%.  Read report >>>

This report assesses 20 vendors of platforms that data scientists and others can use to source data, build models and operationalize machine learning. It will help them make the right choice from a crowded field in a maturing DSML platform market that continues to show rapid product development. Market Definition/Description Gartner  defines a data science and machine learning (DSML) platform as a core product and supporting portfolio of coherently integrated products, components, libraries and frameworks (including proprietary, partner-sourced and open-source). Its primary users are data science professionals, including expert data scientists, citizen data scientists, data engineers, application developers and machine learning (ML) specialists. The core product and supporting portfolio: Are sufficiently well-integrated to provide a consistent “look and feel.” Create a user experience in which all components are reasonably interoperable in support of an analytics pipeline. The...

  Data integration tools address a wide range of use cases that rely on key data delivery capabilities. This research helps data and analytics leaders identify vendors’ relative strengths across these capabilities and select the right tool in support of their data management solutions. Key Findings All data integration tool vendors were rated as “meeting/exceeding expectations” for their support for bulk/batch data movement and streaming data integration. However, support for other data delivery styles (data virtualization and data replication, for example) is less consistently delivered across the range of products evaluated. Active metadata is now critical as organizations continue to focus on metadata-driven optimization and automation of integration flows. The cohort of products in this evaluation averaged 3.3 out of a possible 5.0. While adequate, these capabilities must improve. Data virtualization has become less prominent as a data integration delivery style, with 30% of su...

Key Takeaways  Alation, Collibra, Alex Solutions, And IBM Lead The Pack Forrester’s research uncovered a market in which Alation, Collibra, Alex Solutions, and IBM are Leaders; data.world, Informatica, Io-Tahoe, and Hitachi Vantara are Strong Performers; and Infogix and erwin are Contenders.  Collaboration, Lineage, And Data Variety Are Key Differentiators As metadata and business glossary technology becomes outdated and less effective, improved machine learning will dictate which providers lead the pack. Vendors that can provide scaleout collaboration, offer detailed data lineage, and interpret any type of data will position themselves to successfully deliver contextualized, trusted, accessible data to their customers. Read full report >>>

While traditional data warehouses often took years to build, deploy, and reap benefits from, today's organizations want simple, agile, integrated, cost-effective, and highly automated solutions to support insights. In addition, traditional architectures are failing to meet new business requirements, especially around high-speed data streaming, real-time analytics, large volumes of messy and complex data sets, and self-service. As a result, firms are revisiting their data architectures, looking for ways to modernize to support new requirements. DMA is a modern architecture that minimizes the complexity of messy data and hides heterogeneity by embodying a trusted model and integrated policies and by adapting to changing business requirements. It leverages metadata, in-memory, and distributed data repositories, running on-premises or in the cloud, to deliver scalable and integrated analytics. Adoption of DMA will grow further as enterprise architects look at overcoming data challeng...

Key Takeaways Software AG, IBM, Microsoft, Google, And TIBCO Software Lead The Pack Forrester's research uncovered a market in which Software AG, IBM, Microsoft, Google, and TIBCO Software are Leaders; Cloudera, SAS, Amazon Web Services, and Impetus are Strong Performers; and EsperTech and Alibaba are Contenders. Analytics Prowess, Scalability, And Deployment Freedom Are Key Differentiators Depth and breadth of analytics types on streaming data are critical. But that is all for naught if streaming analytics vendors cannot also scale to handle potentially huge volumes of streaming data. Also, it's critical that streaming analytics can be deployed where it is most needed, such as on-premises, in the cloud, and/or at the edge. Read report >>>

Here’s a curated list of resources for data engineers, with sections for algorithms and data structures, SQL, databases, programming, tools, distributed systems, and more. Useful articles The AI Hierarchy of Needs The Rise of Data Engineer The Downfall of the Data Engineer A Beginner’s Guide to Data Engineering Part I Part II Part III Functional Data Engineering — a modern paradigm for batch data processing How to become a Data Engineer (in Russian) Talks Data Engineering Principles - Build frameworks not pipelines by Gatis Seja Functional Data Engineering - A Set of Best Practices by Maxime Beauchemin Advanced Data Engineering Patterns with Apache Airflow by Maxime Beauchemin Creating a Data Engineering Culture by Jesse Anderson Algorithms & Data Structures Algorithmic Toolbox in Russian Data Structures in Russian Data Structures & Algorithms Specialization on Coursera Algorithms Specialization from Stanford on Coursera SQL Com...

Key Takeaways MongoDB, Microsoft, Couchbase, AWS, Google, And Redis Labs Lead The Pack Forrester's research uncovered a market in which MongoDB, Microsoft, Couchbase, AWS, Google, and Redis Labs are Leaders; MarkLogic, DataStax, Aerospike, Oracle, Neo4j, and IBM are Strong Performers; and SAP, ArangoDB, and RavenDB are Contenders. Performance, Scalability, Multimodel, And Security Are Key Differentiators The Leaders we identified support a broader set of use cases, automation, good scalability and performance, and security offerings. The Strong Performers have turned up the heat on the incumbents. Contenders offer lower costs and are ramping up their core NoSQL functionality. THE RISE OF BIG DATA NOSQL PLATFORMS NoSQL is more than a decade old. It has gone from supporting simple schemaless apps to becoming a mission-critical data platform for large Fortune 1000 companies. It has already disrupted the database market, which was dominated for decades by relational datab...

Gartner defines a data management solution for analytics (DMSA) as a complete software system that supports and manages data in one or many file management systems, most commonly a database or multiple databases. These management systems include specific optimization strategies designed for supporting analytical processing — including, but not limited to, relational processing, nonrelational processing (such as graph processing), and machine learning or programming languages such as Python or R. Data is not necessarily stored in a relational structure, and can use multiple data models — relational, XML, JavaScript Object Notation (JSON), key-value, graph, geospatial and others. Our definition also states that: A DMSA is a system for storing, accessing, processing and delivering data intended for one or more of the four primary use cases Gartner identifies that support analytics (see Note 1). A DMSA is not a specific class or type of technology; it is a use case. A DMSA ma...

The Five Use Cases and 15 Critical Capabilities of an Analytics and BI Platform We define and assess product capabilities across the following five use cases: Agile, centralized BI provisioning: Supports an agile IT-enabled workflow, from data to centrally delivered and managed analytic content, using the platform’s self-contained data management capabilities. Decentralized analytics: Supports a workflow from data to self-service analytics, and includes analytics for individual business units and users. Governed data discovery: Supports a workflow from data to self-service analytics to system of record (SOR), IT-managed content with governance, reusability and promotability of user-generated content to certified data and analytics content. OEM or embedded analytics: Supports a workflow from data to embedded BI content in a process or application. Extranet deployment: Supports a workflow similar to agile, centralized BI provisioning for the external customer or, in the pu...

Evaluated Vendors And Inclusion Criteria Forrester included 14 vendors in the assessment: Alibaba, AWS, Exasol, Google, Hortonworks, Huawei, IBM, MarkLogic, Micro Focus, Microsoft, Oracle, Pivotal, Snowflake, and Teradata. Each of these vendors has ( see Figure 1 ): A comprehensive CDW offering. Key components of the CDW include the provisioning, storing, processing, transforming, and accessing of data. The CDW should provide features to secure data, enable elastic scale, provide high availability and disaster recovery options, support loading and unloading of data, and provide various data access tools. A standalone data warehouse service running in the public cloud. Vendors included in this evaluation provide a CDW service that organizations can implement or use independent of analytics, data science, and visualization tools. The service should not be technologically tied to or bundled with any particular application or solution. Data warehouse use cases. The CDW service shoul...