Notes about Cutting-Edge Technologies and Everything

  Database management systems continue their move to the cloud — a move that is producing an increasingly complex landscape of vendors and offerings. This Magic Quadrant will help data and analytics leaders make the right choices in a complex and fast-evolving market. Strategic Planning Assumptions By 2025, cloud preference for data management will substantially reduce the vendor landscape while the growth in multicloud will increase the complexity for data governance and integration. By 2022, cloud database management system (DBMS) revenue will account for 50% of the total DBMS market revenue. These DBMSs reflect optimization strategies designed to support transactions and/or analytical processing for one or more of the following use cases:     Traditional and augmented transaction processing     Traditional and logical data warehouse     Data science exploration/deep learning     Stream/event processing   ...

  Data helps companies take the guesswork out of decision-making. Teams can use data-driven evidence to decide which products to build, which features to add, and which growth initiatives to pursue. And, such insights-driven businesses grow at an annual rate of over 30%. But, there’s a difference between being merely data-aware and insights-driven. Discovering insights requires finding a way to analyze data in near real-time, which is where cloud data warehouses play a vital role. As scalable repositories of data, warehouses allow businesses to find insights by storing and analyzing huge amounts of structured and semi-structured data. And, running a data warehouse is more than a technical initiative. It’s vital to the overall business strategy and can inform an array of future product, marketing, and engineering decisions. But, choosing a cloud data warehouse provider can be challenging. Users have to evaluate costs, performance, the ability to handle real-time workloads, and other...

  Snowflake data sharing and data marketplace can support modern data sharing techniques and eliminate the need for data movement. In Snowflake, there is no need to extract the data from the provider database and use some secure data transfer mechanism to share it with the consumers. Snowflake supports data sharing embedded into their SQL language so databases can be shared from within SQL commands. And on top of that, the data provider can update the data in real-time ensuring that all consumers will have a consistent, up-to-date view of their data sets. How Data Sharing Works Snowflake can share regular and external tables, and secure views and secure materialized views. Snowflake enables the sharing of databases through the concept of shares. Continue reading >>>

Once an outsider category, cloud computing now powers every industry. Look no further than this year’s Forbes Cloud 100 list, the annual ranking of the world’s top private cloud companies, where this year's standouts are keeping businesses surviving—and thriving—from real estate to retail, data to design. Produced for the fifth consecutive year in partnership with Bessemer Venture Partners and Salesforce Ventures, the Cloud 100 recognizes standouts in tech’s hottest category from small startups to private-equity-backed giants, from Silicon Valley to Australia and Hong Kong. The companies on the list are selected for their growth, sales, valuation and culture, as well as a reputation score derived in consultation with 43 CEO judges and executives from their public-cloud-company peers. This year’s new No. 1 has set a record for shortest time running atop the list. Database leader Snowflake takes the top slot, up from No. 2 last year and just hours before graduating from the list by g...

While traditional data warehouses often took years to build, deploy, and reap benefits from, today's organizations want simple, agile, integrated, cost-effective, and highly automated solutions to support insights. In addition, traditional architectures are failing to meet new business requirements, especially around high-speed data streaming, real-time analytics, large volumes of messy and complex data sets, and self-service. As a result, firms are revisiting their data architectures, looking for ways to modernize to support new requirements. DMA is a modern architecture that minimizes the complexity of messy data and hides heterogeneity by embodying a trusted model and integrated policies and by adapting to changing business requirements. It leverages metadata, in-memory, and distributed data repositories, running on-premises or in the cloud, to deliver scalable and integrated analytics. Adoption of DMA will grow further as enterprise architects look at overcoming data challeng...

Datanami  is pleased to announce the results of its fourth annual Readers’ and Editors’ Choice Awards, which recognizes the companies, products, and projects that have made a difference in the big data community this year. These awards, which are nominated and voted on by Datanami readers, give us insight into the state of the community. We’d like to thank our dedicated readers for weighing in on their top picks for the best in big data. It’s been a privilege for us to present these awards, and we extend our congratulations to this year’s winners. Best Big Data Product or Technology: Machine Learning Readers’ Choice: Elastic Editor’s Choice: SAS Visual Data Mining & Machine Learning Best Big Data Product or Technology: Internet of Things Readers’ Choice: SAS Analytics for IoT Editor’s Choice:  The Striim Platform Best Big Data Product or Technology: Big Data Security Readers’ Choice: Cloudera Enterprise Editor’s Choice: Elastic Stack Best Big ...

After many years of Big Data, NoSQL, and Schema-on-Read detours, there is a clear return to SQL as the lingua franca for data operations. Developers need the comprehensive expressiveness that SQL provides. A world without SQL ignores more than 40 years of database research and results in hard-coded spaghetti code in applications to handle functionality that SQL handles extremely efficiently such as joins, groupings, aggregations, and (most importantly) rollback when updates go wrong. Luckily, there is a modern architecture for SQL called Distributed SQL that no longer suffers from the challenges of traditional SQL systems (cost, scalability, performance, elasticity, and schema flexibility). The key attribute of Distributed SQL is that data is stored across many distributed storage locations and computation takes place across a cluster of networked servers. This yields unprecedented performance and scalability because it distributes work on each worker node in the cluster in parall...

In terms of performance tuning in Snowflake, there are very few options available. However, it is worth understanding how the Snowflake architecture includes various levels of caching to help speed your queries. This article provides an overview of the techniques used, and some best practice tips on how to maximise system performance using caching. Snowflake Database Architecture Before starting it’s worth considering the underlying Snowflake architecture, and explaining when Snowflake caches data. The diagram below illustrates the overall architecture which consists of three layers:- Service Layer:   Which accepts SQL requests from users, coordinates queries, managing transactions and results.  Logically, this can be assumed to hold the  result cache  – a cached copy of the results of every query executed. Compute Layer:   Which actually does the heavy lifting.  ...

Gartner defines a data management solution for analytics (DMSA) as a complete software system that supports and manages data in one or many file management systems, most commonly a database or multiple databases. These management systems include specific optimization strategies designed for supporting analytical processing — including, but not limited to, relational processing, nonrelational processing (such as graph processing), and machine learning or programming languages such as Python or R. Data is not necessarily stored in a relational structure, and can use multiple data models — relational, XML, JavaScript Object Notation (JSON), key-value, graph, geospatial and others. Our definition also states that: A DMSA is a system for storing, accessing, processing and delivering data intended for one or more of the four primary use cases Gartner identifies that support analytics (see Note 1). A DMSA is not a specific class or type of technology; it is a use case. A DMSA ma...

Evaluated Vendors And Inclusion Criteria Forrester included 14 vendors in the assessment: Alibaba, AWS, Exasol, Google, Hortonworks, Huawei, IBM, MarkLogic, Micro Focus, Microsoft, Oracle, Pivotal, Snowflake, and Teradata. Each of these vendors has ( see Figure 1 ): A comprehensive CDW offering. Key components of the CDW include the provisioning, storing, processing, transforming, and accessing of data. The CDW should provide features to secure data, enable elastic scale, provide high availability and disaster recovery options, support loading and unloading of data, and provide various data access tools. A standalone data warehouse service running in the public cloud. Vendors included in this evaluation provide a CDW service that organizations can implement or use independent of analytics, data science, and visualization tools. The service should not be technologically tied to or bundled with any particular application or solution. Data warehouse use cases. The CDW service shoul...