Notes about Cutting-Edge Technologies and Everything

While traditional data warehouses often took years to build, deploy, and reap benefits from, today's organizations want simple, agile, integrated, cost-effective, and highly automated solutions to support insights. In addition, traditional architectures are failing to meet new business requirements, especially around high-speed data streaming, real-time analytics, large volumes of messy and complex data sets, and self-service. As a result, firms are revisiting their data architectures, looking for ways to modernize to support new requirements. DMA is a modern architecture that minimizes the complexity of messy data and hides heterogeneity by embodying a trusted model and integrated policies and by adapting to changing business requirements. It leverages metadata, in-memory, and distributed data repositories, running on-premises or in the cloud, to deliver scalable and integrated analytics. Adoption of DMA will grow further as enterprise architects look at overcoming data challeng...

Datanami  is pleased to announce the results of its fourth annual Readers’ and Editors’ Choice Awards, which recognizes the companies, products, and projects that have made a difference in the big data community this year. These awards, which are nominated and voted on by Datanami readers, give us insight into the state of the community. We’d like to thank our dedicated readers for weighing in on their top picks for the best in big data. It’s been a privilege for us to present these awards, and we extend our congratulations to this year’s winners. Best Big Data Product or Technology: Machine Learning Readers’ Choice: Elastic Editor’s Choice: SAS Visual Data Mining & Machine Learning Best Big Data Product or Technology: Internet of Things Readers’ Choice: SAS Analytics for IoT Editor’s Choice:  The Striim Platform Best Big Data Product or Technology: Big Data Security Readers’ Choice: Cloudera Enterprise Editor’s Choice: Elastic Stack Best Big ...

Key Takeaways Software AG, IBM, Microsoft, Google, And TIBCO Software Lead The Pack Forrester's research uncovered a market in which Software AG, IBM, Microsoft, Google, and TIBCO Software are Leaders; Cloudera, SAS, Amazon Web Services, and Impetus are Strong Performers; and EsperTech and Alibaba are Contenders. Analytics Prowess, Scalability, And Deployment Freedom Are Key Differentiators Depth and breadth of analytics types on streaming data are critical. But that is all for naught if streaming analytics vendors cannot also scale to handle potentially huge volumes of streaming data. Also, it's critical that streaming analytics can be deployed where it is most needed, such as on-premises, in the cloud, and/or at the edge. Read report >>>

Cloud Hadoop/Spark (HARK) platforms accelerate insights by automating the storage, processing, and accessing of big data. In our 25-criterion evaluation of HARK providers, we identified the 11 most significant ones — Amazon Web Services (AWS), Cloudera, Google, Hortonworks, Huawei, MapR, Microsoft, Oracle, Qubole, Rackspace, and SAP — and researched, analyzed, and scored them.  This report shows how each provider measures up and helps enterprise architecture (EA) professionals select the right one for their needs. Note: Cloudera and Hortonworks completed their planned merger on January 3, 2019, and will continue as Cloudera. This Forrester Wave reflects our evaluation of each company's independent HARK platforms prior to the completion of the merger. Full report available here >>>

Key Takeaways Talend, Denodo Technologies, Oracle, IBM, And Paxata Lead The Pack Forrester's research uncovered a market in which Talend, Denodo Technologies, Oracle, IBM, and Paxata are Leaders; Hortonworks, Cambridge Semantics, SAP, Trifacta, Cloudera, and Syncsort are Strong Performers; and Podium Data, TIBCO Software, Informatica, and Hitachi Vantara are Contenders. EA Pros Are Looking To Support Multiple Use Cases With Big Data Fabric The big data fabric market is growing because more EA pros see big data fabric as critical for their enterprise big data strategy. Scale, Performance, AI/Machine Learning, And Use-Case Support Are Key Differentiators The Leaders we identified support a broader set of use cases, enhanced AI and machine learning capabilities, and offer good scalability features. ...

Details >> (shared by MemSQL here )

Users of the latest release of the Genome Analysis Toolkit, an open source framework for analyzing high-throughput DNA sequencing data, can now choose Apache Spark for data processing. Ever since the Human Genome Project produced the first draft sequence of the human genome in 2000, the cost of sequencing has dropped exponentially, from around US$100 million per genome then to around US$1,000 today. Over the same period, we have seen massive growth in the storage and processing capabilities of big data technologies like Apache Hadoop. It’s very fitting, then, to use tools from the Hadoop ecosystem for genomics, which is why Cloudera, in cooperation with the Broad Institute and other industry partners, is pleased to announce the alpha release of the Genome Analysis Toolkit (GATK) version 4 running on Apache Spark. Details: http://blog.cloudera.com/blog/2016/04/genome-analysis-toolkit-now-using-apache-spark-for-data-processing/

HDFS by default replicates each block three times. Replication provides a simple and robust form of redundancy to shield against most failure scenarios. It also eases scheduling compute tasks on locally stored data blocks by providing multiple replicas of each block to choose from. However, replication is expensive: the default 3x replication scheme incurs a 200% overhead in storage space and other resources (e.g., network bandwidth when writing the data). For datasets with relatively low I/O activity, the additional block replicas are rarely accessed during normal operations, but still consume the same amount of storage space. Therefore, a natural improvement is to use erasure coding (EC) in place of replication, which uses far less storage space while still providing the same level of fault tolerance. Under typical configurations, EC reduces the storage cost by ~50% compared with 3x replication. Motivated by this substantial cost saving opportunity, engineers from Cloudera and Intel ...

Apache HBase is all about giving you random, real-time, read/write access to your Big Data, but how do you efficiently get that data into HBase in the first place? Intuitively, a new user will try to do that via the client APIs or by using a MapReduce job with TableOutputFormat, but those approaches are problematic, as you will learn below. Instead, the HBase bulk loading feature is much easier to use and can insert the same amount of data more quickly. This blog post will introduce the basic concepts of the bulk loading feature, present two use cases, and propose two examples: https://blog.cloudera.com/blog/2013/09/how-to-use-hbase-bulk-loading-and-why/