Notes about Cutting-Edge Technologies and Everything

  Database management systems continue their move to the cloud — a move that is producing an increasingly complex landscape of vendors and offerings. This Magic Quadrant will help data and analytics leaders make the right choices in a complex and fast-evolving market. Strategic Planning Assumptions By 2025, cloud preference for data management will substantially reduce the vendor landscape while the growth in multicloud will increase the complexity for data governance and integration. By 2022, cloud database management system (DBMS) revenue will account for 50% of the total DBMS market revenue. These DBMSs reflect optimization strategies designed to support transactions and/or analytical processing for one or more of the following use cases:     Traditional and augmented transaction processing     Traditional and logical data warehouse     Data science exploration/deep learning     Stream/event processing   ...

  The following post focuses on AWS, MS Azure, and GCP in detail. Learn more about each cloud service and how to choose the best one for your business needs.  Digitalization is being embraced by all of us across the globe, especially cloud computing technology. Whether it's because of its scalability or security or reduced costs, cloud platforms have sprung up to a great extent over a few years. Gone are the days when businesses were confused about whether to choose a cloud service provider or not. Now the confusion surrounds the question of which cloud service provider to use. AWS, Azure, and Google Cloud are our top three contenders. Recently, I happen to stumble upon an informative post focusing on AWS Lambda vs Azure Functions. I must say this one was quite detailed and well-structured. Here they have successfully covered all the aspects that are essential and dominating while we compare lambda vs azure. And I am pretty sure considering both the posts together will act a...

  Data helps companies take the guesswork out of decision-making. Teams can use data-driven evidence to decide which products to build, which features to add, and which growth initiatives to pursue. And, such insights-driven businesses grow at an annual rate of over 30%. But, there’s a difference between being merely data-aware and insights-driven. Discovering insights requires finding a way to analyze data in near real-time, which is where cloud data warehouses play a vital role. As scalable repositories of data, warehouses allow businesses to find insights by storing and analyzing huge amounts of structured and semi-structured data. And, running a data warehouse is more than a technical initiative. It’s vital to the overall business strategy and can inform an array of future product, marketing, and engineering decisions. But, choosing a cloud data warehouse provider can be challenging. Users have to evaluate costs, performance, the ability to handle real-time workloads, and other...

What Is Mainframe Modernization? Mainframe Modernization entails the process of migrating or improving the IT operations to reduce IT spending efficiently. In the realm of improving, we can define Mainframe Modernization as the process of enhancing legacy infrastructure by incorporating modern interfaces, code modernization, and performance modernization. In terms of migration, it is the process of shifting the enterprise’s code and functionality to a newer platform technology like cloud systems. The strategy employed to modernize Mainframe structures relies on factors like business/customer objectives, IT budgets, and costs of running new technology vs. costs incurred from not modernizing. B enefits of Mainframe Modernization to Cloud Cloud can offer economies of scale and new functions that are not available through mainframe computing. The benefits of cloud technologies and the law of diminishing returns in the Mainframe are calling for an increased demand for migration strate...

Databricks SQL   already provides a first-class user experience for BI and SQL directly on the data lake, and today, we are excited to announce another step in making data and AI simple with Databricks Serverless SQL. This new capability for Databricks SQL provides instant compute to users for their BI and SQL workloads, with minimal management required and capacity optimizations that can lower overall cost by an average of 40%. This makes it even easier for organizations to expand adoption of the lakehouse for business analysts who are looking to access the rich, real-time datasets of the lakehouse with a simple and performant solution. Under the hood of this capability is an active server fleet, fully managed by Databricks, that can transfer compute capacity to user queries, typically in about 15 seconds. The best part? You only pay for Serverless SQL when users start running reports or queries. Organizations with business analysts who want to analyze data in the data lake with t...

  One of the challenges with cloud migration is that you’re solving a puzzle with multiple pieces. In addition to a number of workloads you could migrate, you’re also solving for challenges you’re facing, the use cases driving you to migrate, and the benefits you’re looking to gain. Each organization’s puzzle will likely get solved in their own unique way, but thankfully there is plenty of guidance on how you can migrate common workloads in successful ways.  In addition to working directly with our Rapid Assessment and Migration Program (RAMP), we also offer a plethora of self-service guides to help you succeed! Some of these guides, which we’ll cover below, are designed to help you identify the best ways to migrate, which include meeting common organizational goals like minimizing time and risk during your migration, identifying the most enterprise-grade infrastructure for your workloads, picking a cloud that aligns with your organization’s sustainability goals...

Serverless computing is redefining the way organizations develop, deploy, and integrate cloud-native applications. According to an industry report, market size of serverless computing is expected to reach 7.72 billion by 2021. A new and compelling paradigm for the deployment of cloud applications, serverless computing is at the precipice of enterprise shift towards containers and microservices. In the year 2021, serverless paradigm shift presents exciting opportunities to organizations by providing a simplified programming model for creating cloud applications by abstracting away most operational concerns. Major cloud vendors, Microsoft, Google, and Amazon are already in the game with their respective offering and there is no reason you shouldn’t aboard the train. 2021 is the year of FaaS All major providers of serverless computing offer several types and tiers of database and storage services to their customers. In addition, all major cloud player such as Amazon, Microsoft and Google ...

The tech landscape changes pretty fast. There are always new terms, techniques and tools emerging. But don't let tech be an enigma: ThoughtWorks Decoder is here to help Simply search for the term you're interested in, and we'll give you the lowdown on what it is, what it can do for your enterprise and what the potential drawbacks are. ThoughtWorks Decoder >>>

 O ver the past few years, companies have been massively shifting their data and applications to the cloud that ended up raising a community of data users. They are encouraged to capture, gather, analyze, and save data for business insights and decision-making. More organizations are leading towards the use of multi-cloud, and the threat of losing data and securing has become challenging. Therefore, managing security policies, rules, metadata details, content traits is becoming critical for the multi-cloud. In this regard, the enterprises are in search of expertise and cloud tool vendors that are capable of providing the fundamental cloud security data governance competencies with excellence. Start with building policies and write them into code, or scripts that can be executed. This requires compliance and cloud security experts working together to build a framework for your complex business. You cannot start from scratch as it will be error-prone and will take too long. Try to in...

Once an outsider category, cloud computing now powers every industry. Look no further than this year’s Forbes Cloud 100 list, the annual ranking of the world’s top private cloud companies, where this year's standouts are keeping businesses surviving—and thriving—from real estate to retail, data to design. Produced for the fifth consecutive year in partnership with Bessemer Venture Partners and Salesforce Ventures, the Cloud 100 recognizes standouts in tech’s hottest category from small startups to private-equity-backed giants, from Silicon Valley to Australia and Hong Kong. The companies on the list are selected for their growth, sales, valuation and culture, as well as a reputation score derived in consultation with 43 CEO judges and executives from their public-cloud-company peers. This year’s new No. 1 has set a record for shortest time running atop the list. Database leader Snowflake takes the top slot, up from No. 2 last year and just hours before graduating from the list by g...

With the v2 Compute-to-Data release, Ocean Protocol provides a means to exchange data while preserving privacy. This guide explains Compute-to-Data without requiring deep technical know-how. Private data is data that people or organizations keep to themselves. It can mean any personal, personally identifiable, medical, lifestyle, financial, sensitive or regulated information. Benefits of Private Data. Private data can help research, leading to life-altering innovations in science and technology. For example, more data improves the predictive accuracy of modern Artificial Intelligence (AI) models. Private data is often considered the most valuable data because it’s so hard to get at, and using it can lead to potentially big payoffs. Risks of Private Data. Sharing or selling private data comes with risk. What if you don’t get hired because of your private medical history? What if you are persecuted for private lifestyle choices? Large organizations that have massive datasets know their d...

Microsoft Azure and Google Cloud have their advantages, but they don’t match the breadth and depth of the Amazon cloud. The reason is simple: AWS has built out so many products and services that it’s impossible to begin to discuss them in a single article or even a book. Many of them were amazing innovations when they first appeared and the hits keep coming. Every year Amazon adds new tools that make it harder and harder to justify keeping those old boxes pumping out heat and overstressing the air conditioner in the server room down the hall. For all of its dominance, though, Amazon has strong competitors. Companies like Microsoft, Google, IBM, Oracle, SAP, Rackspace, Linnode, and Digital Ocean know that they must establish a real presence in the cloud and they are finding clever ways to compete and excel in what is less and less a commodity business. These rivals offer great products with different and sometimes better approaches. In many cases, they’re running neck and neck wi...

In recent years, building a large-scale distributed storage system has become a hot topic.  Distributed consensus algorithms like Paxos and Raft are the focus of many technical articles. But those articles tend to be introductory, describing the basics of the algorithm and log replication. They seldom cover how to build a large-scale distributed storage system based on the distributed consensus algorithm.  Since April 2015, we PingCAP have been building TiKV, a large-scale open source distributed database based on Raft. It’s the core storage component of TiDB, an open source distributed NewSQL database that supports Hybrid Transactional and Analytical Processing (HTAP) workloads. Earlier in 2019, we conducted an official Jepsen test on TiDB, and the Jepsen test report was published in June 2019. In July the same year, we announced that TiDB 3.0 reached general availability, delivering stability at scale and performance boost. In this article, I’d like to share some of our firs...

While traditional data warehouses often took years to build, deploy, and reap benefits from, today's organizations want simple, agile, integrated, cost-effective, and highly automated solutions to support insights. In addition, traditional architectures are failing to meet new business requirements, especially around high-speed data streaming, real-time analytics, large volumes of messy and complex data sets, and self-service. As a result, firms are revisiting their data architectures, looking for ways to modernize to support new requirements. DMA is a modern architecture that minimizes the complexity of messy data and hides heterogeneity by embodying a trusted model and integrated policies and by adapting to changing business requirements. It leverages metadata, in-memory, and distributed data repositories, running on-premises or in the cloud, to deliver scalable and integrated analytics. Adoption of DMA will grow further as enterprise architects look at overcoming data challeng...

On November fourth, we announced Azure Synapse Analytics, the next evolution of Azure SQL Data Warehouse. Azure Synapse is a limitless analytics service that brings together enterprise data warehousing and Big Data analytics. It gives you the freedom to query data on your terms, using either serverless on-demand or provisioned resources—at scale. Azure Synapse brings these two worlds together with a unified experience to ingest, prepare, manage, and serve data for immediate business intelligence and machine learning needs. With Azure Synapse, data professionals can query both relational and non-relational data using the familiar SQL language. This can be done using either serverless on-demand queries for data exploration and ad hoc analysis or provisioned resources for your most demanding data warehousing needs. A single service for any workload. In fact, it’s the first and only analytics system to have run all the TPC-H queries at petabyte-scale. For current SQL Data Warehouse custome...

With the challenge of the vast amount of data generated by devices at the edge of networks, new architecture needs a well-established data service model that accounts for privacy concerns. This paper presents an architecture of data transmission and a data portfolio with privacy for fog-to-cloud (DPPforF2C). We would like to propose a practical data model with privacy from a digitalized information perspective at fog nodes. In addition, we also propose an architecture for implicating the privacy of DPPforF2C used in fog computing. Technically, we design a data portfolio based on the Message Queuing Telemetry Transport (MQTT) and the Advanced Message Queuing Protocol (AMQP). We aim to propose sample data models with privacy architecture because there are some differences in the data obtained from 10T devices and sensors. Thus, we propose an architecture with the privacy of DPPforF2C for publishing data from edge devices to fog and to cloud servers that could be applied to fog architectu...