Notes about Cutting-Edge Technologies and Everything

Why Read This Report In our 30-criterion evaluation of value stream management (VSM) providers, we identified the 11 most significant ones — Atlassian, Blueprint, CloudBees, ConnectALL, Digital.ai, GitLab, IBM, Plutora, ServiceNow, Targetprocess, and Tasktop — and researched, analyzed, and scored them. This report shows how each provider measures up and helps application development and delivery (AD&D) professionals select the right one for their needs. Strong interest in VSM is driven primarily by three roles: 1) product owners and/or program managers who need data to help drive strategies, set priorities, and unlock team potential; 2) development leaders who use VSM to create connected, automated, and self-governed CI/CD pipelines with observability for improving and accelerating the pace of delivery; and 3) release engineers who use VSM for governance, compliance, and upstream observability to manage risk. (see endnote 3) To serve these roles effectively, customers should look f...

Have we entered the Golden Age of Data? Modern enterprises are collecting, producing, and processing more data than ever before. According to a February 2020 IDG survey of data professionals, average corporate data volumes are increasing by 63% per month. 10% of respondents even reported that their data volumes double every month. Large companies are investing heavily to transform themselves into data-driven organizations that can quickly adapt to the fast pace of a modern economy. They gather huge amounts of data from customers and generate reams of data from transactions. They continuously process data in an attempt to personalize customer experiences, optimize business processes, and drive strategic decisions. The Real Challenge with Data In theory, breakthrough open-source technologies, such as Spark, Kafka, and Druid are supposed to help just about any organization benefit from massive amounts of customer and operational data just like they benefit Facebook, Apple, Google, Microso...

Diagrams lets you draw the cloud system architecture in Python code. It was born for prototyping a new system architecture without any design tools. You can also describe or visualize the existing system architecture as well. Diagram as Code allows you to track the architecture diagram changes in any version control system. Diagrams currently supports six major providers: AWS, Azure, GCP, Kubernetes, Alibaba Cloud and Oracle Cloud. It now also supports On-Premise nodes. diagrams.mingrammer.com >>>

If you have been trying to harness the power of data science and machine learning — but, like many teams, struggling to produce results — there’s a secret you are missing out on. All of those models and sophisticated insights require lots of good data, and the best way to get good data quickly is by using DataOps. What is DataOps? It’s a way of thinking about how an organization deals with data. It’s a set of tools to automate processes and empower individuals. And it’s a new DataOps Engineer role designed to make that thinking real by managing and building those tools. DataOps Principles DataOps was inspired by DevOps, which brought the power of agile development to operations (infrastructure management and production deployment).  DevOps transformed the way that software development is done; and now DataOps is transforming the way that data management is done. For larger enterprises with a dedicated data engineering team, DataOps is about breaking down barriers and re-...

In this post, I am going to discuss Apache Airflow, a workflow management system developed by Airbnb. Earlier I had discussed writing basic ETL pipelines in Bonobo. Bonobo is cool for write ETL pipelines but the world is not all about writing ETL pipelines to automate things. There are other use cases in which you have to perform tasks in a certain order once or periodically. For instance: Monitoring Cron jobs transferring data from one place to other. Automating your DevOps operations. Periodically fetching data from websites and update the database for your awesome price comparison system. Data processing for recommendation based systems. Machine Learning Pipelines. Possibilities are endless. Before we move on further to implement Airflow in our systems, let’s discuss what actually is Airflow and it’s terminologies. What is Airflow? From the Website: Airflow is a platform to programmatically author, schedule and monitor workflows....