Notes about Cutting-Edge Technologies and Everything

The AGILE-IoT project ( www.agile-iot.eu ), co-funded by the Horizon 2020 programme of the European Union, aims to address this concern, by providing a solution based on four main pillars: Agnosticity : depending on the technical background of the platform user, the background of his organisation or the software components he has to (re-) use, we cannot predict what would be the programming language of the solution. It might be built with a combination of languages, some of them being compiled (e.g., C, C++), others translated into intermediate languages (e.g., Java, Python) and, finally, some others interpreted (e.g., JavaScript). If the user chooses one platform because of the programming language(s) it supports, he may limit his options for developing his solution. AGILE-IoT, by leveraging a micro-service-based architecture, supports all the programming languages a platform user might require to implement their solution. Openness : Lots of platforms are provided by ...

By 2020, the Internet of Things (IoT) is predicted to generate an additional $344B in revenues, as well as to drive $177B in cost reductions. IoT and smart devices are already increasing performance metrics of major US-based factories. They are in the hands of employees, covering routine management issues and boosting their productivity by 40-60% [1]. The following list of predictions (Figure 1) explores the state of IoT in 2019 and covering IoT impact on many aspects business and technology including Digital Transformation, Blockchain, AI, and 5G. Read full article >>>

In this post, I am going to discuss Apache Airflow, a workflow management system developed by Airbnb. Earlier I had discussed writing basic ETL pipelines in Bonobo. Bonobo is cool for write ETL pipelines but the world is not all about writing ETL pipelines to automate things. There are other use cases in which you have to perform tasks in a certain order once or periodically. For instance: Monitoring Cron jobs transferring data from one place to other. Automating your DevOps operations. Periodically fetching data from websites and update the database for your awesome price comparison system. Data processing for recommendation based systems. Machine Learning Pipelines. Possibilities are endless. Before we move on further to implement Airflow in our systems, let’s discuss what actually is Airflow and it’s terminologies. What is Airflow? From the Website: Airflow is a platform to programmatically author, schedule and monitor workflows....

I will be writing a series of blog posts about Apache Pulsar, including some Kafka vs Pulsar posts. First up though I will be running some chaos tests on a Pulsar cluster like I have done with RabbitMQ and Kafka to see what failure modes it has and its message loss scenarios. I will try to do this by either exploiting design defects, implementation bugs or poor configuration on the part of the admin or developer. In this post we’ll go through the Apache Pulsar design so that we can better design the failure scenarios. This post is not for people who want to understand how to use Apache Pulsar but who want to understand how it works. I have struggled to write a clear overview of its architecture in a way that is simple and easy to understand. I appreciate any feedback on this write-up. Claims The main claims that I am interested in are: guarantees of no message loss (if recommended configuration applied and your whole data center doesn't burn to the ground) strong...

Evaluated Vendors And Inclusion Criteria Forrester included 14 vendors in the assessment: Alibaba, AWS, Exasol, Google, Hortonworks, Huawei, IBM, MarkLogic, Micro Focus, Microsoft, Oracle, Pivotal, Snowflake, and Teradata. Each of these vendors has ( see Figure 1 ): A comprehensive CDW offering. Key components of the CDW include the provisioning, storing, processing, transforming, and accessing of data. The CDW should provide features to secure data, enable elastic scale, provide high availability and disaster recovery options, support loading and unloading of data, and provide various data access tools. A standalone data warehouse service running in the public cloud. Vendors included in this evaluation provide a CDW service that organizations can implement or use independent of analytics, data science, and visualization tools. The service should not be technologically tied to or bundled with any particular application or solution. Data warehouse use cases. The CDW service shoul...

Facebook Marketplace was introduced in 2016 as a place for people to buy and sell items within their local communities. Today in the U.S., more than one in three people on Facebook use Marketplace, buying and selling products in categories ranging from cars to shoes to dining tables. Managing the posting and selling of that volume of products with speed and relevance is a daunting task, and the fastest, most scalable way to handle that is to incorporate custom AI solutions. On Marketplace’s second anniversary, we are sharing how we use AI to power it. Whether someone is discovering an item to buy, listing a product to sell, or communicating with a buyer or seller, AI is behind the scenes making the experience better. In addition to the product index and content retrieval systems, which leverage our AI-based computer vision and natural language processing (NLP) platforms, we recently launched some new features that make the process simpler for both buyers and sellers. Multimodal...

Here’s what’s NEW: Up to 100x performance improvement for a wide range of query workloads, using Apache Arrow’s new kernel – Gandiva. Gandiva performs just-in-time compilation of SQL queries to machine code to get the fastest possible performance. (Our blog post explains more about how.) Support for Teradata, Azure Data Lake Store, AWS S3 GovCloud, and the latest version of Elasticsearch. Expect more soon! We’ve got a new connector framework that improves performance, stability, and development velocity for all data sources. Cluster Workload Manager, which lets you deploy diverse workloads on a single operational cluster while ensuring critical SLAs for performance and availability. More data catalog features, including wikis and tags for your data sets. That makes it even easier to discover, organize, curate, and share datasets from all your data sources.  Improved security and governance controls, like end-to-end encryption over TLS, and integration with Apache Ranger, ...

Kubernetes is really cool because managing services as flocks of little containers is a really cool way to make computing happen. We can get away from the idea that the computer will run the program and get into the idea that a service happens because a lot of little computing just happens. This idea is crucial to making reliable services that don’t require a ton of heroism to stand up or keep running. But there is a dark side here. Containers want to be agile because that is the point of containers in the first place. We want containers because we want to make computing more like a gas made up of indistinguishable atoms instead of like a few billiard balls with colors and numbers on their sides. Stopping or restarting containers should be cheap so we can push flocks of containers around easily and upgrade processes incrementally. If ever a container becomes heavy enough that we start thinking about that specific container, the whole metaphor kind of dissolves. So that metap...

Preview of Apache Hadoop 3.0 in Azure HDInsight 4.0 Led by Hortonworks, Apache Hadoop 3.0 represents over 5 years of work across the community since the last major update to the Hadoop stack. Enterprises can now realize their data lake vision while efficiently incorporating deep learning frameworks in to their applications all on the same Hadoop stack that they are comfortable with. Some of the key enhancements include: With ACID semantics enabled by default, Apache Hive 3.0 becomes more like a traditional database, making it easier for customers to build LOB applications on top of very large data sets. Apache Druid is an open source data store with indexing/caching capabilities on top of a column-oriented storage layout. With Apache Hive and Apache Druid (now available by default), customers can do near real time exploratory analytics on incoming data. With Tensorflow, available by default, and GPU support, Apache Hadoop 3.0 squarely targets the machine learning...

Learning about data governance Use these introductory books, videos, and articles to understand the basics of data governance. Data Governance: What You Need to Know  — Jon Bruner explains how a data governance program provides the intellectual and institutional grounding to address the data needs across an organization, anticipate new issues, and provide for development according to the company’s strategic plan. Data Governance  — John Adler leads you through the maze of data governance issues facing companies today—security breaches, regulatory agencies, in-house turf battles over who controls the data, monetizing data, and more. The Rise of Big Data Governance: Insight on this Emerging Trend from Active Open Source Initiatives  — John Mertic and Maryna Strelchuk detail the benefits of a vendor-neutral approach to data governance. Understanding the Chief Data Officer  — Through interviews with current and former chief data officers (CDO), Julie Steele lo...