Amundsen — Lyft’s data discovery & metadata engine

The problem

Unprecedented growth in Data volumes has led to 2 big challenges:

• Productivity — Whether it’s building a new model, instrumenting a new metric, or doing adhoc analysis, how can I most productively and effectively make use of this data? 
• Compliance — When collecting data about a company’s users, how do organizations comply with increasing regulatory and compliance demands and uphold the trust of their users?
  

The key to solving these problems lies not in data, but in the metadata. And, to show you how, let’s go through a journey of how we solved a part of the productivity problem at Lyft using metadata.

Productivity

At a 50,000 feet level, the data scientist workflow looks like the following. 



Read full article >>>