Notes about Cutting-Edge Technologies and Everything

Intro Recently I was put in charge of setting up a workflow for dealing with a large amount of raw DNA sequencing (well technically a SNP chip) data for my lab. The goal was to be able to quickly get data for a given genetic location (called a SNP) for use for modeling etc. Using vanilla R and AWK I was able to cleanup and organize the data in a natural way, massively speeding up the querying. It certainly wasn’t easy and it took lots of iterations. This post is meant to help others avoid some of the same mistakes and show what did eventually work. The Data The data was delivered to us by our university’s genetics processing center as 25 TB of tsvs. Before handing it off to me, my advisor split and gzipped these files into five batches each composed of roughly 240 four gigabyte files. Each row contained a data for a single SNP for a single person. There were ~2.5 million SNPS and ~60 thousand people Along with the SNP value there were multiple numeric columns on things like intensity o...

A curated list of awesome machine learning interpretability resources .  Comprehensive Software Examples and Tutorials Explainability- or Fairness-Enhancing Software Packages Browser Python R Free Books Other Interpretability and Fairness Resources and Lists Review and General Papers Limitations of Interpretability Teaching Resources Interpretable ("Whitebox") or Fair Modeling Packages C/C++ Python R

Glassdoor names “Data Scientist” as the best job in the United States for 2019 and LinkedIn ranks it number one among the top 10. Topping the list for four years in a row,  Data Scientist has a job score of 4.7, job satisfaction rating of 4.3 with 6,510 open positions paying a median base salary of $108,000 in the U.S. But what is the scenario for Data Scientists in Europe? What is the demand and supply? Which countries in EU are the best destinations for Data Scientists and what salaries can they expect? A recent report titled Data Science Salary Report 2019 Europe by Big Cloud  answers some of these critical questions.   First, a little flashback: According to a report by the European Commission in 2017, the number of data workers in Europe will increase up to 10.43 million, with a compound average growth rate of 14.1% by 2020. The EU forecasted to face a data skills gap corresponding to 769,000 unfilled positions by 2020 in the baseline scenario and being concen...

The best trained soldiers can’t fulfill their mission empty-handed. Data scientists have their own weapons  —  machine learning (ML) software. There is already a cornucopia of articles listing reliable machine learning tools with in-depth descriptions of their functionality. Our goal, however, was to get the feedback of industry experts. And that’s why we interviewed data science practitioners — gurus, really  — regarding the useful tools they choose for  their  projects. The specialists we contacted have various fields of expertise and are working in such companies as Facebook and Samsung. Some of them represent AI startups (Objection Co, NEAR.AI, and Respeecher); some teach at universities (Kharkiv National University of Radioelectronics). The AltexSoft data science team joined the discussion, too. And if you’re looking for a particular type of tools, just skip to your sector of interest: Languages used in machine learning Data analytics an...

Data Science Summarized in One Picture   R for Big Data in One Picture   A Cheat Sheet on Probability   Data Science in Python: Pandas Cheat Sheet   Cheat Sheet: Data Visualisation in Python   Machine Learning Cheat Sheet   The Periodic Table Of AI   Three Periodic Tables   40 maps that explain the Internet   A Guide to the Internet of Things   IoT Tectonics   13 Great Data Science Infographics   Unstructured Data: InfoGraphics   Great Machine Learning Infographics   What is Hadoop? Infog...