Loading…
This event has ended. View the official site or create your own event → Check it out
This event has ended. Create your own
View analytic
Thursday, July 6 • 2:42pm - 3:00pm
*implyr**: A **dplyr** Backend for a Apache Impala

Sign up or log in to save this to your schedule and see who's attending!

Keywords: Tidyverse, dplyr, SQL, Apache Impala, Big Data
Webpages: https://CRAN.R-project.org/package=implyr
This talk introduces implyr, a new dplyr backend for Apache Impala (incubating). I compare the features and performance of implyr to that of dplyr backends for other distributed query engines including sparklyr for Apache Spark’s Spark SQL, bigrquery for Google BigQuery, and RPresto for Presto.
Impala is a massively parallel processing query engine that enables low-latency SQL queries on data stored in the Hadoop Distributed File System (HDFS), Apache HBase, Apache Kudu, and Amazon Simple Storage Service (S3). The distributed architecture of Impala enables fast interactive queries on petabyte-scale data, but it imposes limitations on the dplyr interface. For example, row ordering of a result set must be performed in the final phase of query processing. I describe the methods used to work around this and other limitations.
Finally, I discuss broader issues regarding the DBI-compatible interfaces that dplyr requires for underlying connectivity to database sources. implyr is designed to work with any DBI-compatible interface to Impala, such as the general packages odbc and RJDBC, whereas other dplyr database backends typically rely on one particular package or mode of connectivity.

Speakers

Thursday July 6, 2017 2:42pm - 3:00pm
2.01 Wild Gallery

Attendees (97)