Keywords: kdb+, big data mining, R-KDB+ Interface, business/industry, high-performance computing
Webpages:
http://code.kx.com/wiki/Cookbook/IntegratingWithR Commercial application of ultra-low latency techniques for data mining and machine learning have been ubiquitous in financial trading and related disciplines for many years. As early as 2005, algorithmic trading desks at hedge funds and large investment banks have relied on in-memory, columnar databases and map-reduce techniques for analysing millions of data points in milliseconds long before such tools were used in other verticals. In particular, technologies such as
kdb+ and
Q - a vector-based programming platform developed as a successor to
APL (
A Programming Language developed in the 1950s/60s in Harvard by Ken Iverson), provide an unchallenged ability to perform both simple and complex data manipulations at scale with speeds that are orders of magnitude faster than contemporary platforms used for Big Data. A lesser-used, but formidable capability that has been used by
R-enthusiasts who were also
kdb+ experts has been the
R-KDB+ Interface used for interprocess-communication to share data between
R and
KDB+ processes all from within the user’s
R-console or
Q-console. In my nearly, 12 years of using
R, I, like many of my colleagues who have worked in financial trading environments have found such capabilities indispensable especially when working with large, oftentimes, TeraByte-scale datasets. The proposed talk features the basics of using the
R-KDB+ interface as a faster, superior and more optimal method to extract aggregated data from TB-scale data warehouses prior to statistical analysis in R.
