{"id":4543,"date":"2019-12-02T16:52:08","date_gmt":"2019-12-02T14:52:08","guid":{"rendered":"https:\/\/engel-wolf.com\/?p=4543"},"modified":"2019-12-03T00:25:15","modified_gmt":"2019-12-02T22:25:15","slug":"does-using-dtplyr-make-sense-for-all-datasets","status":"publish","type":"post","link":"https:\/\/engel-wolf.com\/?p=4543","title":{"rendered":"Does using dtplyr make sense for all datasets? (UPDATED)"},"content":{"rendered":"\n

When Hadley Wickham published dtplyr 1.0.0<\/a> last week, I was super excited. Moreover, I wanted to know, if it really is a useful package. Therefore I performed some tiny research being presented in this blog entry.<\/h3>\n\n\n\n

This article was updated at December 2nd 2019 due to upcoming issues: https:\/\/github.com\/zappingseb\/dtplyrtest\/issues?q=is%3Aissue+is%3Aclosed<\/a>
Many thanks to:
Stefan Fritsch<\/a>, Dirk Raetzel<\/a>, Jan Gorecki<\/a> for qualified review of the article and the code.<\/p>\n\n\n\n

Ok. So dtplyr <\/em>gives you the speed of data.table<\/em> with the nice API of dplyr<\/em>. This is the main feature. You will see this in the following code. The speedup of dtplyr<\/em> should allow converting variables in large scale data sets by simple calls. So I first constructed an example call which is partly taken from Hadley Wickham`s blog entry<\/a>: <\/p>\n\n\n

\nn <- 100\n# Create a word vector\nwords <- vapply(1:15, FUN = function(x) {\n  paste(sample(LETTERS, 15, TRUE), collapse = "")\n}, FUN.VALUE = character(1))\n\n# Create an example dataset\ndf <- data.frame(\n  word = sample(words, n, TRUE),\n  letter = sample(LETTERS, n, TRUE),\n  id = 1:n,\n  numeric_int = sample(1:100, n, TRUE),\n  numeric_double = sample(seq(from = 0, to = 2, by = 1\/(10^6)), n, TRUE),\n  missing = sample(c(NA, 1:100), n, TRUE)\n)\n\n# Convert the numeric values\ndf %>%\n  filter(!is.na(missing)) %>% \n  group_by(letter) %>% \n  summarise(n = n(), delay = mean(numeric_int))\n<\/pre><\/div>\n\n\n
The call shown is the one used within dplyr.<\/em> In dtplyr<\/em> a lazy evaluation term needs to be added. At the end of the dplyr <\/em>call a %>% as_tibble()<\/code> is needed to start the evaluation. In data.table<\/em> the call looks totally different.<\/p>\n\n\n
\n# dtplyr call\nds_lazy <- lazy_dt(ds)\nds_lazy %>% fil.... %>%\n  as_tibble()\n\nDT <- as.data.table(ds)\n# data.table call\nDT_int[!is.na(missing), .(n = .N, delay = mean(numeric_int)), letter]\n<\/pre><\/div>\n\n\n
To evaluate the execution I created testing datasets of different sizes. The smallest dataset will contain 100 rows. The largest dataset 10E7 rows.  Afterward I ran the data conversion operations and logged the time of execution using the bench<\/em> package and 1000 iterations.  All calculations were performed on an AWS t3.2xlarge<\/a> machine hosting the <\/a>rocker\/tidyverse<\/a><\/em> docker image.  This is the result of the calculations plotted in a linear manner:<\/p>\n\n\n\n
\"\"
  The execution time of data transformations using dplyr, dtplyr and data.table plotted in a linear manner.  <\/figcaption><\/figure>\n\n\n\n
One can clearly see that dtplyr <\/em>(ds_lazy …) and data.table<\/em> (red line, calc_check_dt) increase nearly linear over time. Both seem to be really good solutions when it comes to large datasets. The difference of both at 10E7 can be explained by the low number of iterations being used at benchmarking. This post should not provide perfect benchmarks, but rather some fast insights into dtplyr. Thus, a reasonable computing time was preferred. <\/p>\n\n\n\n
Please consider, conversion and copying of a large data.frame<\/em> into a data.table or tibble may take some time. This shall be considered. When using data.table<\/em> all datasets should be data.tables right from the start.<\/p>\n\n\n\n
From the linear plot you cannot see at what point it is useful to use dtplyr against dplyr. This can thus be seen in the following plot:<\/p>\n\n\n\n
\"\"
 The execution time of dtplyr, dplyr and data.table plotted for the conversion of different datasets in a logarithmic manner.  <\/figcaption><\/figure>\n\n\n\n
The above image clearly shows that it is worth using dtplyr <\/em>over dplyr <\/em>in case there are more than 10E4 <\/strong>rows inside the dataset. So if you never work with datasets of this size it might not be worth considering dtplyr<\/em> at all. On the other hand, you can see the data.table<\/em> package itself is always the fastest solution. <\/p>\n\n\n\n
As in my project we do not just deal with numeric transformations of data, but also character transformations, I wanted to know if the dtplyr<\/em> package could be helpful for this. The conversion I performed for character strings looks like this:<\/p>\n\n\n
\n# dplyr code\ndf %>%\n  filter(letter == "A") %>% \n  mutate(word_new = paste0(word, numeric_int)) %>% \n  select(word_new)\n\n# data.table code\nDT[letter == "A", .(word_new = paste0(word, numeric_int))]\n<\/pre><\/div>\n\n\n
The calculation results are similar to those gained with numeric conversions<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Inside this plot the difference between dplyr <\/em>and data.table<\/em> is nearly invisible. All methods increase the computing time in linear relation to the data set size.<\/p>\n\n\n\n
\"\"
 The execution time of large scale data conversions for character evaluations with dtplyr, dplyr and data.table.  <\/figcaption><\/figure>\n\n\n\n
For character transformations the performance of dplyr <\/em>is pretty good. It is just useful to use dtplyr<\/em> for datasets with >10E5 <\/strong>rows. In case of the largest dataset tested in this approach with 10E7 rows, dtplyr<\/em> is twice as fast as dplyr.<\/em><\/p>\n\n\n\n
Conclusion<\/h3>\n\n\n\n
The code you need to write to use dtplyr<\/em> is really simple. Especially if you are like me dplyr<\/em> native and do not want to start using data.table<\/em>. But there will be a lot of use-cases where it is not necessary or not even useful to use the dtplyr<\/em> package. So before you start writing fancy dtplyr<\/em> pipelines, think about the necessity. <\/p>\n\n\n\n
All code for this project is available at:  https:\/\/github.com\/zappingseb\/dtplyrtest<\/a> – Please see updates from last commit<\/a><\/p>\n","protected":false},"excerpt":{"rendered":"
When Hadley Wickham published dtplyr 1.0.0 last week, I was super excited. Moreover, I wanted to know, if it really is a useful package. Therefore I performed some tiny research […]<\/p>\n","protected":false},"author":1,"featured_media":4566,"comment_status":"closed","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"ngg_post_thumbnail":0,"footnotes":""},"categories":[1],"tags":[424,441,440,438,439,391,384,381],"_links":{"self":[{"href":"https:\/\/engel-wolf.com\/index.php?rest_route=\/wp\/v2\/posts\/4543"}],"collection":[{"href":"https:\/\/engel-wolf.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/engel-wolf.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/engel-wolf.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/engel-wolf.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=4543"}],"version-history":[{"count":8,"href":"https:\/\/engel-wolf.com\/index.php?rest_route=\/wp\/v2\/posts\/4543\/revisions"}],"predecessor-version":[{"id":4569,"href":"https:\/\/engel-wolf.com\/index.php?rest_route=\/wp\/v2\/posts\/4543\/revisions\/4569"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/engel-wolf.com\/index.php?rest_route=\/wp\/v2\/media\/4566"}],"wp:attachment":[{"href":"https:\/\/engel-wolf.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=4543"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/engel-wolf.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=4543"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/engel-wolf.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=4543"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}