The goal of mde is to ease exploration of missingness.
Loading the package
library(mde)To get a simple missingness report, use na_summary:
na_summary(airquality)
#> variable missing complete percent_complete percent_missing
#> 1 Day 0 153 100.00000 0.000000
#> 2 Month 0 153 100.00000 0.000000
#> 3 Ozone 37 116 75.81699 24.183007
#> 4 Solar.R 7 146 95.42484 4.575163
#> 5 Temp 0 153 100.00000 0.000000
#> 6 Wind 0 153 100.00000 0.000000To sort this summary by a given column :
na_summary(airquality,sort_by = "percent_complete")
#> variable missing complete percent_complete percent_missing
#> 3 Ozone 37 116 75.81699 24.183007
#> 4 Solar.R 7 146 95.42484 4.575163
#> 1 Day 0 153 100.00000 0.000000
#> 2 Month 0 153 100.00000 0.000000
#> 5 Temp 0 153 100.00000 0.000000
#> 6 Wind 0 153 100.00000 0.000000If one would like to reset (drop) row names, then one can set row_names to TRUE This may especially be useful in cases where rownames are simply numeric and do not have much additional use.
na_summary(airquality,sort_by = "percent_complete", reset_rownames = TRUE)
#> variable missing complete percent_complete percent_missing
#> 1 Ozone 37 116 75.81699 24.183007
#> 2 Solar.R 7 146 95.42484 4.575163
#> 3 Day 0 153 100.00000 0.000000
#> 4 Month 0 153 100.00000 0.000000
#> 5 Temp 0 153 100.00000 0.000000
#> 6 Wind 0 153 100.00000 0.000000To sort by percent_missing instead:
na_summary(airquality, sort_by = "percent_missing")
#> variable missing complete percent_complete percent_missing
#> 1 Day 0 153 100.00000 0.000000
#> 2 Month 0 153 100.00000 0.000000
#> 5 Temp 0 153 100.00000 0.000000
#> 6 Wind 0 153 100.00000 0.000000
#> 4 Solar.R 7 146 95.42484 4.575163
#> 3 Ozone 37 116 75.81699 24.183007To sort the above in descending order:
na_summary(airquality, sort_by="percent_missing", descending = TRUE)
#> variable missing complete percent_complete percent_missing
#> 3 Ozone 37 116 75.81699 24.183007
#> 4 Solar.R 7 146 95.42484 4.575163
#> 1 Day 0 153 100.00000 0.000000
#> 2 Month 0 153 100.00000 0.000000
#> 5 Temp 0 153 100.00000 0.000000
#> 6 Wind 0 153 100.00000 0.000000To exclude certain columns from the analysis:
na_summary(airquality, exclude_cols = c("Day", "Wind"))
#> variable missing complete percent_complete percent_missing
#> 1 Month 0 153 100.00000 0.000000
#> 2 Ozone 37 116 75.81699 24.183007
#> 3 Solar.R 7 146 95.42484 4.575163
#> 4 Temp 0 153 100.00000 0.000000To include or exclude via regex match:
na_summary(airquality, regex_kind = "inclusion",pattern_type = "starts_with", pattern = "O|S")
#> variable missing complete percent_complete percent_missing
#> 1 Ozone 37 116 75.81699 24.183007
#> 2 Solar.R 7 146 95.42484 4.575163na_summary(airquality, regex_kind = "exclusion",pattern_type = "regex", pattern = "^[O|S]")
#> variable missing complete percent_complete percent_missing
#> 1 Day 0 153 100 0
#> 2 Month 0 153 100 0
#> 3 Temp 0 153 100 0
#> 4 Wind 0 153 100 0To get this summary by group:
test2 <- data.frame(ID= c("A","A","B","A","B"), Vals = c(rep(NA,4),"No"),ID2 = c("E","E","D","E","D"))
na_summary(test2,grouping_cols = c("ID","ID2"))
#> # A tibble: 2 x 7
#> ID ID2 variable missing complete percent_complete percent_missing
#> <chr> <chr> <chr> <dbl> <dbl> <dbl> <dbl>
#> 1 B D Vals 1 1 50 50
#> 2 A E Vals 3 0 0 100
na_summary(test2, grouping_cols="ID")
#> Warning in na_summary.data.frame(test2, grouping_cols = "ID"): All non grouping
#> values used. Using select non groups is currently not supported
#> # A tibble: 4 x 6
#> ID variable missing complete percent_complete percent_missing
#> <chr> <chr> <dbl> <dbl> <dbl> <dbl>
#> 1 A Vals 3 0 0 100
#> 2 A ID2 0 3 100 0
#> 3 B Vals 1 1 50 50
#> 4 B ID2 0 2 100 0get_na_countsThis provides a convenient way to show the number of missing values column-wise. It is relatively fast(tests done on about 400,000 rows, took a few microseconds.)
To get the number of missing values in each column of airquality, we can use the function as follows:
get_na_counts(airquality)
#> Ozone Solar.R Wind Temp Month Day
#> 1 37 7 0 0 0 0The above might be less useful if one would like to get the results by group. In that case, one can provide a grouping vector of names in grouping_cols.
test <- structure(list(Subject = structure(c(1L, 1L, 2L, 2L), .Label = c("A",
"B"), class = "factor"), res = c(NA, 1, 2, 3), ID = structure(c(1L,
1L, 2L, 2L), .Label = c("1", "2"), class = "factor")), class = "data.frame", row.names = c(NA,
-4L))
get_na_counts(test, grouping_cols = "ID")
#> # A tibble: 2 x 3
#> ID Subject res
#> <fct> <int> <int>
#> 1 1 0 1
#> 2 2 0 0percent_missingThis is a very simple to use but quick way to take a look at the percentage of data that is missing column-wise.
percent_missing(airquality)
#> Ozone Solar.R Wind Temp Month Day
#> 1 24.18301 4.575163 0 0 0 0We can get the results by group by providing an optional grouping_cols character vector.
percent_missing(test, grouping_cols = "Subject")
#> # A tibble: 2 x 3
#> Subject res ID
#> <fct> <dbl> <dbl>
#> 1 A 50 0
#> 2 B 0 0To exclude some columns from the above exploration, one can provide an optional character vector in exclude_cols.
percent_missing(airquality,exclude_cols = c("Day","Temp"))
#> Ozone Solar.R Wind Month
#> 1 24.18301 4.575163 0 0sort_by_missingnessThis provides a very simple but relatively fast way to sort variables by missingness. Unless otherwise stated, this does not currently support arranging grouped percents.
Usage:
sort_by_missingness(airquality, sort_by = "counts")
#> variable percent
#> 1 Wind 0
#> 2 Temp 0
#> 3 Month 0
#> 4 Day 0
#> 5 Solar.R 7
#> 6 Ozone 37To sort in descending order:
sort_by_missingness(airquality, sort_by = "counts", descend = TRUE)
#> variable percent
#> 1 Ozone 37
#> 2 Solar.R 7
#> 3 Wind 0
#> 4 Temp 0
#> 5 Month 0
#> 6 Day 0To use percentages instead:
sort_by_missingness(airquality, sort_by = "percents")
#> variable percent
#> 1 Wind 0.000000
#> 2 Temp 0.000000
#> 3 Month 0.000000
#> 4 Day 0.000000
#> 5 Solar.R 4.575163
#> 6 Ozone 24.183007Please note that the mde project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.
For further exploration, please browseVignettes("mde").
To raise an issue, please do so here
Thank you, feedback is always welcome :)