Talks archive, 2010–2014 – Data Visualization using R

Overview Data & Data Visualization R Resources References Data Visualization using R Damian W. Betebenner National Center for the Improvement of Educational Assessment Dover, NH NCME Training Session Denver, Colorado April 29th, 2010 Damian W. Betebenner Data Visualization using R

Overview Data & Data Visualization About the R Software R Resources References Purpose of today’s session To learn about R and its capabilities for visualizing data. To provide participants with a comprehensive list of resources for producing their own data visualizations. To demonstrate the power of programatic drawing over WYSIWYG drawing. To inspire participants to develop their own customized visualizations and push the envelope for what is possible Damian W. Betebenner Data Visualization using R

Overview Data & Data Visualization About the R Software R Resources References About Me Dr. Damian Betebenner, PhD Senior Associate at the Center for Assessment (NCIEA). Developed student growth percentiles and percentile growth trajectories to help states and educational associations employ student growth in decision making [Betebenner, 2008, Betebenner, 2009]. In the process of refining and sharing these techniques with other states including Colorado, Massachusetts, Arizona, Indiana, and 15 other states in various stages of investigation/adoption. Interested in the rise of the data sciences/scientist: Data analysis and data visualization and their use within education. Began using R in 1998. Damian W. Betebenner Data Visualization using R

Overview Data & Data Visualization About the R Software R Resources References About the R Software R is an GNU open source, free, statistical software environment (package/language) that is available for source compilation or in pre-compiled binary form for numerous operating systems. R can be downloaded from CRAN (Comprehensive R Archive Network) http://cran.r-project.org/ The R language has become a de facto standard among statisticians for the development of statistical software. The philosophy behind R/S (John Chambers): “To Turn Ideas into Software Quickly and Faithfully” [Chambers, 2000, p. v] Damian W. Betebenner Data Visualization using R

Overview Data & Data Visualization About the R Software R Resources References About the R Software Daryl Pregibon, Google R is really important to the point that it’s hard to overvalue it. R had a recent New York Times article written about it http://www.nytimes.com/2009/01/07/technology/ business-computing/07program.html. A particular strength of R is its data visualization capabilities The greatest strength of R (in my humble opinion) is the IMMENSE amount of code available online to learn from. Learn from the masters. This training session will introduce users to data visualization using R as well as the resources available to continue their explorations. Damian W. Betebenner Data Visualization using R

Overview Data & Data Visualization Data R Resources Data Visualization References About Data Rutherford D. Roger We are drowning in information [data] and starving for knowledge [information]. The Economist, February 27th, 2010, devoted a special issue to “The Data Deluge”. In 2008, 1,200 exabytes (1.2 ZB) of digital data was created (n.b., giga, tera, peta, exa, zetta, yotta). Compound annual increase of data at 60%. Petabyte level computing is reaching commodity levels with Amazon EC2/S3 offering specials for petabyte scale projects. Damian W. Betebenner Data Visualization using R

Overview Data & Data Visualization Data R Resources Data Visualization References About Data Hal Varian, Chief Economist, Google A new kind of professional has emerged, the data scientist, who combines the skills of a software programmer, statistician, and story- teller/artist to extract nuggets of gold hidden under mountains of data. The job of the data scientist will become the sexiest around. Data are widely available; what is scarce is the ability to extract wisdom from them. The Data doesn’t speak for itself. The data can tell a thousand stories. Transforming data into information and ultimately into knowledge requires a broad range of expertise (i.e., the emerging data scientist). Goal: The right data to the right people at the right time in the right format! To reach this goal subject matter experts MUST work more closely with IT specialists, ensuring the right stories are told. Damian W. Betebenner Data Visualization using R

Overview Data & Data Visualization Data R Resources Data Visualization References About Data Visualization Data visualization leverages the immense capacity of the human eye to extract patterns from visual stimuli. Communicating with data is a form of storytelling. Visualization allows the storyteller to communicate more effectively and engage the user in more complicated stories. Color, pattern, and symbols can be combined to communicate highly complicated stories that, without pictures, are difficult even for experts to follow. As such, data visualization, as communication, is a form of teaching. Damian W. Betebenner Data Visualization using R

Overview Data & Data Visualization Data R Resources Data Visualization References About Programmatic Drawing Byron Ellis via Twitter [R] is for making new things. Point and click is for redoing old things. Almost all graphical programs (especially those incorporated in statistical analysis packages) rely upon WYSIWYG interface. WYSIWYG pros: Easy to learn. WYSIWYG cons: Limited ability to customize. Because of constrained options, WYSIWYG interfaces both dictate and limit the stories you can tell with your data. Programming your drawings (i.e., programatic drawing) allows for unlimited possibilities, constrained only by the creator’s imagination. Damian W. Betebenner Data Visualization using R

Observed Percent Proficient = 10 % Observed Percent Proficient = 20 % Observed Percent Proficient = 30 % Observed Percent Proficient = 40 % Observed Percent Proficient = 50 % Observed Percent Proficient = 60 % Observed Percent Proficient = 70 % Observed Percent Proficient = 80 % Observed Percent Proficient = 90 % Bias in Observed Percent Proficient (Observed − True) Observed under−estimates true Observed over−estimates true

2008 CSAP Mathematics Growth and Achievement Chart 2008 Achievement Percentile Percentile Growth Trajectory Scale Score Grade 3 Grade 4 Grade 5 Grade 6 Grade 7 Grade 8 Grade 9 Grade 10

Advanced Reading High Achievement CSAP Reading ● Typical Scale Score Proficient Growth Low ● Level Percentiles High 66th − 99th Typical 35th − 65th Part Proficient ● Low 1st − 34th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Next Year Scale Score 462 539 563 609 Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement Growth Percentile 66 66 90 Growth Level High High High Growth

Example Student 12 (1234567812) Sample Middle School Advanced Math Achievement High CSAP Math ● Scale Score Typical ● Growth Proficient ● Level Percentiles Low High 66th − 99th ● Typical 35th − 65th Part Proficient Low 1st − 34th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Next Year Scale Score 379 388 444 462 Achievement Level Part Proficient Part Proficient Part Proficient Part Proficient Achievement Growth Percentile 16 69 51 Growth Level Low High Typical Growth Advanced Reading High Achievement CSAP Reading ● Typical Scale Score Proficient Growth Low ● Level Percentiles High 66th − 99th Typical 35th − 65th Part Proficient ● Low 1st − 34th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Next Year Scale Score 462 539 563 609 Achievement Level Unsatisfactory Part Proficient Part Proficient Proficient Achievement Growth Percentile 66 66 90 Growth Level High High High Growth Advanced Writing Achievement High CSAP Writing ● Scale Score Typical ● Growth Proficient Level Percentiles ● Low High 66th − 99th Part Proficient ● Typical 35th − 65th Low 1st − 34th Unsatisfactory Grade 3 Grade 4 Grade 5 Grade 6 Next Year Scale Score 425 449 469 504 Achievement Level Part Proficient Part Proficient Part Proficient Part Proficient Achievement Growth Percentile 55 53 79 Growth Level Typical Typical High Growth Copyright 2009, Colorado Department of Education/Center for Assessment, Inc.

District C: 2008 CSAP Math School Results Student Growth versus Student Achievement by Percent Free/Reduced Lunch School Percent Higher Achievement Higher Achievement Free/Reduced Lunch Lower Growth Higher Growth Percent at/above Proficient in School Less than 20 percent ● 20 to 40 percent ● 40 to 60 percent ● 60 to 80 percent ● More than 80 percent School Size 50 Students 100 Students 200 Students 30 500 Students 1,000 Students 10 Lower Growth Higher Growth Lower Achievement Lower Achievement Median of Student Growth Percentiles in School

2009 Statewide Middle School Performance Growth and Achievement MCAS Math by School Poverty 100 Higher Achievement Higher Achievement School Size 90 Lower Growth Higher Growth 50 Students 100 Students X XX X XX 250 Students 80 X XXX X X X X X X X X XXX X XXXX X X 500 Students X X XXXX 2009 Percent at/above Proficient XX X X XX XX X X X XX X XXX X 70 X XX XX X X XX X X X X XX X X 1000 Students XX X X XXX XX X X XXXX XXX X X X XX XXX XXX X XX X X X XX X X X 60 X X XX X X X XXXXX X X X X X XX X X XX X X XXX X XXX X XX X X X X X X XX X XX X X School Percent X X X X X XXX X X X X XX XX XX XXXX X X Free/Reduced Lunch 50 XX X XXXX XXX XXXXX X XXX X X X X X X XX XXXXX X XX X XXX XXX X X XXX X XX XX ● Less than 20 percent 40 XXXXXX XXXX X XX X X XX XX X XX X XX XX X XXX X X X ● 20 to 40 percent XXX X X 30 XXX XX X X X ● 40 to 60 percent X XX X X X X X XXXX XX X XX ● 60 to 80 percent X X X XX ● More than 80 percent X XX X 10 X X XX X X Lower Growth X X X X X XX Higher Growth 0 Lower Achievement Lower Achievement X 2009 Median Student Growth Percentile

2009 Statewide Middle School Performance Growth and Achievement MCAS Math by School Poverty 100 Higher Achievement Higher Achievement School Size 90 Lower Growth Higher Growth 50 Students 100 Students 250 Students 500 Students 2009 Percent at/above Proficient 70 1000 Students School Percent Free/Reduced Lunch X XX ● Less than 20 percent X ● 20 to 40 percent XX X X ● 40 to 60 percent X XXX X ● 60 to 80 percent X XX X X X X X ● More than 80 percent XX X XX Lower Growth X X X X X XX Higher Growth 0 Lower Achievement Lower Achievement 2009 Median Student Growth Percentile

Overview Data & Data Visualization Online/Free R Resources Books References So Many Amazing Resources: Free Numerous help manuals installed when R is installed (Under the Help dropdown menu in Windows) One can download R and numerous resources at the CRAN (Comprehensive R Archive Network) http://cran.r-project.org/. Click on the “Contributed” hyperlink at CRAN to access dozens of free resources in many languages. Most resources provide many nuggets of wisdom that proves to be useful. Good places to start include: John Maindonald’s Using R for Data Analysis and Graphics–Introduction, Examples and Commentary John Verzani’s Simple R R has a very active listserve for R (R-help) that is easy to search using Google and keywords. Damian W. Betebenner Data Visualization using R

Overview Data & Data Visualization Online/Free R Resources Books References So Many Amazing Resources: Free The R Graph Gallery. Great examples with source code: http://addictedtor.free.fr/graphiques/. Spatial Data and R. Great examples with source code http://r-spatial.sourceforge.net/. A four part online video tutorial for using ggplot http://blog.revolution-computing.com/2010/03/ video-hadley-wickham-gives-a-short-course-on-graphics-with-r html Revolution Computing (a company that produces a version of R that cost $$$) has a blog that is REALLY cool: http://blog.revolution-computing.com Data Mashups using R. An amazing tutorial ($4.99 from O’Reilly). Damian W. Betebenner Data Visualization using R

Overview Data & Data Visualization Online/Free R Resources Books References So Many Amazing Resources: Published Books The Grid Graphics Package (and traditional graphics overview) [Murrell, 2006] Lattice Graphics Package: Depyan Sarkar [Sarkar, 2008]. ggplot Graphics Package: Hadley Wickham [Wickham, 2009]. Spatial Data Analysis and Visualization [Bivand et al., 2008]. Damian W. Betebenner Data Visualization using R

Overview Data & Data Visualization Online/Free R Resources Books References Inspiration “A Thing of Beauty is a Joy Forever” John Keats Damian W. Betebenner Data Visualization using R

Overview Data & Data Visualization R Resources References References Betebenner, D. W. (2008). Toward a normative understanding of student growth. In Ryan, K. E. and Shepard, L. A., editors, The Future of Test-Based Educational Accountability, pages 155–170. Taylor & Francis, New York. Betebenner, D. W. (2009). Norm- and criterion-referenced student growth. Educational Measurement: Issues and Practice, 28(4):42–51. Bivand, R. S., Pebesma, E. J., and Gòmez-Rubio, V. (2008). Applied Spatial Data Analysis with R. Springer, New York. Chambers, J. M. (2000). Programming with Data. Springer, New York. Murrell, P. (2006). R Graphics. Springer, New York. Sarkar, D. (2008). Lattice: Multivariate Data Visualization with R. Springer, New York. Wickham, H. (2009). ggplot2: Elegant Graphics for Data Analysis. Springer, New York. Damian W. Betebenner Data Visualization using R

About this copy

Data Visualization using R — NCME Annual Meeting, April 2010.

This is an image-based copy of the original beamer slides; builds and animations are flattened.