Designing Frameworks for Reliability in Deep Learning Systems
Downloads
There has been a great amount of progress in deep learning models in the last decade. Such models are most accurate when applied to test data drawn from the same distribution as their training set. However, in practice, the data confronting models in real-world settings rarely match the training distribution.
This study explores the use of co-design approaches for developing reliable design frameworks for deep learning systems. It aims to raise awareness on how to develop reliable ML models within the context of recommender systems. While much work needs to be done in this field, the study provides suggestions and practical tips for how to develop reliable ML models such as in the case of recommender systems.
Assuncao, Marcos D., et al. “Big Data Computing and Clouds: Challenges, Solutions, and Future Directions.”Journal of Parallel and Distributed Computing, vol. 79–80, Dec. 2013,
https://doi.org/10.1016/j.jpdc.2014.08.003.
Athanases, Steven Z., et al. “Fostering Data Literacy through Preservice Teacher Inquiry in English Language Arts.” The Teacher Educator, vol. 48, no. 1, 2013, pp. 8–28.
Bargagliotti, Anna, et al. Pre-K-12 Guidelines for Assessment and Instruction in Statistics Education II (GAISE II): A Framework for Statistics and Data Science Education. American Statistical Association,2020,https://www.amstat.org/docs/default-source/amstat-documents/gaiseiiprek-12_full.pdf.
Bersin Insights Team. Insights from IMPACT 2018. Deloitte Development LLC, 2018,
Bhargava, Rahul, et al. “Beyond Data Literacy: Reinventing Community Engagement and Empowerment in the Age of Data.” Data-Pop Alliance, Data-Pop Alliance, Nov. 2015, https://datapopalliance.org/item/beyonddata-literacy-reinventing-community-engagement-and-empowerment-in-the-age-of-data.
Cowell, Matt. “A Roadmap for Creating a Data Literacy Program - QuantHub.” QuantHub, QuantHub, 18 June 2020,
https://quanthub.com/data-literacy-program. DataLiteracy. “The Data Literacy Score | Data Literacy.”
Data Literacy | Learn the Language of Data, Feb. 2021, https://dataliteracy.com/data-literacy-score. ---. “About This Project - DataLiteracy.Ca.” DataLiteracy.Ca, Intyernet Archive, 22 Dec. 2021, https://web.archive.org/web/20211222131031/http:/dataliteracy.ca/about-this-data-literacy-project.
Deahl, Erica. Better the Data You Know: Developing Youth Data Literacy in Schools and Informal Learning Environments. Massachusetts Institute of Technology, 2007,
Duncan, Alan D., Donna Medeiros, Aron Clarke, et al. “How to Measure the Value of Data Literacy.” Gartner, Gartner, Apr. 2022,
https://www.gartner.com/en/documents/4003941. ---. “Toolkit: Data Literacy Individual Assessment.” Gartner, Gartner, Apr. 2020,
https://www.gartner.com/en/documents/3983897.
Ebbeler, Johanna, et al. “Effects of a Data Use Intervention on Educators’ Use of Knowledge and Skills.” Studies in Educational Evaluation, vol. 48, Mar. 2016, pp. 19–31,
https://doi.org/10.1016/j.stueduc.2015.11.002.
Kleitz, Lauren, and Joe Shelley. EDUCAUSE Data Literacy Institute. EDUCAUSE, 24 Jan. 2022, https://events.educause.edu/educause-institute/data-literacy-institute/2022/online-1.
Kolbe, Richard H., and Melissa S. Burnett. “Content-Analysis Research: An Examination of Applications with Directives for Improving Research Reliability and Objectivity.” Journal of Consumer Research, vol. 18, no. 2, Sept. 1991, pp. 243–50, https://doi.org/10.1086/209256.
Koltay, Tibor. “Data Governance, Data Literacy and the Management of Data Quality.” IFLA Journal, vol. 42, no. 4, Nov. 2016, pp. 303–12, https://doi.org/10.1177/0340035216672238.
Kumar, Vivekanandan, and David Boulanger. “Explainable Automated Essay Scoring: Deep Learning Really Has Pedagogical Value.” Frontiers in Education, Oct. 2020,
https://doi.org/10.3389/feduc.2020.572367.
Linn, Robert L., and David M. Miller. Measurement and Assessment in Teaching, 9th Edition. Pearson, 2005,https://www.pearson.com/us/highereducation/product/Linn-Measurement-and-Assessment-in-Teaching9th-Edition/9780131137721.html.
Lorente, Clara Llebot. LibGuides: Research Data Services: Data Types & File Formats. Oregon State University, June 2021,
https://guides.library.oregonstate.edu/research-data-services/data-managementtypes-formats.
Means, Barbara, et al. Teachers’ Ability to Use Data to Inform Instruction: Challenges and Supports. US Department of Education, Office of Planning, Evaluation and Policy Development, Feb. 2011, https://eric.ed.gov/?id=ED516494.
National Defence. The Department of National Defence and Canadian Armed Forces Data Strategy - Canada.Ca. Government of Canada, 3 Dec. 2019, https://www.canada.ca/en/departmentnationaldefence/corporate/reports-publications/data-strategy.html.
Nikou, Melpomeni, and Panagiotis Tziachris. “Prediction and Uncertainty Capabilities of Quantile Regression Forests in Estimating Spatial Distribution of Soil Organic Matter.” ISPRS International Journal of GeoInformation, vol. 11, no. 2, Feb. 2022, p. 130,
https://doi.org/10.3390/ijgi1102013
Salleh, Kahirol Mohd, et al. “Competency of Adult Learners in Learning: Application of the Iceberg Competency Model.” Procedia - Social and Behavioral Sciences, vol. 204, Aug. 2015, pp. 326–34, https://doi.org/10.1016/j.sbspro.2015.08.160.
Samuel, A. L. “Some Studies in Machine Learning Using the Game of Checkers.” IBM Journal of Research and Development, vol. 3, no. 3, July 1959, pp. 210–29, https://doi.org/10.1147/rd.33.0210.
Schield, Milo. “Information Literacy, Statistical Literacy and Data Literacy.” IASSIST Quarterly, 2004, http://www.statlit.org/pdf/2004-Schield-IASSIST.pdf.
Walsh, John. Implementing DND/CAF Data Strategy. Government of Canada, 26 Sept. 2021, https://publicsectornetwork.co/wp-content/uploads/2021/09/John-Walsh-PDF.pdf.
Wells, Dave. Building a Data Literacy Program. 4 Jan. 2021, https://www.eckerson.com/articles/the-dataliteracy-imperative-part-i-building-a-data-literacy-program.
Williams, Paula G., et al. “On the Validity of Self-Report Assessment of Cognitive Abilities: Attentional Control Scale Associations With Cognitive Performance, Emotional Adjustment, and Personality.” Psychological Assessment, vol. 29, no. 5, 2017, pp. 519–30.
Wolff, Annika, et al. “Creating an Understanding of Data Literacy for a Data-Driven Society.” Journal of Community Informatics, vol. 12, no. 3, Aug. 2016, https://doi.org/10.15353/joci.v12i3.3275.
Yi Min Lim, Delia, et al. “Datastorming: Crafting Data into Design Materials for Design Students’ Creative Data Literacy.” C&C ’21: Creativity and Cognition, Association for Computing Machinery, 2021, pp. 1–9,
