ITNPBD6 Machine Learning Assignment Spring 2022 Computing and Maths, University of Stirling StirCom is a telecoms company that has spent the past few years digging up the streets of UK cities to connect people to their network. They are now keen to expand into the 5G market so they have been running a marketing campaign to encourage their existing landline customers to take out mobile contracts. Unfortunately, the call centre costs to run this campaign are high, so they would like to target customers more effectively to avoid wasted calls. This is where you come in. StirCom have provided data describing over 50000 customers and their responses to previous marketing campaigns. The data tells us if each customer responded to the marketing by setting up a new contract “new_contract_this_campaign”. The question is simple: is there any way to predict which customers are more likely to respond positively to a marketing call Your assignment is to answer that question using machine learning techniques and produce a system that would be able to tell StirCom which customers it should target the marketing at. You can use Orange, Python, R, or any data mining package of your choice. The data for the assignment is in a file stircom.csv provided for you. A data dictionary is provided describing the columns in this file. Please note that this is an individual assignment, not group work. Requirement You should submit a report describing the modelling process you followed and your results. You should try to frame the problem in the form of CRISP-DM framework to better facilitate the discussion. Refer to the relevant CRISP-DM stages at each stage of your report. You do not need to submit code or data. The report is worth 100 marks in total and must cover the following (with weightings per section as shown): Business Understanding [10%] Describe the task you were given: is it clustering, classification or regression ; describe the data you received and the requirements of the finished system, including why machine learning is suitable for this task. Define any terminology that you will use in the report (for example, model, variable, task, etc.). Comment on any issues around ethics or trust that may be relevant to the framing of the problem. Data Summary [10%] List the variables that you found in the file provided by the company. For each one, say whether it should be treated as categorical or numeric; nominal, ordinal, continuous or discrete; and whether or not it is likely to be of use in building the solution. Explain your decisions: if you rule out any variables at this stage, you can justify your choice using summary statistics, or a histogram plot of its distribution. Data Preparation [10%] Describe what you did with the data prior to the modelling process. Show histograms of the data before and after any pre-processing that you carried out. (you do not need to give histograms of all variables, just the ones that need some cleaning) If you corrected any mis-typed or corrupted entries in the data, report what you changed, such as any rules you used, or examples of specific data points that were cleaned. Modelling [30%] You must use three different techniques and build models with each: these should include one tree-based model, one based on logistic regression, and one based on neural networks. Try to make each model perform as well as it can: if you varied the hyperparameters of a model, show which hyperparameters you varied and how this impacted on the results. Describe how you split the data for training, validation and testing purposes. Be methodical and record each result. This stage is a little like scientific research – you are carrying out experiments in your search for the best solution. Once you have a solution, show how you verified its robustness. For the three different techniques report on their comparative ability to make predictions for this problem, but only select a single model for the final test. Don’t try to find a perfect or extremely accurate model – one does not exist! We are interested in the procedure you followed and the justification you give for choosing particular model types/parameters/features. Evaluation [10%] Analyse and describe the level of accuracy the model achieves and the errors your model makes. Show a confusion matrix for your model. Are there any areas of the data where it performs worse than in others, and are there any types of error that the company would want to avoid more than others Show a lift curve or a ROC curve for the model’s predictive capability, and explain what this tells you. Comment on whether the modelling approach chosen and the results raise any trust or ethical issues. Overall Approach and Creativity [30%] You should adopt a structured approach to the whole process and clearly identify the five CRISP-DM stages excluding deployment in your report. Additional marks are available within this category for going beyond the basic approaches mentioned in the sections above and those covered within the course; for example, you might consider feature construction or adding some explanation to the predictions the model makes. Those showing (per the common marking scheme) originality and exceptional analytical, problem-solving and/or creative skills will be awarded the highest marks. Submission You do not need to submit the models you built, just the report. The deadline for this assignment is 4pm on Friday 8 April 2022. Please submit your report via Canvas at https://canvas.stir.ac.uk/courses/11062/assignments/81475 . You should save it as a doc or pdf file bearing your university username (3 letters + 5 digits, e.g., xyz00001.pdf). Just write what you need to provide the required information clearly and concisely. You can assume that the client has a good technical understanding of data mining and statistics, so do not avoid technical terms in your report, but where you use them please explain what they mean in plain language too. To maximise your mark, closely follow the instructions you’ve been given. You can post general questions about the assignment in the Teams “Assessment Questions” channel. Marking Scheme Submissions will be marked in alignment with the University Postgraduate Common Marking Scheme, which can be found here: https://bit.ly/3sztOIt Word Count Limit For reports that exceed the 3500 maximum word count by: Less than 5% (less than 175 extra words), no marks will be reduced. From 5% – 10% (176 to 350 extra words), the mark will be reduced by five percentage points (5 marks). More than 10% (351 or more extra words), the mark will be reduced by ten percentage points (10 marks) for every 350 extra words. Word count penalties will not reduce the final mark below the passing threshold (50 marks) so if your mark was 52 and you have a 5% (5 marks) penalty, your final mark would still be 50. Note on Avoiding Academic Misconduct: Work which is submitted for assessment must be your own work. All students should note that the University has a formal policy on Academic Integrity and Academic Misconduct (including plagiarism) which can be found at https://bit.ly/37fYxPw Plagiarism: We know that assignment solutions by previous students can sometimes be found posted on GitHub or other public repositories. Do not be tempted to include any such code or writing in your submission. Using work that is not your own will be treated as “poor academic practice” or “plagiarism” and will be penalized. To avoid the risk of your own work being plagiarised by others, do not share copies of your solution, and keep your work secure both during and after the assignment period. Collusion: This is an individual assignment: working together with other students is not permitted. If students submit the same, or very similar work, this will be treated as “collusion” and all students involved will be penalized. Contract cheating: Asking or paying someone else to do assignment work for you (contract cheating) is considered gross academic misconduct, and will result in termination of your studies with no award. Late submissions: This assignment is subject to the usual grade penalties for a late submission. You can email questions about these to alexander.brownlee@stir.ac.uk.
