Blog

Project 4, APS1070 Fall 2022 Linear Regression – 14 points Deadline: Dec 1st, 21:00 Academic Integrity This
project is individual – it is to be completed on your own. If you have
questions, please post your query in the APS1070 Piazza Q&A forums
(the answer might be useful to others!). Do not share your code
with others, or post your work online. Do not submit code that you have
not written yourself. Students suspected of plagiarism on a project, midterm or exam will be referred to the department for formal discipline for breaches of the Student Code of Conduct. Please fill out the following: Your name: Your student number: How to submit (HTML [This notebook] + IPYNB + GitHub [Part 7]) 1. Download your notebook: File -> Download .ipynb 2. Click on the Files icon on the far left menu of Colab 3.
Select & upload your .ipynb file you just downloaded, and then
obtain its path (right click) (you might need to hit the Refresh button
before your file shows up) 1. execute the following in a Colab cell: %%shell jupyter nbconvert –to html /PATH/TO/YOUR/NOTEBOOKFILE.ipynb 2. An HTML version of your notebook will appear in the files, so you can download it. 3. Submit both `HTML` and `IPYNB` files for this notebook on Quercus for grading. 4.
Make sure you compeleted the last part on **GitHub** and added your
link to the last cell here. The teaching team will view and grade your
part 7 on GitHub. Ref: https://stackoverflow.com/a/64487858 (https://stackoverflow.com/a/64487858) In [ ]: %%shell jupyter nbconvert –to html /PATH/TO/YOUR/NOTEBOOKFILE.ipynb Part 1 – Getting Started [1 marks] Ailerons
are small hinged sections on the outboard portion of a wing used to
control the roll of an airplane. In this project, we are going to design
a controller to manage the ailerons of an aircraft based on supervised learning. The
following dataset contains 13750 instances, where each instance is a
set of 40 features describing the airplane’s status. Our goal is to use
these features to predict the Goal column, which is a command that
our controller should issue. We will make our predictions by
implementing linear regression. In [ ]: import pandas as pd import numpy as np df=pd.read_csv(“https://raw.githubusercontent.com/aps1070-2019/datasets/master/F16L.csv” , skipinitialspace=True) In [ ]: from sklearn.model_selection import train_test_split from sklearn.preprocessing import StandardScaler from scipy import linalg from sklearn.metrics import mean_squared_error import matplotlib.pyplot as plt Here are the steps to complete this portion: 1. Print the dataframe. 2. Prepare your dataset as follows: [0.5] Using
train_test_split from Sklearn, split the dataset into training,
validation, and test sets ( training, validation, and test). When splitting, set random_state=1 . Standardize the data using StandardScaler from sklearn. Insert the first column of all s in the training, validation, and test set. 3. Explain the difference between epoch and iteration in the Gradient descent algorithm (SGD/mini-batch) [0.5] 70% 15% 15% 1 In [ ]: # 1: # Print the dataframe. df In [ ]: # 2: # Prepare your dataset as follows. # 70:15:15 = 7:3 X1 = df.drop(‘Goal’, axis = 1).values y1 = df[‘Goal’].values X1_train, X1_val_test, y_train, y1_val_test = train_test_split(X1, y1, test_size=0.3, random_state=1) X1_val, X1_test, y_val, y_test = train_test_split(X1_val_test, y1_val_test, test_size=0.5, random_state=1) X1_train.shape, X1_val.shape, X1_test.shape, y_train.shape, y_val.shape, y_test.shape Out[ ]: inputs ClimbRate
Sgz P Q CurPitch CurRoll AbsRoll DiffClb DiffRollRate DiffDiffClb …
DiffSeTime8 DiffSeTime9 DiffSeTime10 DiffSeTime11 DiffSeTime12 0 -17 -6 -0.03 0.14 0.59 -0.5 -11 -9 0.015 -0.2 … 0.0 0.000 0.0 0.000 1 -791 15 -0.28 -0.34 1.11 0.9 -10 27 -0.002 4.7 … 0.0 0.000 0.0 0.000 2 -46 48 -0.42 0.07 0.57 0.6 -5 -8 -0.008 0.0 … 0.0 0.000 0.0 0.000 3 496 -5 -0.21 -0.04 0.03 -0.5 -7 6 -0.002 -0.1 … 0.0 0.000 0.0 0.001 4 -124 16 -0.02 0.05 0.61 -0.2 -9 -3 0.000 0.0 … 0.0 0.000 0.0 0.000 … … … … … … … … … … … … … … … … 13745 -53 -13 -0.07 0.02 0.58 -2.3 -15 12 0.022 -0.8 … 0.0 0.000 0.0 0.000 13746 1 4 -0.11 -0.07 0.47 0.8 -5 8 -0.003 -0.2 … 0.0 0.000 0.0 0.000 13747 425 -14 0.40 0.07 0.41 -2.1 -16 12 0.031 -0.7 … 0.0 0.001 0.0 0.000 13748 102 -10 -0.19 0.08 1.30 0.5 -10 -3 -0.003 -0.2 … 0.0 0.000 0.0 0.000 13749 -165 52 0.08 0.42 1.07 -1.6 -22 -15 0.007 0.0 … 0.0 0.000 0.0 0.000 13750 rows × 41 columns Out[ ]: ((9625, 40), (2062, 40), (2063, 40), (9625,), (2062,), (2063,)) In [ ]: # Standardization scl = StandardScaler() Xtr_std = scl.fit_transform(X1_train) Xtr_std = pd.DataFrame(Xtr_std) Xtr_std.columns = df.iloc[:, 0:40].columns Xvl_std = scl.fit_transform(X1_val) Xvl_std = pd.DataFrame(Xvl_std) Xvl_std.columns = df.iloc[:, 0:40].columns Xts_std = scl.fit_transform(X1_test) Xts_std = pd.DataFrame(Xts_std) Xts_std.columns = df.iloc[:, 0:40].columns In [ ]: # Insert the first column of all 1s Xtr_std[‘All_1’] = 1 Xtr_std.insert(0, ‘All_1’, Xtr_std.pop(‘All_1’)) Xvl_std[‘All_1’] = 1 Xvl_std.insert(0, ‘All_1’, Xvl_std.pop(‘All_1’)) Xts_std[‘All_1’] = 1 Xts_std.insert(0, ‘All_1’, Xts_std.pop(‘All_1’)) Part 2 – Linear Regression Using Direct Solution [1 marks] Implement the direct solution of the linear regression problem on the training set. [0.5] Note: You should use scipy.linalg.inv to perform the matrix inversion, as numpy.linalg.inv may cause numerical issues. Report the root-mean-square error (RMSE) for both the training and validation sets. [0.5] You may use mean_squared_error from Sklearn for computing the RMSE. In [ ]: # 1: # Implement the direct solution of the linear regression problem on the training set. W = np.dot(linalg.inv(np.dot(Xtr_std.T, Xtr_std)), np.dot(Xtr_std.T, y_train)) In [ ]: y_train_pred = np.dot(Xtr_std, W) y_val_pred = np.dot(Xvl_std, W) In [ ]: # 1: # Report the root-mean-square error (RMSE) for both the training and validation sets. print(‘The RMSE of standardized training set is: ‘, np.sqrt(mean_squared_error(y_train_pred, y_train))) print(‘The RMSE of standardized validation set is: ‘, np.sqrt(mean_squared_error(y_val_pred, y_val))) Part 3 – Full Batch Gradient Descent [2 marks] The RMSE of standardized training set is: 0.000181615454715037 The RMSE of standardized validation set is: 0.00018403000289419626 We
will now implement a “full batch” gradient descent algorithm and record
the training time for our model. Recall that the full batch gradient
descent is, where is the learning rate and is your gradient, computed on the entire data. Here are the steps for this part: Implement
gradient descent for linear regression using a fixed learning rate of ,
and iterate until your model’s validation RMSE converges. We consider the gradient descent as having converged when RMSE on the validation set using gradient descent satisfies: where is the RMSE on the validation set using the direct solution that you have calculated in the previous part. We refer to the quantity as the convergence threshold (CT). Record
the training time (from the first iteration until convergence) using
the time.time() function. Be sure to compute the gradients yourself! Take a look at the code provided in the tutorial. [0.5] Plot the training RMSE and the validation RMSE vs. epoch on the same figure. [0.5] Comment on overfitting/underfitting by observing the training and validation RMSE [1] Hint: Initialize your weights with small random numbers (< ) = α wt wt 1 gt α gt α = 0.01 RMS ≤ 1.001 ×RMSEGD EDirect Solution RMSEDirect Solution RMS × 1.001EDirect Solution 0.001 In [ ]: (np.random.random(41)) In [ ]: import time start_time = time.time() ## Records current time CT = 1.001 * np.sqrt(mean_squared_error(y_val_pred, y_val)) lr = 0.01 w = np.random.random(41)/1500 rmse_train = [] rmse_val = [] for epoch in range (0, 1500): y_train_p = np.dot(Xtr_std, w.T) rmse_train.append(np.sqrt(mean_squared_error(y_train_p, y_train))) gradient = (1 / len(y_train_p)) * (np.dot(Xtr_std.T, y_train_p - y_train)) w = w - lr * gradient y_val_p = np.dot(Xvl_std, w.T) rmse_val_value = np.sqrt(mean_squared_error(y_val_p, y_val)) rmse_val.append(rmse_val_value) print('Epoch {} done with rmse val = {}, CT = {}'.format(epoch, rmse_val_value, CT)) if np.sqrt(mean_squared_error(y_val_p, y_val)) <= CT: print('Loop breaks at epoch = ', epoch) break print("--- Total Training Time: %s (s) ---" % (time.time() - start_time)) Out[ ]: array([0.99005339, 0.40187839, 0.74079032, 0.70957531, 0.94749885, 0.32595822, 0.40486802, 0.50385905, 0.08968792, 0.78185192, 0.65739725, 0.70435831, 0.76225397, 0.28109569, 0.56646975, 0.27255346, 0.68183414, 0.5858784 , 0.78941335, 0.64134047, 0.33142455, 0.77132565, 0.76388711, 0.41956906, 0.15063067, 0.29680645, 0.89510859, 0.66710086, 0.65975106, 0.08489766, 0.99586304, 0.34496044, 0.40701317, 0.50469564, 0.49941891, 0.10419883, 0.91130101, 0.34872421, 0.96049002, 0.58452437, 0.13296187]) Epoch 0 done with rmse val = 0.005044755128855716, CT = 0.00018421403289709044 Epoch 1 done with rmse val = 0.004355113825555625, CT = 0.00018421403289709044 Epoch 2 done with rmse val = 0.0037975512614875236, CT = 0.00018421403289709044 Epoch 3 done with rmse val = 0.0033502706362310546, CT = 0.00018421403289709044 Epoch 4 done with rmse val = 0.002994397518787471, CT = 0.00018421403289709044 Epoch 5 done with rmse val = 0.002713467625344955, CT = 0.00018421403289709044 Epoch 6 done with rmse val = 0.002493118494109315, CT = 0.00018421403289709044 Epoch 7 done with rmse val = 0.0023209390184699467, CT = 0.00018421403289709044 Epoch 8 done with rmse val = 0.002186399103093229, CT = 0.00018421403289709044 Epoch 9 done with rmse val = 0.0020807795394066054, CT = 0.00018421403289709044 Epoch 10 done with rmse val = 0.0019970511913146153, CT = 0.00018421403289709044 Epoch 11 done with rmse val = 0.0019296930029653082, CT = 0.00018421403289709044 Epoch 12 done with rmse val = 0.0018744681501961652, CT = 0.00018421403289709044 Epoch 13 done with rmse val = 0.00182818819638647, CT = 0.00018421403289709044 Epoch 14 done with rmse val = 0.0017884908733822693, CT = 0.00018421403289709044 Epoch 15 done with rmse val = 0.0017536468993614815, CT = 0.00018421403289709044 Epoch 16 done with rmse val = 0.001722401536032142, CT = 0.00018421403289709044 Epoch 17 done with rmse val = 0.0016938499987242772, CT = 0.00018421403289709044 Epoch 18 done with rmse val = 0.0016673424277287312, CT = 0.00018421403289709044 Epoch 19 done with rmse val = 0.001642412987692755, CT = 0.00018421403289709044 Epoch 20 done with rmse val = 0.0016187278061809988, CT = 0.00018421403289709044 Epoch 21 done with rmse val = 0.0015960471850331296, CT = 0.00018421403289709044 Epoch 22 done with rmse val = 0.001574198401198882, CT = 0.00018421403289709044 Epoch 23 done with rmse val = 0.0015530562495890024, CT = 0.00018421403289709044 Epoch 24 done with rmse val = 0.001532529187319462, CT = 0.00018421403289709044 Epoch 25 done with rmse val = 0.0015125495003230065, CT = 0.00018421403289709044 Epoch 26 done with rmse val = 0.0014930663427481722, CT = 0.00018421403289709044 Epoch 27 done with rmse val = 0.0014740408198975815, CT = 0.00018421403289709044 Epoch 28 done with rmse val = 0.0014554425204410914, CT = 0.00018421403289709044 Epoch 29 done with rmse val = 0.0014372470740473353, CT = 0.00018421403289709044 Epoch 30 done with rmse val = 0.0014194344331678172, CT = 0.00018421403289709044 Epoch 31 done with rmse val = 0.0014019876653987846, CT = 0.00018421403289709044 Epoch 32 done with rmse val = 0.0013848921053171805, CT = 0.00018421403289709044 Epoch 33 done with rmse val = 0.0013681347590587247, CT = 0.00018421403289709044 Epoch 34 done with rmse val = 0.0013517038863522306, CT = 0.00018421403289709044 Epoch 35 done with rmse val = 0.0013355887069718313, CT = 0.00018421403289709044 Epoch 36 done with rmse val = 0.0013197791942869352, CT = 0.00018421403289709044 Epoch 37 done with rmse val = 0.0013042659296819924, CT = 0.00018421403289709044 Epoch 38 done with rmse val = 0.0012890399994377967, CT = 0.00018421403289709044 Epoch 39 done with rmse val = 0.0012740929211731802, CT = 0.00018421403289709044 Epoch 40 done with rmse val = 0.0012594165908205925, CT = 0.00018421403289709044 Epoch 41 done with rmse val = 0.0012450032438321977, CT = 0.00018421403289709044 Epoch 42 done with rmse val = 0.0012308454262247288, CT = 0.00018421403289709044 Epoch 43 done with rmse val = 0.0012169359724114605, CT = 0.00018421403289709044 Epoch 44 done with rmse val = 0.0012032679877076577, CT = 0.00018421403289709044 Epoch 45 done with rmse val = 0.0011898348340511908, CT = 0.00018421403289709044 Epoch 46 done with rmse val = 0.0011766301179368332, CT = 0.00018421403289709044 Epoch 47 done with rmse val = 0.0011636476798803667, CT = 0.00018421403289709044 Epoch 48 done with rmse val = 0.0011508815849487024, CT = 0.00018421403289709044 Epoch 49 done with rmse val = 0.0011383261140441388, CT = 0.00018421403289709044 Epoch 50 done with rmse val = 0.0011259757557352144, CT = 0.00018421403289709044 Epoch 51 done with rmse val = 0.001113825198497815, CT = 0.00018421403289709044 Epoch 52 done with rmse val = 0.0011018693232784395, CT = 0.00018421403289709044 Epoch 53 done with rmse val = 0.0010901031963238595, CT = 0.00018421403289709044 Epoch 54 done with rmse val = 0.0010785220622428016, CT = 0.00018421403289709044 Epoch 55 done with rmse val = 0.0010671213372792075, CT = 0.00018421403289709044 Epoch 56 done with rmse val = 0.0010558966027854398, CT = 0.00018421403289709044 Epoch 57 done with rmse val = 0.0010448435988892123, CT = 0.00018421403289709044 Epoch 58 done with rmse val = 0.0010339582183511468, CT = 0.00018421403289709044 Epoch 59 done with rmse val = 0.0010232365006114985, CT = 0.00018421403289709044 Epoch 60 done with rmse val = 0.00101267462602528, CT = 0.00018421403289709044 Epoch 61 done with rmse val = 0.0010022689102851142, CT = 0.00018421403289709044 Epoch 62 done with rmse val = 0.0009920157990309041, CT = 0.00018421403289709044 Epoch 63 done with rmse val = 0.0009819118626449835, CT = 0.00018421403289709044 Epoch 64 done with rmse val = 0.0009719537912309016, CT = 0.00018421403289709044 Epoch 65 done with rmse val = 0.0009621383897734893, CT = 0.00018421403289709044 Epoch 66 done with rmse val = 0.0009524625734773443, CT = 0.00018421403289709044 Epoch 67 done with rmse val = 0.0009429233632804441, CT = 0.00018421403289709044 Epoch 68 done with rmse val = 0.0009335178815391992, CT = 0.00018421403289709044 Epoch 69 done with rmse val = 0.000924243347880943, CT = 0.00018421403289709044 Epoch 70 done with rmse val = 0.0009150970752195836, CT = 0.00018421403289709044 Epoch 71 done with rmse val = 0.0009060764659299486, CT = 0.00018421403289709044 Epoch 72 done with rmse val = 0.0008971790081761923, CT = 0.00018421403289709044 Epoch 73 done with rmse val = 0.0008884022723895394, CT = 0.00018421403289709044 Epoch 74 done with rmse val = 0.0008797439078905677, CT = 0.00018421403289709044 Epoch 75 done with rmse val = 0.0008712016396512129, CT = 0.00018421403289709044 Epoch 76 done with rmse val = 0.0008627732651916752, CT = 0.00018421403289709044 Epoch 77 done with rmse val = 0.0008544566516074422, CT = 0.00018421403289709044 Epoch 78 done with rmse val = 0.0008462497327216923, CT = 0.00018421403289709044 Epoch 79 done with rmse val = 0.0008381505063584098, CT = 0.00018421403289709044 Epoch 80 done with rmse val = 0.000830157031731632, CT = 0.00018421403289709044 Epoch 81 done with rmse val = 0.0008222674269463382, CT = 0.00018421403289709044 Epoch 82 done with rmse val = 0.000814479866606603, CT = 0.00018421403289709044 Epoch 83 done with rmse val = 0.0008067925795267419, CT = 0.00018421403289709044 Epoch 84 done with rmse val = 0.0007992038465412999, CT = 0.00018421403289709044 Epoch 85 done with rmse val = 0.0007917119984098547, CT = 0.00018421403289709044 Epoch 86 done with rmse val = 0.000784315413812729, CT = 0.00018421403289709044 Epoch 87 done with rmse val = 0.0007770125174338332, CT = 0.00018421403289709044 Epoch 88 done with rmse val = 0.0007698017781269873, CT = 0.00018421403289709044 Epoch 89 done with rmse val = 0.0007626817071621954, CT = 0.00018421403289709044 Epoch 90 done with rmse val = 0.0007556508565484713, CT = 0.00018421403289709044 Epoch 91 done with rmse val = 0.0007487078174299386, CT = 0.00018421403289709044 Epoch 92 done with rmse val = 0.00074185121855205, CT = 0.00018421403289709044 Epoch 93 done with rmse val = 0.0007350797247948895, CT = 0.00018421403289709044 Epoch 94 done with rmse val = 0.0007283920357706389, CT = 0.00018421403289709044 Epoch 95 done with rmse val = 0.0007217868844824049, CT = 0.00018421403289709044 Epoch 96 done with rmse val = 0.0007152630360417125, CT = 0.00018421403289709044 Epoch 97 done with rmse val = 0.0007088192864420806, CT = 0.00018421403289709044 Epoch 98 done with rmse val = 0.0007024544613862001, CT = 0.00018421403289709044 Epoch 99 done with rmse val = 0.0006961674151643371, CT = 0.00018421403289709044 Epoch 100 done with rmse val = 0.000689957029581681, CT = 0.00018421403289709044 Epoch 101 done with rmse val = 0.0006838222129324564, CT = 0.00018421403289709044 Epoch 102 done with rmse val = 0.000677761899018704, CT = 0.00018421403289709044 Epoch 103 done with rmse val = 0.000671775046211732, CT = 0.00018421403289709044 Epoch 104 done with rmse val = 0.0006658606365543175, CT = 0.00018421403289709044 Epoch 105 done with rmse val = 0.0006600176749018283, CT = 0.00018421403289709044 Epoch 106 done with rmse val = 0.0006542451881005062, CT = 0.00018421403289709044 Epoch 107 done with rmse val = 0.000648542224201239, CT = 0.00018421403289709044 Epoch 108 done with rmse val = 0.0006429078517072108, CT = 0.00018421403289709044 Epoch 109 done with rmse val = 0.000637341158853905, CT = 0.00018421403289709044 Epoch 110 done with rmse val = 0.0006318412529199869, CT = 0.00018421403289709044 Epoch 111 done with rmse val = 0.00062640725956767, CT = 0.00018421403289709044 Epoch 112 done with rmse val = 0.0006210383222112286, CT = 0.00018421403289709044 Epoch 113 done with rmse val = 0.0006157336014123751, CT = 0.00018421403289709044 Epoch 114 done with rmse val = 0.0006104922743012865, CT = 0.00018421403289709044 Epoch 115 done with rmse val = 0.0006053135340221113, CT = 0.00018421403289709044 Epoch 116 done with rmse val = 0.0006001965892018467, CT = 0.00018421403289709044 Epoch 117 done with rmse val = 0.0005951406634415233, CT = 0.00018421403289709044 Epoch 118 done with rmse val = 0.0005901449948286846, CT = 0.00018421403289709044 Epoch 119 done with rmse val = 0.0005852088354701939, CT = 0.00018421403289709044 Epoch 120 done with rmse val = 0.0005803314510444456, CT = 0.00018421403289709044 Epoch 121 done with rmse val = 0.0005755121203721023, CT = 0.00018421403289709044 Epoch 122 done with rmse val = 0.0005707501350045153, CT = 0.00018421403289709044 Epoch 123 done with rmse val = 0.0005660447988290308, CT = 0.00018421403289709044 Epoch 124 done with rmse val = 0.000561395427690415, CT = 0.00018421403289709044 Epoch 125 done with rmse val = 0.0005568013490276716, CT = 0.00018421403289709044 Epoch 126 done with rmse val = 0.0005522619015255559, CT = 0.00018421403289709044 Epoch 127 done with rmse val = 0.0005477764347801248, CT = 0.00018421403289709044 Epoch 128 done with rmse val = 0.0005433443089776896, CT = 0.00018421403289709044 Epoch 129 done with rmse val = 0.0005389648945865705, CT = 0.00018421403289709044 Epoch 130 done with rmse val = 0.0005346375720610785, CT = 0.00018421403289709044 Epoch 131 done with rmse val = 0.0005303617315571802, CT = 0.00018421403289709044 Epoch 132 done with rmse val = 0.0005261367726593201, CT = 0.00018421403289709044 Epoch 133 done with rmse val = 0.0005219621041179073, CT = 0.00018421403289709044 Epoch 134 done with rmse val = 0.0005178371435969919, CT = 0.00018421403289709044 Epoch 135 done with rmse val = 0.0005137613174316795, CT = 0.00018421403289709044 Epoch 136 done with rmse val = 0.0005097340603948549, CT = 0.00018421403289709044 Epoch 137 done with rmse val = 0.0005057548154728044, CT = 0.00018421403289709044 Epoch 138 done with rmse val = 0.0005018230336493479, CT = 0.00018421403289709044 Epoch 139 done with rmse val = 0.0004979381736981074, CT = 0.00018421403289709044 Epoch 140 done with rmse val = 0.0004940997019825586, CT = 0.00018421403289709044 Epoch 141 done with rmse val = 0.0004903070922635279, CT = 0.00018421403289709044 Epoch 142 done with rmse val = 0.0004865598255138132, CT = 0.00018421403289709044 Epoch 143 done with rmse val = 0.0004828573897396225, CT = 0.00018421403289709044 Epoch 144 done with rmse val = 0.00047919927980853786, CT = 0.00018421403289709044 Epoch 145 done with rmse val = 0.0004755849972837264, CT = 0.00018421403289709044 Epoch 146 done with rmse val = 0.00047201405026413545, CT = 0.00018421403289709044 Epoch 147 done with rmse val = 0.000468485953230417, CT = 0.00018421403289709044 Epoch 148 done with rmse val = 0.000465000226896343, CT = 0.00018421403289709044 Epoch 149 done with rmse val = 0.0004615563980654802, CT = 0.00018421403289709044 Epoch 150 done with rmse val = 0.0004581539994929094, CT = 0.00018421403289709044 Epoch 151 done with rmse val = 0.00045479256975177807, CT = 0.00018421403289709044 Epoch 152 done with rmse val = 0.00045147165310449047, CT = 0.00018421403289709044 Epoch 153 done with rmse val = 0.00044819079937834477, CT = 0.00018421403289709044 Epoch 154 done with rmse val = 0.00044494956384543815, CT = 0.00018421403289709044 Epoch 155 done with rmse val = 0.0004417475071066682, CT = 0.00018421403289709044 Epoch 156 done with rmse val = 0.00043858419497966734, CT = 0.00018421403289709044 Epoch 157 done with rmse val = 0.00043545919839051335, CT = 0.00018421403289709044 Epoch 158 done with rmse val = 0.00043237209326906845, CT = 0.00018421403289709044 Epoch 159 done with rmse val = 0.00042932246044780526, CT = 0.00018421403289709044 Epoch 160 done with rmse val = 0.0004263098855639836, CT = 0.00018421403289709044 Epoch 161 done with rmse val = 0.0004233339589650496, CT = 0.00018421403289709044 Epoch 162 done with rmse val = 0.00042039427561713506, CT = 0.00018421403289709044 Epoch 163 done with rmse val = 0.00041749043501653794, CT = 0.00018421403289709044 Epoch 164 done with rmse val = 0.00041462204110407334, CT = 0.00018421403289709044 Epoch 165 done with rmse val = 0.00041178870218218724, CT = 0.00018421403289709044 Epoch 166 done with rmse val = 0.00040899003083473177, CT = 0.00018421403289709044 Epoch 167 done with rmse val = 0.00040622564384930256, CT = 0.00018421403289709044 Epoch 168 done with rmse val = 0.0004034951621420482, CT = 0.00018421403289709044 Epoch 169 done with rmse val = 0.00040079821068485835, CT = 0.00018421403289709044 Epoch 170 done with rmse val = 0.0003981344184348499, CT = 0.00018421403289709044 Epoch 171 done with rmse val = 0.0003955034182660659, CT = 0.00018421403289709044 Epoch 172 done with rmse val = 0.0003929048469033117, CT = 0.00018421403289709044 Epoch 173 done with rmse val = 0.0003903383448580523, CT = 0.00018421403289709044 Epoch 174 done with rmse val = 0.0003878035563663006, CT = 0.00018421403289709044 Epoch 175 done with rmse val = 0.0003853001293284268, CT = 0.00018421403289709044 Epoch 176 done with rmse val = 0.00038282771525082403, CT = 0.00018421403289709044 Epoch 177 done with rmse val = 0.00038038596918936744, CT = 0.00018421403289709044 Epoch 178 done with rmse val = 0.0003779745496946061, CT = 0.00018421403289709044 Epoch 179 done with rmse val = 0.0003755931187586288, CT = 0.00018421403289709044 Epoch 180 done with rmse val = 0.00037324134176355014, CT = 0.00018421403289709044 Epoch 181 done with rmse val = 0.0003709188874315612, CT = 0.00018421403289709044 Epoch 182 done with rmse val = 0.0003686254277764942, CT = 0.00018421403289709044 Epoch 183 done with rmse val = 0.0003663606380568526, CT = 0.00018421403289709044 Epoch 184 done with rmse val = 0.0003641241967302564, CT = 0.00018421403289709044 Epoch 185 done with rmse val = 0.00036191578540925915, CT = 0.00018421403289709044 Epoch 186 done with rmse val = 0.0003597350888184911, CT = 0.00018421403289709044 Epoch 187 done with rmse val = 0.0003575817947530859, CT = 0.00018421403289709044 Epoch 188 done with rmse val = 0.00035545559403834913, CT = 0.00018421403289709044 Epoch 189 done with rmse val = 0.00035335618049062953, CT = 0.00018421403289709044 Epoch 190 done with rmse val = 0.0003512832508793543, CT = 0.00018421403289709044 Epoch 191 done with rmse val = 0.0003492365048901902, CT = 0.00018421403289709044 Epoch 192 done with rmse val = 0.00034721564508929443, CT = 0.00018421403289709044 Epoch 193 done with rmse val = 0.0003452203768886219, CT = 0.00018421403289709044 Epoch 194 done with rmse val = 0.00034325040851225286, CT = 0.00018421403289709044 Epoch 195 done with rmse val = 0.00034130545096370873, CT = 0.00018421403289709044 Epoch 196 done with rmse val = 0.00033938521799422466, CT = 0.00018421403289709044 Epoch 197 done with rmse val = 0.0003374894260719464, CT = 0.00018421403289709044 Epoch 198 done with rmse val = 0.00033561779435202325, CT = 0.00018421403289709044 Epoch 199 done with rmse val = 0.00033377004464756554, CT = 0.00018421403289709044 Epoch 200 done with rmse val = 0.0003319459014014391, CT = 0.00018421403289709044 Epoch 201 done with rmse val = 0.0003301450916588696, CT = 0.00018421403289709044 Epoch 202 done with rmse val = 0.00032836734504082814, CT = 0.00018421403289709044 Epoch 203 done with rmse val = 0.000326612393718172, CT = 0.00018421403289709044 Epoch 204 done with rmse val = 0.0003248799723865158, CT = 0.00018421403289709044 Epoch 205 done with rmse val = 0.00032316981824180693, CT = 0.00018421403289709044 Epoch 206 done with rmse val = 0.0003214816709565804, CT = 0.00018421403289709044 Epoch 207 done with rmse val = 0.0003198152726568702, CT = 0.00018421403289709044 Epoch 208 done with rmse val = 0.0003181703678997528, CT = 0.00018421403289709044 Epoch 209 done with rmse val = 0.0003165467036515004, CT = 0.00018421403289709044 Epoch 210 done with rmse val = 0.0003149440292663212, CT = 0.00018421403289709044 Epoch 211 done with rmse val = 0.00031336209646566534, CT = 0.00018421403289709044 Epoch 212 done with rmse val = 0.0003118006593180744, CT = 0.00018421403289709044 Epoch 213 done with rmse val = 0.0003102594742195544, CT = 0.00018421403289709044 Epoch 214 done with rmse val = 0.0003087382998744511, CT = 0.00018421403289709044 Epoch 215 done with rmse val = 0.0003072368972768079, CT = 0.00018421403289709044 Epoch 216 done with rmse val = 0.00030575502969218757, CT = 0.00018421403289709044 Epoch 217 done with rmse val = 0.0003042924626399368, CT = 0.00018421403289709044 Epoch 218 done with rmse val = 0.00030284896387587705, CT = 0.00018421403289709044 Epoch 219 done with rmse val = 0.0003014243033754022, CT = 0.00018421403289709044 Epoch 220 done with rmse val = 0.0003000182533169654, CT = 0.00018421403289709044 Epoch 221 done with rmse val = 0.00029863058806593783, CT = 0.00018421403289709044 Epoch 222 done with rmse val = 0.0002972610841588217, CT = 0.00018421403289709044 Epoch 223 done with rmse val = 0.0002959095202878023, CT = 0.00018421403289709044 Epoch 224 done with rmse val = 0.00029457567728562157, CT = 0.00018421403289709044 Epoch 225 done with rmse val = 0.00029325933811075725, CT = 0.00018421403289709044 Epoch 226 done with rmse val = 0.0002919602878328935, CT = 0.00018421403289709044 Epoch 227 done with rmse val = 0.0002906783136186667, CT = 0.00018421403289709044 Epoch 228 done with rmse val = 0.00028941320471767145, CT = 0.00018421403289709044 Epoch 229 done with rmse val = 0.00028816475244871476, CT = 0.00018421403289709044 Epoch 230 done with rmse val = 0.00028693275018630137, CT = 0.00018421403289709044 Epoch 231 done with rmse val = 0.0002857169933473386, CT = 0.00018421403289709044 Epoch 232 done with rmse val = 0.0002845172793780481, CT = 0.00018421403289709044 Epoch 233 done with rmse val = 0.0002833334077410694, CT = 0.00018421403289709044 Epoch 234 done with rmse val = 0.00028216517990274496, CT = 0.00018421403289709044 Epoch 235 done with rmse val = 0.00028101239932057446, CT = 0.00018421403289709044 Epoch 236 done with rmse val = 0.0002798748714308252, CT = 0.00018421403289709044 Epoch 237 done with rmse val = 0.00027875240363628873, CT = 0.00018421403289709044 Epoch 238 done with rmse val = 0.00027764480529417326, CT = 0.00018421403289709044 Epoch 239 done with rmse val = 0.00027655188770411864, CT = 0.00018421403289709044 Epoch 240 done with rmse val = 0.0002754734640963274, CT = 0.00018421403289709044 Epoch 241 done with rmse val = 0.0002744093496197991, CT = 0.00018421403289709044 Epoch 242 done with rmse val = 0.00027335936133065966, CT = 0.00018421403289709044 Epoch 243 done with rmse val = 0.0002723233181805774, CT = 0.00018421403289709044 Epoch 244 done with rmse val = 0.00027130104100525523, CT = 0.00018421403289709044 Epoch 245 done with rmse val = 0.0002702923525129923, CT = 0.00018421403289709044 Epoch 246 done with rmse val = 0.0002692970772733062, CT = 0.00018421403289709044 Epoch 247 done with rmse val = 0.0002683150417056081, CT = 0.00018421403289709044 Epoch 248 done with rmse val = 0.0002673460740679242, CT = 0.00018421403289709044 Epoch 249 done with rmse val = 0.000266390004445656, CT = 0.00018421403289709044 Epoch 250 done with rmse val = 0.00026544666474037187, CT = 0.00018421403289709044 Epoch 251 done with rmse val = 0.0002645158886586262, CT = 0.00018421403289709044 Epoch 252 done with rmse val = 0.00026359751170079693, CT = 0.00018421403289709044 Epoch 253 done with rmse val = 0.0002626913711499389, CT = 0.00018421403289709044 Epoch 254 done with rmse val = 0.0002617973060606451, CT = 0.00018421403289709044 Epoch 255 done with rmse val = 0.00026091515724791295, CT = 0.00018421403289709044 Epoch 256 done with rmse val = 0.0002600447672760097, CT = 0.00018421403289709044 Epoch 257 done with rmse val = 0.0002591859804473325, CT = 0.00018421403289709044 Epoch 258 done with rmse val = 0.0002583386427912607, CT = 0.00018421403289709044 Epoch 259 done with rmse val = 0.0002575026020529943, CT = 0.00018421403289709044 Epoch 260 done with rmse val = 0.00025667770768237686, CT = 0.00018421403289709044 Epoch 261 done with rmse val = 0.00025586381082269913, CT = 0.00018421403289709044 Epoch 262 done with rmse val = 0.00025506076429948, CT = 0.00018421403289709044 Epoch 263 done with rmse val = 0.00025426842260922353, CT = 0.00018421403289709044 Epoch 264 done with rmse val = 0.00025348664190814803, CT = 0.00018421403289709044 Epoch 265 done with rmse val = 0.00025271528000088596, CT = 0.00018421403289709044 Epoch 266 done with rmse val = 0.00025195419632915344, CT = 0.00018421403289709044 Epoch 267 done with rmse val = 0.0002512032519603862, CT = 0.00018421403289709044 Epoch 268 done with rmse val = 0.000250462309576342, CT = 0.00018421403289709044 Epoch 269 done with rmse val = 0.0002497312334616681, CT = 0.00018421403289709044 Epoch 270 done with rmse val = 0.00024900988949243176, CT = 0.00018421403289709044 Epoch 271 done with rmse val = 0.00024829814512461533, CT = 0.00018421403289709044 Epoch 272 done with rmse val = 0.00024759586938257274, CT = 0.00018421403289709044 Epoch 273 done with rmse val = 0.00024690293284744975, CT = 0.00018421403289709044 Epoch 274 done with rmse val = 0.00024621920764556583, CT = 0.00018421403289709044 Epoch 275 done with rmse val = 0.00024554456743675876, CT = 0.00018421403289709044 Epoch 276 done with rmse val = 0.0002448788874026923, CT = 0.00018421403289709044 Epoch 277 done with rmse val = 0.0002442220442351269, CT = 0.00018421403289709044 Epoch 278 done with rmse val = 0.00024357391612415431, CT = 0.00018421403289709044 Epoch 279 done with rmse val = 0.00024293438274639717, CT = 0.00018421403289709044 Epoch 280 done with rmse val = 0.00024230332525317384, CT = 0.00018421403289709044 Epoch 281 done with rmse val = 0.00024168062625862997, CT = 0.00018421403289709044 Epoch 282 done with rmse val = 0.0002410661698278381, CT = 0.00018421403289709044 Epoch 283 done with rmse val = 0.0002404598414648664, CT = 0.00018421403289709044 Epoch 284 done with rmse val = 0.0002398615281008178, CT = 0.00018421403289709044 Epoch 285 done with rmse val = 0.00023927111808184153, CT = 0.00018421403289709044 Epoch 286 done with rmse val = 0.00023868850115711838, CT = 0.00018421403289709044 Epoch 287 done with rmse val = 0.0002381135684668217, CT = 0.00018421403289709044 Epoch 288 done with rms