1 00:00:00,150 --> 00:00:04,100 Okay guys Discrete Optimization, Mail Bag Week 1. 2 00:00:04,100 --> 00:00:07,316 So so whatever we are going to do, we're going to try to do because some of us may 3 00:00:07,316 --> 00:00:12,998 be travelling during some of the classes. it's basically giving you some update, 4 00:00:12,998 --> 00:00:16,790 weekly update as we can of what's going on in the, in the class. 5 00:00:16,790 --> 00:00:19,650 As you know giving you feedback, you know, answering some of the questions and 6 00:00:19,650 --> 00:00:21,844 so on. Okay, so that's what we going to do 7 00:00:21,844 --> 00:00:23,612 today. the first thing that I want to say is 8 00:00:23,612 --> 00:00:26,236 that's it's pretty cool to see all you guys actually doing these assignments 9 00:00:26,236 --> 00:00:30,770 and, you know, interacting on the forum. This is really, this is really cool. 10 00:00:30,770 --> 00:00:33,410 For the first time in this class we're actually realizing that there are people 11 00:00:33,410 --> 00:00:36,894 we are talking to, right? So so this is a little bit of the 12 00:00:36,894 --> 00:00:41,540 statistics. There are about 17,000 students active. 13 00:00:41,540 --> 00:00:46,104 in the class. 10,000 have viewed the, have viewed the 14 00:00:46,104 --> 00:00:50,506 introductory video. these are the completion of the 15 00:00:50,506 --> 00:00:53,058 assignments, so the screen name assignment, which was really tough, 16 00:00:53,058 --> 00:00:58,900 right? Was it was about 4,500 students. 17 00:00:58,900 --> 00:01:03,536 Knapsack we have more than 2,000 people completed, completing the knapsack 18 00:01:03,536 --> 00:01:07,440 assignment and rough coloring at this point, although it's due in a week time 19 00:01:07,440 --> 00:01:12,215 is about 250. Okay you know, a week time from the time 20 00:01:12,215 --> 00:01:17,051 I'm basically speaking, right? we have a lot of auditors which is 21 00:01:17,051 --> 00:01:20,528 typically common in these massively online courses, this is not to worry 22 00:01:20,528 --> 00:01:24,612 about. this is essentially the overview of the 23 00:01:24,612 --> 00:01:28,480 student body by country. There are some interesting things here. 24 00:01:28,480 --> 00:01:33,010 so the three top countries are the US, India, and Russia. 25 00:01:33,010 --> 00:01:36,360 Okay well, you know, you would expect something like. 26 00:01:36,360 --> 00:01:40,776 this is actually a picture of the entire world okay with a various countries and 27 00:01:40,776 --> 00:01:45,500 the number students in every one of the countries. 28 00:01:45,500 --> 00:01:49,028 It's a very nice coverage so we would like to encourage from, you know, that, 29 00:01:49,028 --> 00:01:53,480 that is not covered. You know, why is everybody not interested 30 00:01:53,480 --> 00:01:57,300 optimization in Greenland lot of in Africa? 31 00:01:57,300 --> 00:02:01,440 some tiny places in in South America, some in the middle of you know, Asia and 32 00:02:01,440 --> 00:02:04,960 so on. I mean, it's kind of an interesting map. 33 00:02:04,960 --> 00:02:09,417 This is the US, okay? So, a lot of people in California and 34 00:02:09,417 --> 00:02:15,210 nobody interested in discrete optimization in Wyoming. 35 00:02:15,210 --> 00:02:18,395 So if you know somebody in Wyoming, tell them just to register to the classes so 36 00:02:18,395 --> 00:02:22,280 that we feel good, right? So nobody in Wyoming, why? 37 00:02:22,280 --> 00:02:25,343 What is happening in Wyoming? Actually, I, so this is India, this is 38 00:02:25,343 --> 00:02:29,429 where a lot of people in India are actually are actually taking. 39 00:02:29,429 --> 00:02:32,662 Well, at least looking at some of the lectures in their, in their, in their 40 00:02:32,662 --> 00:02:36,389 class, okay. So some inten, intense activity in this 41 00:02:36,389 --> 00:02:40,790 part of India, okay? You can guess which one it is. 42 00:02:40,790 --> 00:02:44,442 this is like, you know, part of Europe and part of Asia, okay? 43 00:02:44,442 --> 00:02:47,544 Other part of Asia, okay? And you see once again the various 44 00:02:47,544 --> 00:02:51,284 activities of the various countries. There is a tiny little things here, very 45 00:02:51,284 --> 00:02:53,610 interesting, right? So, what is this thing? 46 00:02:53,610 --> 00:02:59,230 I have no clue. anyway so, so there are four you know, 47 00:02:59,230 --> 00:03:05,140 about 4,500 student active. That means students who've actually 48 00:03:05,140 --> 00:03:09,850 submitted some assignment, okay? So overall, there were 50,000 submission 49 00:03:09,850 --> 00:03:12,208 made. This, this is actually really, really 50 00:03:12,208 --> 00:03:14,602 interesting. I mean, we are really surprised that so 51 00:03:14,602 --> 00:03:17,096 many submissions have already been done, okay? 52 00:03:17,096 --> 00:03:20,156 For the knapsack assignments there are more than 2,000 people, students active, 53 00:03:20,156 --> 00:03:22,348 okay? And for the graph coloring assignment, 54 00:03:22,348 --> 00:03:25,036 although it's due in a week from the time I'm speaking, there are 250 students 55 00:03:25,036 --> 00:03:29,084 active right now, okay? so we're going to also give you progress 56 00:03:29,084 --> 00:03:32,565 by the various part in the statistics and, and so there are six parts for 57 00:03:32,565 --> 00:03:37,025 everyone of the assignments. And on track is going to be this 58 00:03:37,025 --> 00:03:40,034 completely crazy standard where essentially you basically very high 59 00:03:40,034 --> 00:03:43,535 standards. Where you have basically 7 to 10 on every 60 00:03:43,535 --> 00:03:46,884 one of the parts, okay? So, you are like, you know, the people in 61 00:03:46,884 --> 00:03:50,400 that track are certain to have a certificate at the end, okay? 62 00:03:50,400 --> 00:03:53,280 So, this is like doing, doing really great work on every one of the parts, 63 00:03:53,280 --> 00:03:56,406 okay? So, this is essentially the assignments 64 00:03:56,406 --> 00:03:59,193 part per student. So, the students who submitted various 65 00:03:59,193 --> 00:04:01,608 assignment. Of course, in this particular bar there 66 00:04:01,608 --> 00:04:04,580 is also the screen name, so that's the The highest. 67 00:04:04,580 --> 00:04:07,928 But you see people completing a lot of the possible, in that side, you know, 68 00:04:07,928 --> 00:04:12,678 about 59 [UNKNOWN] students have completed six or more parts here, okay? 69 00:04:12,678 --> 00:04:16,868 So, that's what you see over there. And obviously, what we want is to take 70 00:04:16,868 --> 00:04:20,900 this line over here and move it over there. 71 00:04:20,900 --> 00:04:25,840 And move all these, you know, body of students, you know to the right, okay? 72 00:04:25,840 --> 00:04:30,760 So, this is this is the same statistics, but for students who are on track, okay? 73 00:04:30,760 --> 00:04:33,425 So, what you see is that there are lot of students here which are close, close to 74 00:04:33,425 --> 00:04:36,769 completing knapsack, okay? We're just going to pull them a little 75 00:04:36,769 --> 00:04:38,822 bit. Such that they, you know, all these 76 00:04:38,822 --> 00:04:41,790 people go over the line, okay? Remember it's a tough class, okay? 77 00:04:41,790 --> 00:04:44,550 So, you don't have to have a perfect grade on everyone of the parts to move 78 00:04:44,550 --> 00:04:47,043 on, okay? So, sometimes you can move to the next 79 00:04:47,043 --> 00:04:48,930 assignment. You have to balance that, right? 80 00:04:48,930 --> 00:04:51,640 So, you can move to the next assignment, okay? 81 00:04:51,640 --> 00:04:54,456 and, and try to do the next assignment, and then you can always come back to 82 00:04:54,456 --> 00:04:57,556 Knapsack and try to get a, you know, a better grade. 83 00:04:57,556 --> 00:05:00,372 On, on better, you know, grade on some multiple part if you have a seven and you 84 00:05:00,372 --> 00:05:04,256 want really to get to 10. So, try to balance the time you spend to 85 00:05:04,256 --> 00:05:08,810 the assignment, you know, to the, to the, to the grade that you have, okay? 86 00:05:08,810 --> 00:05:13,035 So common question that we saw on the, on the forum, is there a mistake in the 87 00:05:13,035 --> 00:05:17,540 dynamic programming table in the knapsack lecture? 88 00:05:17,540 --> 00:05:19,503 Yes. Okay, once again, once we found the 89 00:05:19,503 --> 00:05:23,406 mistake or once it's reported, we typically try to correct it in the slide. 90 00:05:23,406 --> 00:05:26,656 It's discussed in many of the forum, it's discussed in the Wiki and things like 91 00:05:26,656 --> 00:05:29,236 this, okay? So that, you know, when you see some 92 00:05:29,236 --> 00:05:32,420 mistake, go to the forum, check if it has been reported or not. 93 00:05:32,420 --> 00:05:35,588 You don't have to repost or you don't have to or your don't have to you know 94 00:05:35,588 --> 00:05:39,010 you don't have to, you know, what until we answer. 95 00:05:39,010 --> 00:05:41,705 Most the the time the answer is on, already, already going to be there. 96 00:05:41,705 --> 00:05:47,480 so there was some interesting questions on how we test various assignments, okay? 97 00:05:47,480 --> 00:05:51,610 So we give you a lot of instances, but we test on all of them. 98 00:05:51,610 --> 00:05:56,508 But we are not completely stupid so we don't tell you which instances we are 99 00:05:56,508 --> 00:06:00,656 testing it on. so otherwise you could tailor everything 100 00:06:00,656 --> 00:06:04,218 you do to those specific instances. So, we choose a random set of instances 101 00:06:04,218 --> 00:06:08,482 from the set we are giving you. So, if your program does well on all the 102 00:06:08,482 --> 00:06:12,002 instances you find, okay? Otherwise, it's going to be on some of 103 00:06:12,002 --> 00:06:16,140 these instances that we're going to test the your assignment, okay? 104 00:06:16,140 --> 00:06:18,690 So, you know, we could do even worse than this, right? 105 00:06:18,690 --> 00:06:21,500 Give you some instance, and test on another test of instance. 106 00:06:21,500 --> 00:06:23,429 We don't do that. That's the right way to do it, but here 107 00:06:23,429 --> 00:06:26,090 we give you, we try to make your work, you know, simpler. 108 00:06:26,090 --> 00:06:28,849 We give you all the instance. If you do well on all these instances, 109 00:06:28,849 --> 00:06:31,930 you are guaranteed to do very well on, on the grading. 110 00:06:31,930 --> 00:06:35,647 so when, when you get out, you know, a feedback which is, which seems to be 111 00:06:35,647 --> 00:06:38,741 wrong. You know, check, check the order of the 112 00:06:38,741 --> 00:06:42,251 variables, and make sure that they match essentially the import order that we 113 00:06:42,251 --> 00:06:46,196 expect, and things like this. Once again, you know, when you see your 114 00:06:46,196 --> 00:06:48,868 assignment and you see the value of your objective function. 115 00:06:48,868 --> 00:06:51,590 You can always go the leader board and see how people are doing, okay? 116 00:06:51,590 --> 00:06:54,530 So, this is what is nice about this leader board in a sense. 117 00:06:54,530 --> 00:06:57,340 What we give you the value of the objective function. 118 00:06:57,340 --> 00:07:00,238 You can always see how well you do. If you do, if you're, you know, like 10 119 00:07:00,238 --> 00:07:02,746 times better than, you know, everybody else, probably that means something is 120 00:07:02,746 --> 00:07:05,077 wrong. If you are very low and you, you get a 121 00:07:05,077 --> 00:07:07,743 low grade, if you get a low grade, you can see all of the people have actually 122 00:07:07,743 --> 00:07:10,944 done better. And that, you know, it can motivate you, 123 00:07:10,944 --> 00:07:12,980 okay? So, so one of the things that this class 124 00:07:12,980 --> 00:07:15,950 does is, you know, trying to encourage some competition. 125 00:07:15,950 --> 00:07:18,360 You can always see how well the other people are doing. 126 00:07:18,360 --> 00:07:20,250 Now on the knapsack, it's reasonably easy. 127 00:07:20,250 --> 00:07:23,466 On graph coloring and other problems, it becomes much more interesting because 128 00:07:23,466 --> 00:07:26,594 some of these problems. It's very hard to actually find the 129 00:07:26,594 --> 00:07:30,280 optimal solution, graph, in graph coloring in particular, okay? 130 00:07:30,280 --> 00:07:34,099 Now, some people ask why branch bound for the knapsack problem can still be can it 131 00:07:34,099 --> 00:07:37,804 still be a exponential does it behave exponentially what are the typical case 132 00:07:37,804 --> 00:07:42,958 where it would be bad, okay? So, I'm going to show you some examples 133 00:07:42,958 --> 00:07:45,715 here, okay? So, once again, you know, if you want to 134 00:07:45,715 --> 00:07:49,279 find out if something is working well or not if an algorithm is working well or is 135 00:07:49,279 --> 00:07:53,354 not working well. Most of the time, what you have to do is 136 00:07:53,354 --> 00:07:56,092 exaggerate. Try to find a case which seems to be 137 00:07:56,092 --> 00:08:00,455 crazy and study hard the algorithm is working on it, okay? 138 00:08:00,455 --> 00:08:04,460 So, I'm going to assume here. Let's look at the problem for branch 139 00:08:04,460 --> 00:08:08,090 bound where we have a lot of items, right? 140 00:08:08,090 --> 00:08:09,674 But, they have all essentially the same ratio between the value in, you know, in 141 00:08:09,674 --> 00:08:11,570 the objective function, the profit, and then the weight, okay? 142 00:08:11,570 --> 00:08:18,080 So, assume they look all very very similar, okay? 143 00:08:18,080 --> 00:08:20,978 So, what's going to happen is that the branch and bound is going to basically 144 00:08:20,978 --> 00:08:24,980 explore many of the possibilities. Many of the combinations of these item. 145 00:08:24,980 --> 00:08:26,786 Why? Because the lower bound here in the 146 00:08:26,786 --> 00:08:30,130 branch and bound is not going to be able to differentiate them. 147 00:08:30,130 --> 00:08:33,206 So, let me give you an example. Assume that we have a knapsack of 148 00:08:33,206 --> 00:08:37,540 capacity, you know, 701, okay? The one is very important, okay? 149 00:08:37,540 --> 00:08:41,330 So, then you have the item weights, okay? And I'm giving you the item weights, 150 00:08:41,330 --> 00:08:43,870 which are basically 2, 2, 2, 2, 2, 2, 2, 2, 2. 151 00:08:43,870 --> 00:08:47,618 You know, huge amount of 2s, and then a 1 at the end, okay? 152 00:08:47,618 --> 00:08:52,430 And then, I'm giving you the item volume. Which are basically almost all the same. 153 00:08:52,430 --> 00:08:55,500 You know, a rate, I'm going to give you a ratio which is very, very similar. 154 00:08:55,500 --> 00:08:59,660 So we have the volume of the item which is 2.00001, and then all this huge number 155 00:08:59,660 --> 00:09:03,945 of items is like that. Except for the last one, okay? 156 00:09:03,945 --> 00:09:06,380 The one that you see there is going to be 1 as well. 157 00:09:06,380 --> 00:09:10,100 So, the ratio, essentially is mostly, you know, 1. 158 00:09:10,100 --> 00:09:13,380 Except that you have 1 plus epsilon for this huge amount of items, okay? 159 00:09:13,380 --> 00:09:16,790 So, you know, you need to think about what is the linear relaxation going to 160 00:09:16,790 --> 00:09:19,977 do, okay? So remember, this is, you need, well, you 161 00:09:19,977 --> 00:09:24,385 haven't seen those lecture yet, probably. But this, you know, the, the relaxation 162 00:09:24,385 --> 00:09:27,450 that we have, okay? Here is, is going to do something very 163 00:09:27,450 --> 00:09:30,388 interesting, okay? And you have to think about what, what it 164 00:09:30,388 --> 00:09:32,700 does, okay? But essentially, which are the items the 165 00:09:32,700 --> 00:09:35,649 most, which are the most valuable? That's the one with a two, okay? 166 00:09:35,649 --> 00:09:38,115 They are barely more valuable than the one with a one, okay? 167 00:09:38,115 --> 00:09:42,787 But since the capacity is, you know, 701, you will have to take item one at some 168 00:09:42,787 --> 00:09:46,632 point, okay? So, so that you really match the 169 00:09:46,632 --> 00:09:49,346 capacity, okay? So if you take an item two, you will only 170 00:09:49,346 --> 00:09:53,528 be able to take a fraction of it, okay? So in a sense, so you, you can see all 171 00:09:53,528 --> 00:09:56,757 the combination. We can take many, many, many, many of the 172 00:09:56,757 --> 00:10:00,562 computation of the twos, okay? Of the, of the items which have a two. 173 00:10:00,562 --> 00:10:04,136 And, but, and, and we go to ignore the 1 essentially until the very end. 174 00:10:04,136 --> 00:10:07,710 In general, if you, if you, if you order them, let's say by value, okay? 175 00:10:07,710 --> 00:10:10,540 Which is typically what the reaczation is doing, right? 176 00:10:10,540 --> 00:10:13,312 And so you, you can see, I know, you just should, you should do the exercise of 177 00:10:13,312 --> 00:10:16,540 seeing what the branch and bound is going to do, okay? 178 00:10:16,540 --> 00:10:19,780 And it's going to be pretty bad on the instances like this, okay? 179 00:10:19,780 --> 00:10:22,804 So in a sense, if you have a very large capacity, all kinds of, you know, items 180 00:10:22,804 --> 00:10:26,470 that are very tiny and we say, most follows the same ratio. 181 00:10:26,470 --> 00:10:29,475 The branch and bound algorithm is likely to spend a long time, okay? 182 00:10:29,475 --> 00:10:32,761 That's so you get exponential time in this particular, in this particular 183 00:10:32,761 --> 00:10:35,470 context. Of course, there are ways around it, 184 00:10:35,470 --> 00:10:37,770 okay? And when you get to the mixed integer 185 00:10:37,770 --> 00:10:41,510 programming lecture, you will see that we actually give you some ways of actually 186 00:10:41,510 --> 00:10:45,491 overcoming this this thing. So, there is a way to fix the branch and 187 00:10:45,491 --> 00:10:48,715 bound such that this doesn't happen. So, I'm giving you some kind of 188 00:10:48,715 --> 00:10:52,234 indication that there are techniques that we'll see later in the class that can fix 189 00:10:52,234 --> 00:10:55,108 this. So, it doesn't mean that all of the 190 00:10:55,108 --> 00:10:58,330 branch and bound algorithms are going to behave badly. 191 00:10:58,330 --> 00:11:02,685 I think that what is presented in class for an example is going to behave badly, 192 00:11:02,685 --> 00:11:06,390 okay? So some suggestions you know, before 193 00:11:06,390 --> 00:11:10,440 reporting a, a, a problem, okay? So, check the forum, okay? 194 00:11:10,440 --> 00:11:13,677 Check if the problem has been reported. You know, if you have, if you believe you 195 00:11:13,677 --> 00:11:16,287 have an issue, go and see the forum of the week to see if that problem has 196 00:11:16,287 --> 00:11:20,554 reported already, okay? check the Twitter account as well for you 197 00:11:20,554 --> 00:11:22,870 know, some of the announcements that we make. 198 00:11:22,870 --> 00:11:25,981 If we find, you know, mistakes in the slides, we would report them there as 199 00:11:25,981 --> 00:11:29,470 well, okay? use the course Wiki. 200 00:11:29,470 --> 00:11:31,020 We're going to tell you where it is, okay? 201 00:11:31,020 --> 00:11:34,206 So, this is the resource column here that you see. 202 00:11:34,206 --> 00:11:38,807 Go through course Wiki over here, and then you, this is the page that you see. 203 00:11:38,807 --> 00:11:42,461 And this is where you know we put the, you can put, we put corrections, you can 204 00:11:42,461 --> 00:11:46,252 put corrections. There will be some implementations and 205 00:11:46,252 --> 00:11:48,965 things like this, okay? So, you can use that, use these 206 00:11:48,965 --> 00:11:51,605 resources, okay? So, once again, there're lot of resources 207 00:11:51,605 --> 00:11:54,613 in the class in the, on the website and you should actually look at them all the 208 00:11:54,613 --> 00:11:58,361 time. no one of the things I told you many 209 00:11:58,361 --> 00:12:02,082 times is that this is a hard class and the knapsack assignment is a warm up, 210 00:12:02,082 --> 00:12:06,170 okay? So, it's the easiest assignment. 211 00:12:06,170 --> 00:12:09,330 Rough coloring is actually is pretty difficult. 212 00:12:09,330 --> 00:12:13,578 Rough coloring is really the assignments when you'll experience exponential growth 213 00:12:13,578 --> 00:12:17,390 okay so it's very difficult to find optimal solution. 214 00:12:17,390 --> 00:12:19,835 It's very difficult to prove optimizing, okay? 215 00:12:19,835 --> 00:12:23,867 So, it's going to be the first experience where you will really see that, wow, 216 00:12:23,867 --> 00:12:28,530 these problems are really tricky, okay? It's really difficult. 217 00:12:28,530 --> 00:12:31,382 At the same time, you can find very high quality solution and, you know, the 218 00:12:31,382 --> 00:12:34,754 leaderboard is going to tell you how good you are doing. 219 00:12:34,754 --> 00:12:37,698 And in general, you know, what, what, what we found is that in this class there 220 00:12:37,698 --> 00:12:41,430 is a wide variety of solutions, okay? That people are going to produce, they 221 00:12:41,430 --> 00:12:44,732 can use very different techniques. And but, but the quality is going to, is 222 00:12:44,732 --> 00:12:47,540 going to differ. typically the run time is going to differ 223 00:12:47,540 --> 00:12:50,740 as well, but in this class we basically don't, you know, we, we, we don't check 224 00:12:50,740 --> 00:12:55,726 the run time, okay? so, so coloring is hard as I just said, 225 00:12:55,726 --> 00:13:01,500 and finding an optimal solution really hard, proving it even worse, okay? 226 00:13:01,500 --> 00:13:04,965 So once again, what you have to remember is that you don't have to have the 227 00:13:04,965 --> 00:13:08,718 ultimate solution. And prove the ultimate solution to, you 228 00:13:08,718 --> 00:13:12,140 know, get a certificate at the end of the class, right? 229 00:13:12,140 --> 00:13:15,220 So, 7 to 10 is fine. And we, we have reasonable standards. 230 00:13:15,220 --> 00:13:16,760 That doesn't mean that this is easy, right? 231 00:13:16,760 --> 00:13:21,660 So, that we don't expect you to find and prove optimality on these problems. 232 00:13:21,660 --> 00:13:24,344 Now if you you know, if you find and prove optimality on all the benchmarks 233 00:13:24,344 --> 00:13:26,750 that you have. Just give me a call. 234 00:13:26,750 --> 00:13:29,720 Don't tell anyone, just give me a call, okay? 235 00:13:29,720 --> 00:13:34,232 good news, okay? So, the, the graph coloring assignment is 236 00:13:34,232 --> 00:13:38,410 covering many of the lectures to some level, okay? 237 00:13:38,410 --> 00:13:40,490 so you have it in constrained programming. 238 00:13:40,490 --> 00:13:44,270 Mostly what I recommend is that before you actually try. 239 00:13:44,270 --> 00:13:47,930 If you use constrained programming, okay? So before you actually try it, you know, 240 00:13:47,930 --> 00:13:51,370 look at, you know, Lecture from 1 to 5 at least. 241 00:13:51,370 --> 00:13:54,710 Six is also useful if you want to be really clever, okay? 242 00:13:54,710 --> 00:13:58,245 So, that's what I would recommend before you try look at these lectures, okay? 243 00:13:58,245 --> 00:14:02,450 Up to Cp6, okay? It's also covered in one of the local 244 00:14:02,450 --> 00:14:07,356 search lectures section number four. Once again, go a little bit further and 245 00:14:07,356 --> 00:14:11,676 look at the metaheuristic lecture. That's when you can get a lot of clues on 246 00:14:11,676 --> 00:14:15,260 actually solving this using local search, okay? 247 00:14:15,260 --> 00:14:18,570 So, this is what I would recommend for people who are really brave, okay? 248 00:14:18,570 --> 00:14:22,200 So you can look at, you know, advanced techniques in mixed integer programming 249 00:14:22,200 --> 00:14:24,670 as well. But this is really advanced for this 250 00:14:24,670 --> 00:14:28,345 particular problem, okay? so thanks for all the feedback that you 251 00:14:28,345 --> 00:14:31,142 gave us, okay? We work very hard, we try to respond to 252 00:14:31,142 --> 00:14:34,879 all the questions as soon as possible. You know, keep going, I think we'd like, 253 00:14:34,879 --> 00:14:38,165 we'd like to take this, you know, this red, red line and move it, you know, to 254 00:14:38,165 --> 00:14:42,964 the, to the right, okay? and we'll be back next week with more, 255 00:14:42,964 --> 00:14:48,150 you know, answers to your questions and more feedback on the class, okay? 256 00:14:48,150 --> 00:14:50,970 Thank you guys, keep the good, keep the good work.