0:00:00.000,0:00:03.650 maintains still some of the flat files in our system 0:00:03.650,0:00:05.320 that should increase speed 0:00:06.140,0:00:07.490 the flat files of our system 0:00:09.600,0:00:11.090 that my primary motivation 0:00:11.100,0:00:12.190 for giving a talk was 0:00:12.310,0:00:13.800 to try to force myself 0:00:13.800,0:00:13.920 into actually doing[br]to try to force myself 0:00:13.920,0:00:15.500 into actually doing 0:00:15.500,0:00:16.560 the work that I'm going 0:00:16.560,0:00:18.340 talk about in this talk [laughts] 0:00:18.340,0:00:19.980 ???? 0:00:19.980,0:00:20.830 lightly successful 0:00:20.830,0:00:22.830 so I may introduce you some of them, but??? 0:00:23.930,0:00:25.570 become too much 0:00:25.570,0:00:27.570 of the professional procrastinator 0:00:27.570,0:00:30.230 to being influenced by even giving a talk 0:00:32.229,0:00:34.230 Let's see how this work out 0:00:35.110,0:00:37.820 The first couple of things I'm going to talk about 0:00:37.820,0:00:39.990 are just some general bugs statistics 0:00:39.990,0:00:43.520 I'll enjoy showing some plots 0:00:43.520,0:00:47.290 I'm going to introduce the basic arquitecture of debbugs 0:00:47.290,0:00:50.300 just to backup, debbug is 0:00:50.300,0:00:53.260 the system behind bugs are debian debug 0:00:53.260,0:00:55.600 so if you file the bug 0:00:55.600,0:00:57.970 you fix the bug 0:00:57.970,0:01:00.640 you wonder where bugs exists 0:01:00.640,0:01:02.340 you interact with that bugs 0:01:02.340,0:01:05.430 ???? 0:01:05.430,0:01:10.290 I'm going to talk about some of the new features 0:01:20.630,0:01:23.330 I'm going to talk about some planned features 0:01:23.330,0:01:26.110 specially features that won't happen If I suddenly 0:01:26.110,0:01:28.630 have two or three more people would help 0:01:28.630,0:01:31.520 and I hopefully would pleased with 0:01:31.520,0:01:33.720 of you or people who are missing 0:01:33.720,0:01:35.080 this talk online 0:01:35.080,0:01:38.020 or who may watch this talk recording later 0:01:38.020,0:01:40.730 to assist me in implementing some of these features 0:01:40.730,0:01:42.230 I'm a nice person 0:01:42.230,0:01:44.840 we are nice people 0:01:44.840,0:01:47.470 and I'd like to see any pro hackers 0:01:47.470,0:01:50.130 to see a css hacker or javascript hackers 0:01:50.130,0:01:52.130 or people who writes documentation helps 0:01:56.790,0:01:59.710 the goal of the BTS is to 0:01:59.710,0:02:01.310 report bugs 0:02:01.310,0:02:03.310 track the evolution of bugs 0:02:03.310,0:02:04.330 fix bugs 0:02:04.330,0:02:06.920 and hopefully reduce the impact of bugs 0:02:06.920,0:02:09.479 ??? 0:02:11.220,0:02:16.240 this is how many bugs do we have versus time 0:02:16.240,0:02:19.500 as you can see our bug growth is 0:02:19.500,0:02:21.490 roughly linear over time 0:02:21.490,0:02:23.490 and actually decreasing slowly but 0:02:23.490,0:02:26.100 that a huge number of bugs 0:02:26.100,0:02:29.940 people like to track exactly how many bugs we have 0:02:29.940,0:02:31.940 ??? 0:02:31.940,0:02:33.800 fun contest 0:02:33.940,0:02:38.180 guessing when the ticket bug will be filed fo ar example 0:02:38.180,0:02:41.940 the 760 000 0:02:41.940,0:02:44.270 bug will be filed 0:02:44.270,0:02:46.270 at I think 0:02:46.270,0:02:48.010 on September 2 0:02:48.010,0:02:53.150 and the 800 000 will be filed almost a year from now 0:02:53.150,0:02:54.700 on September 13 0:02:54.700,0:02:56.700 observing the linear progression 0:02:56.700,0:02:58.700 maintains itself 0:02:58.700,0:03:01.450 ??? enjoys that but he is not here 0:03:01.450,0:03:03.250 ?? 0:03:03.260,0:03:05.450 Anyway that's to show the bug reporting rate 0:03:05.450,0:03:08.740 We average roughly have a 42 bugs found by day 0:03:10.190,0:03:13.300 as you see, that's a huge number bugs 0:03:13.300,0:03:18.930 This is the bug closing graph 0:03:18.930,0:03:23.870 It's actually taking not bug closures. It's actually bugs being archetyped ¿? 0:03:23.870,0:03:27.250 but for the most part it's aproximated 0:03:27.250,0:03:30.870 bugs closure rate ??? 0:03:30.870,0:03:35.700 So, we close roughly 95 bugs per day 0:03:35.700,0:03:39.950 So from that you could imagine that the bug system is gaining 0:03:39.950,0:03:44.370 50 or so much bugs everyday that are not going to be fixed 0:03:44.370,0:03:48.750 Further than fortunately in this graph you can see that 0:03:48.750,0:03:52.610 the bug closing rate is decreasing 0:03:52.610,0:03:57.610 in contrast with the bug reporting rate awesome increasing 0:03:57.610,0:03:59.020 something that i've seen 0:03:59.020,0:04:03.820 in previous post I'v made on my blog 0:04:03.820,0:04:05.680 this is actually kind of disturbing 0:04:05.680,0:04:08.340 I'm not sure what that means for Debian as a whole 0:04:08.340,0:04:10.660 what means anything but 0:04:10.660,0:04:14.240 I'd much rather seen the overall rating increasing 0:04:14.240,0:04:17.519 than decreasing 0:04:18.560,0:04:21.440 In this graph you are familiar with 0:04:21.440,0:04:24.290 this is RC bugs from the bug ?? 0:04:24.290,0:04:30.210 lukely the RC bugs are those that matter for the next relase are decreasing 0:04:32.980,0:04:36.420 getting in line for a new release there 0:04:36.420,0:04:38.420 ?? 0:04:39.260,0:04:42.400 Ok. So those is enough on graphs. 0:04:42.400,0:04:45.960 Now I'm going to talk about 0:04:45.960,0:04:47.960 debbugs system and how it works 0:04:49.510,0:04:52.140 debbugs two main components 0:04:52.140,0:04:54.140 there is a mail backend 0:04:54.140,0:04:59.900 wich is what interact with any email 0:04:59.900,0:05:01.900 submits bugs ?? 0:05:01.900,0:05:04.540 or a bug number in bugs at ?? 0:05:04.540,0:05:08.020 Next system on deb ?? 0:05:08.020,0:05:12.850 wich has and old files and process you email ¿? 0:05:14.630,0:05:17.580 The other aspect in debbugs is a web project 0:05:17.580,0:05:24.900 That is what displays information on what bugs are on wich packets 0:05:24.900,0:05:27.440 and the bug ?? 0:05:27.440,0:05:29.440 and it's also here 0:05:29.440,0:05:31.500 on to another machine 0:05:31.500,0:05:36.690 on beach so that ?? faster ?? 0:05:38.000,0:05:40.550 that bugs interact with dak 0:05:40.550,0:05:44.280 wich is a software wich is responsable from maintaining the ?? 0:05:44.280,0:05:46.500 so duk tells debbugs 0:05:46.500,0:05:49.140 wich who maintains wich packages 0:05:51.000,0:05:53.140 so debbugs know who is ?? 0:05:53.140,0:05:54.790 ?? 0:05:54.790,0:05:58.770 It also tells debbugs wich packages are in wich suite 0:05:58.770,0:06:01.110 and wich arquitecture 0:06:01.110,0:06:07.880 so debbugs can calculate wether a bug is present in a particular suite 0:06:07.880,0:06:14.720 for example if the bug is fixed on stable or whether it's fixed in testing oin r stable 0:06:14.720,0:06:17.690 what was calculated by previous ?? 0:06:19.690,0:06:22.010 britney also 0:06:22.010,0:06:25.010 is the testing migration 0:06:25.010,0:06:28.150 software that migrates software from unstable 0:06:28.150,0:06:29.340 to testing 0:06:29.340,0:06:32.180 it uses information from the ¿buxtehude? as well 0:06:32.180,0:06:36.900 it regards to wether a package is becoming more buggy 0:06:36.900,0:06:38.060 or less buggy 0:06:38.060,0:06:40.060 by operating 0:06:41.030,0:06:43.440 the actual ?? that does that is 0:06:43.440,0:06:46.340 ??? 0:06:46.340,0:06:48.340 and it provides a list of bugs 0:06:48.340,0:06:50.690 and also does the RC bug graphs 0:06:50.690,0:06:53.640 but ir provides ???? 0:06:58.030,0:07:00.590 debbugs ?? is like this 0:07:00.590,0:07:02.940 mail comes in 0:07:02.940,0:07:06.470 there is spam processing that happens on the first step ?? 0:07:06.470,0:07:07.940 ?? 0:07:07.940,0:07:12.380 ??? 0:07:13.810,0:07:18.960 is primary responsable from keeping bugtrack system free of spam 0:07:18.960,0:07:21.250 and these spamming 0:07:25.190,0:07:29.270 ?? the little bit spam that actually make its way ?? 0:07:29.270,0:07:40.770 ?? 0:07:40.770,0:07:43.240 is not a great job 0:07:43.240,0:07:46.130 after that mails been despammed 0:07:46.130,0:07:48.670 then, it goes to processall 0:07:48.670,0:07:50.870 and sub process 0:07:50.870,0:07:53.580 is responsable from handleing 0:07:53.580,0:07:55.580 emails that ? submit 0:07:55.580,0:07:57.950 and emails that ?? bug numbers 0:07:57.950,0:08:03.140 like this ??? 0:08:03.140,0:08:07.110 service is responsable from handleing ?? 0:08:07.110,0:08:10.930 so any email you send ?? 0:08:10.930,0:08:14.760 now with the advent of 0:08:14.760,0:08:16.990 control at submit¿? 0:08:16.990,0:08:19.350 ???? 0:08:19.350,0:08:25.110 this diagram is getting a little bit bluried. It's actually an abstraction ?? 0:08:25.110,0:08:28.610 ??? 0:08:28.610,0:08:33.659 then all of the information is stored in flat files 0:08:33.659,0:08:37.419 and a db-h ? wich has 0:08:37.419,0:08:40.330 small hash functions split ?? 0:08:40.330,0:08:44.890 and is indexed with a couple of flat files indexes 0:08:44.890,0:08:53.470 and then a cgi scripts use both the indexes and the flat files system to display bugs 0:08:57.500,0:08:59.840 so that's how 0:09:01.230,0:09:04.890 debbugs worked before I started working on it 0:09:04.890,0:09:10.770 The current plan is to add on and basically replace this indexes 0:09:10.770,0:09:12.770 with a databse layer 0:09:12.770,0:09:19.120 and so I'm going to keep part of the flat files just because that's a well tested system 0:09:19.120,0:09:22.150 there are lots of things already parts of flat files ¿? 0:09:22.150,0:09:23.720 and I don't know how to deal with them ¿? 0:09:23.720,0:09:27.820 and add on top a postgresql based database 0:09:27.990,0:09:34.420 that, the cgi scripts ?? in order to display information ?? 0:09:34.420,0:09:41.060 this will help both, increase the speed you get results back 0:09:41.060,0:09:43.980 you are looking and ?? 0:09:43.980,0:09:47.850 and also enable you to do more complicated things like 0:09:47.850,0:09:51.960 ?? bugs that actually affects a particular version 0:09:51.960,0:09:56.010 without waiting for huge amounts of time 0:09:56.010,0:09:57.810 for a query to complete 0:09:57.810,0:10:01.140 so for example, you want to look at all security bugs 0:10:01.140,0:10:03.390 wich affects unstable 0:10:03.390,0:10:06.040 well that's actually a really hard query to do 0:10:06.040,0:10:08.310 without a database layer 0:10:08.310,0:10:11.840 so that's one of the major things a database to do ?? 0:10:11.840,0:10:16.870 so the script that actually handles loading things in the database is called dbugloader.sql 0:10:18.900,0:10:20.630 Debbugs is run in parallel 0:10:26.290,0:10:29.860 but perl has recently come quite a wise 0:10:30.080,0:10:33.000 in handling databases 0:10:33.000,0:10:35.930 Most everybody has adopted 0:10:35.930,0:10:38.820 the perl ?? by using DBIx 0:10:38.820,0:10:41.000 in order to talk to databases 0:10:41.000,0:10:43.490 very successful database abstraction 0:10:43.760,0:10:47.140 Anybody who has ever written code in DBIx 0:10:47.140,0:10:49.140 knows that it is extremely tedious 0:10:49.140,0:10:52.770 to do joing and complicated statements 0:10:52.770,0:10:58.110 ?? writing sql and scaping and etc. 0:10:58.110,0:11:00.110 using place orders 0:11:00.110,0:11:02.110 ?? you have to keep track them 0:11:02.500,0:11:05.600 DBA has classes and extensions 0:11:05.600,0:11:09.000 that bloms together a huge number of 0:11:09.000,0:11:14.310 ?? into a really concurrent database abstraction service 0:11:14.310,0:11:17.120 where if you give it your schema 0:11:17.120,0:11:22.770 It would build classes and enable you to talk to ?? result groups 0:11:22.770,0:11:25.790 from your database 0:11:28.210,0:11:29.460 It's a complete system 0:11:29.460,0:11:33.300 you can actually write a schema entirely in DBIx class 0:11:33.300,0:11:35.820 that you can then to SQL alike 0:11:35.820,0:11:39.800 You can convert into a PostgreSQL, mySQL, etc 0:11:41.780,0:11:45.140 In this case I'm interested in 0:11:45.140,0:11:46.670 In write for PostgreSQL 0:11:46.670,0:11:49.740 that actually will be the database package? 0:11:50.410,0:11:53.910 there may be an option eventually to use SQL Lite for testing but 0:11:53.910,0:11:57.880 my primary goal is to deploy PostgreSQL 0:11:58.390,0:12:00.200 I'm also in using 0:12:00.200,0:12:03.360 a bunch of classes that are specific to Debian 0:12:03.360,0:12:07.950 for example the Debian Version Extension to PostgreSQL 0:12:07.950,0:12:10.700 that enables you to sort by Debian's version 0:12:10.700,0:12:13.770 That's extremely important to DBI ¿? 0:12:13.770,0:12:17.610 And that is something that handles very well in PostgreSQL 0:12:17.610,0:12:21.430 What I've actually done is rewrite ?? directing SQL 0:12:21.430,0:12:24.880 and DBIx class has an extension called Schema Loader 0:12:24.880,0:12:27.470 which handles converting the SQL schema 0:12:27.470,0:12:32.660 into the class decorations for DBIx class automatically 0:12:32.660,0:12:35.850 so you just write plain old SQL like you used to 0:12:35.850,0:12:41.070 and it automatically creates all the database related 0:12:41.070,0:12:45.040 Plurar classes that you used to talk results from database 0:12:45.790,0:12:49.000 There is another module which handles deployment 0:12:49.000,0:12:52.690 it can do automated upgrades from 0:12:54.040,0:12:55.960 different schema revisions 0:12:55.960,0:12:58.540 so as you change your schema 0:12:58.540,0:13:01.080 it handles doing both upgrades 0:13:01.080,0:13:03.200 and new installs of the new schema 0:13:03.200,0:13:08.160 wich you can also do downgrades 0:13:08.160,0:13:09.770 and yo can do others things 0:13:10.350,0:13:14.100 in addition to just execute SQL alter statements you can also run 0:13:14.100,0:13:17.310 ¿pro code? or anything else you want at the database 0:13:17.310,0:13:19.000 IH operations¿? 0:13:19.190,0:13:26.690 so that enables much easier changes to the schema in the future 0:13:27.150,0:13:33.240 and finally the actual module in Debbugs ??? 0:13:33.240,0:13:34.930 is DebBugs DB 0:13:34.930,0:13:40.990 and so all of the database interactions classes in Debbugs ¿? 0:13:43.090,0:13:46.020 and so this is the ..... 0:13:46.020,0:13:49.810 It's kind of complicated but this basically tracks 0:13:49.810,0:13:51.810 all of the bugs relationships 0:13:51.810,0:13:55.030 It tracks who correspond with the DBs 0:13:55.030,0:13:59.810 has all this source package versions imaginary packae versions 0:13:59.810,0:14:05.290 version depences show for example when you uploaded version from a previous version 0:14:05.290,0:14:10.120 this ables all that to be tracked 0:14:11.060,0:14:14.830 Taken the DQL schema as 0:14:16.260,0:14:18.600 as inspiration but unfortunately 0:14:18.600,0:14:24.450 the debbugs ???? 0:14:24.450,0:14:27.880 classes but I think it makes sense so 0:14:27.880,0:14:31.390 Anyway if somebody uses PostgreSQL genius 0:14:31.390,0:14:34.410 or an SQL hacker and is interested in 0:14:34.410,0:14:39.720 maybe offering suggestions right ??? 0:14:39.720,0:14:41.720 ¿talk about? 0:14:45.740,0:14:47.500 Is actually pretty easy 0:14:47.500,0:14:49.750 just called debbugs SQL bugs 0:14:49.750,0:14:50.880 uploads them 0:14:51.070,0:14:53.980 There are two different part of bugs in debbugs 0:14:53.980,0:14:56.390 There is the one that you can actually modify 0:14:56.390,0:14:58.390 and there are the archive bugs 0:14:58.390,0:15:03.180 so this handles join both with both sets of bugs 0:15:03.180,0:15:05.840 can also load versioning information 0:15:05.840,0:15:09.030 this logs wich packages are depending on the ones 0:15:09.030,0:15:12.910 and de Debian ¿? the architecture 0:15:12.910,0:15:14.910 and source version 0:15:15.880,0:15:19.120 So, the SQL is actually working 0:15:19.120,0:15:22.960 ?? sql query 0:15:22.960,0:15:25.550 which you also write this easy in a DBIx class 0:15:25.930,0:15:28.880 let me show you that this actually works 0:15:45.530,0:15:49.280 ??? 0:15:49.350,0:15:54.370 I can run the SELECT statement wich is selecting the count of bugs WHERE 0:15:54.370,0:16:00.720 which has been modify since June or July 0:16:00.720,0:16:03.020 which are not done and 0:16:03.020,0:16:05.510 and which are 0:16:07.710,0:16:10.190 which are done and which there are an outer set 0:16:10.190,0:16:12.190 and the answer is 0:16:13.210,0:16:15.290 521 0:16:15.290,0:16:19.880 as you can see that is a full load of all the bugs in Debian 0:16:19.880,0:16:24.550 The actual SQL query executes very quick 0:16:24.550,0:16:31.530 ?? 0:16:37.430,0:16:41.340 I hopped to have more of this done by the time of this talk but 0:16:41.340,0:16:43.340 There still is a lot of work that's needed 0:16:44.700,0:16:47.480 The bugs file currently are not loaded 0:16:47.480,0:16:51.270 the bugs files are ?? 0:16:51.890,0:16:56.240 That's needed to enable full text searching index 0:16:56.240,0:16:59.840 It also currently doesn't do ?? 0:16:59.840,0:17:05.210 thats ?? faster loading ??