Summary of Computational Learning TheoryKui Tang December 2012Prepared from notes and homeworks from Professor Rocco Servedio’s Fall 2012 course in preparation for the final. No guarantees of completeness or correctness. Contents 1 Online Learning 2 1.1 Elementary Algorithms . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 Perceptron . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.3 Winnow . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.4 (Randomized) Weighted Majority . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 1.5 General Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 2 PAC Learning 5 2.1 Probability Inequalities . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.2 OLMB to PAC . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.3 Consistent Hypothesis Finder; Occam’s Razor . . . . . . . . . . . . . . . . . . . . . . . . . . . 6 2.4 Hardness of 3-Term DNF Learning; Proper and Improper Learning . . . . . . . . . . . . . . . 7 2.5 VC Dimension Lower-Bounds PAC Sample Complexity . . . . . . . . . . . . . . . . . . . . . . 8 3 Bounds for Infinite Hypothesis Classes 8 3.1 Growth Functions and Examples . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 3.2 Sauer’s Lemma . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 3.3 Infinite CHF (Method of Double Sampling) . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 4 Boosting 10 4.1 Three-Stage Boosting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 4.2 Recursive Boosting . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 4.3 AdaBoost . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 5 Noisy PAC 12 5.1 Statistical Queries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12 5.2 Example: Monotone Disjunctions Noisy PAC . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.3 Lower Bounds . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.3.1 Sample Complexity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 5.3.2 Noise Rate and Adversarial Noise . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13 6 Representation-Independent (Cryptographic) Hardness 13 6.1 Pseudorandom Functions are Hard to Learn . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 6.2 Discrete Cube Roots are Hard to Learn . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 1Background image of page 1Info iconThis preview has intentionally blurred sections. Sign up to view the full version.View Full DocumentRight Arrow Icon1 Online Learning 1.1 Elementary Algorithms To learn a one-decision list, use extended decision (multiple rules; if multiple antecedents are satisfied, arbitrarily choose one) lists are your hypotheses. Begin with all 4 n + 2 rules. Each time a rule fires and returns a wrong answer, such that that rule is responsible for the error, we push the rule down to the next level. A rule is pushed iff a mistake is made (show the converse side), so we have at most O ( n ) mistakes. Decision lists can be encoded as LTFs with exponentially decaying weights. Recall that a matching rule short-circuits the rest of the decision list evaluation. In an LTF with exponentially-decaying weights, the first matching rule overwhelm all subsequent coefficients [HW1.1b]. [HW1.2] Any OLMB algorithm can be made conservative: Suppose algorithm A has mistake bound M . Algorithm A0 predicts with its current hypothesis. If it turned out to be a mistake, A0 gives the current example to A and takes A0 s output hypothesis. The two algorithms make the same number of mistakes on the subsequence that causes mistakes. This is because A only sees this sequence of mistake examples. Therefore, A0 can make no more mistakes than A itself would have. Randomized halving algorithm: if we remove the oblivious adversary, a more efficient algorithm (in expectation) can be obtained: maintain the set of consistent-so-far hypothesis, and on each mistake, pick h uniformly at random from the new set of consistent hypotheses. Then for any fixed target c and training sequence ± x i ² (this is the oblivious adversary assumption), we have E [ M ] ≤ ln |C| + O (1) . Proof idea: order the concepts in CONSIST by the order in which the fixed training sequence will eliminate them
Try it now!
How it works?
Follow these simple steps to get your paper done
Place your order
Fill in the order form and provide all details of your assignment.
Proceed with the payment
Choose the payment system that suits you most.
Receive the final file
Once your paper is ready, we will email it to you.
Our Services
Ace Writing Center has stood as the world’s leading custom essay writing services providers. Once you enter all the details in the order form under the place order button, the rest is up to us.
Essays
At Ace Writing Center, Nowadays, students normally have extremely busy schedules. You will note that some of them have to take on some evening or weekend jobs in order to get some income that can help them to sustain in college or in the university. This can deny them a chance to write all the essays given. Others usually get bombarded with a lot of work by their lecturers. This can still delay such students from working on all their essays. However, some of them usually try to work on all these essays but end up delivering their work late. This can prevent them from graduating since most lecturers are strict on deadlines. If you want to write a business essay, the wise way is to hire an outstanding essay writing service like us, so that you can get the best results. If you are keen, you will note that many companies usually overcharge their customers. Some of them are there only to make money. And in reality, they really don’t care to build a long term commitment with students. You should not choose such companies. You should take your time and choose a reliable company to work with. Ace Writing Center is the ultimate solution for you. We have been offering our writing service for more than 7 years. This is a clear indication that you will get quality essay writing service. We have a wide range of writers who can work on any business essay that you might have. We believe in doing extensive research so that we can provide quality work to all our clients. .
Admissions
Admission and Business Papers
Have you ever had to write an admission essay for college? The majority of students face the same issues when applying to a university or college and many in such situations decide they need professional help to cope with this matter. They get in a situation when the deadline keeps coming closer but lack motivation to start because they are just not sure if their writing skills are strong enough. We have a solution for you! Ace Writing Center is the best admission essay writing service with a large professional team and years of experience in providing high-quality papers to students of all levels and faculties. The mission of our team is to help students make their dreams of entering a good college come true and that’s what we offer!.
Editing
Editing and Proofreading
Sometimes all the words for your paper just flow out of your mind and into your fingers. You type quickly at your keyboard and there they are, your beautiful words right there on the screen. But you have no idea how to polish it up. You may be wishing there was a paper writing service that offered this type of writing service. Look no more! Here at Ace Writing Center, we offer you an editing and proofreading option that you can't find anywhere else..
Coursework
College Essay Writing
In case you are familiar Ace Writing Center, you know the way to distinguish a better company from a cheap one exactly. First of all, poor service website does not have a sufficient support. We think support team is an essential part of success; it has to answer all clients’ questions and be a connecting link between clients and their writers. On our web-service you will get answers about anything you need and your writer will receive all your instructions, assignments and requirements exactly and swiftly. A writing service that we run has got a flexible pricing system that will save you from senseless wastes and many bonus systems that let you sparing money for something important for you.
