life is short - you need Python!

generate all subsets using recursion | how to

Yesterday I wrote another program to generate all subsets of a set (array / list) during my preparation for today's class on backtracking at National Programming Camp. I wrote the code in C but just converted it into Python for the readers of the blog, so that they can understand and appreciate the beauty of recursion / backtracking.

 taken = []  
     
 def process(X):  
     print X  
   
 def generate_all_subsets(li, X, i, k, m):      
     process(X[0:i])  
   
     for j in range(k, m):  
         if taken[j] == False:              
             if i > 0:  
                 if li[j] <= X[i-1]:  
                     continue  
   
             taken[j] = True  
             X.append(li[j])  
             generate_all_subsets(li, X, i+1, j+1, m)  
             taken[j] = False  
             X.pop(i)  
           
 if __name__ == "__main__":  
     li = [1, 2, 3, 4]  
     X = []  
     taken.extend([False, False, False, False])  
     generate_all_subsets(li, X, 0, 0, li.__len__())  
       

life is short - you need Python!

Python code to search tweets in twitter

Recently I wrote a code to search keywords in twitter and extract the tweets as a part of a larger program. It uses twitter search api. I guess some of you might find this code snippet useful. :)

 import requests  
 import sys  
 import time  
 from datetime import datetime  
   
   
   
 def store_tweets(keyword, results, max_id):  
   tweet_list = []  
     
   for item in results:  
     tweet_id = item['id_str']  
     tweet_text = item['text'].encode('utf-8', 'ignore')  
     if len(tweet_text) > 255:  
       tweet_text = tweet_text[0:255]  
     from_user = item['from_user'].encode('utf-8', 'ignore')  
     from_user_id = item['from_user_id']  
     from_user_name = item['from_user_name'].encode('utf-8', 'ignore')  
     profile_image = item['profile_image_url'].encode('utf-8', 'ignore')  
     created_at = datetime.strptime(item['created_at'][:-6], "%a, %d %b %Y %H:%M:%S")  
       
     tweet_list.append((keyword, tweet_id, tweet_text, from_user, from_user_id, from_user_name, profile_image, created_at))  
       
   # Now store the tweet list in a database or file  
   return  
   
   
   
 def search_tweets(keyword, max_id = '', result_type = 'mixed'):  
   url = 'http://search.twitter.com/search.json'  
     
   params = {'q': keyword, 'rpp': '100'}  
     
   if len(max_id) > 0:  
     params['since_id'] = max_id  
       
   params['result_type'] = result_type  
   r = requests.get(url, params=params)  
   time.sleep(5)  
     
   if r.status_code != 200:  
     print r.status_code  
     print r.text  
     time.sleep(30)  
     return r.status_code  
   
   json_content = r.json  
     
   max_id = json_content['max_id_str']  
     
   while True:  
     results = json_content['results']  
     if len(results) == 0:  
       print "no result"  
       break  
         
     store_tweets(keyword, results, max_id)  
       
     if 'next_page' in json_content.keys():  
       next_page = json_content['next_page']  
     else:  
       break  
       
     r = requests.get(url + next_page)  
     time.sleep(5)  
     print r.status_code  
     if r.status_code != 200:  
       print r.status_code  
       print r.text  
       time.sleep(30)  
       return r.status_code  
   
     json_content = r.json  
       
     
     
 if __name__ == "__main__" :  
   keyword = 'python'  
   search_tweets(keyword)  
     
     
   

life is short - you need Python!

Python Code to Search Youtube Comments

Today I am sharing a code that I wrote few months ago. It searches the comments in youtube video and stores those in a database (postgresql in my code). I used the youtube api. Instead of describing the code in detail, I think posting the code should be enough. If you have any queries or any suggestion to improve the code, feel free to comment.

 import gdata.youtube  
 import gdata.youtube.service  
 import psycopg2  
 import sys  
 from datetime import datetime  
 import time  
 import re  
   
   
   
   
 try:  
   # you should read these from a config file  
   conn = psycopg2.connect(host='localhost', database='databae_name', user='user', password='pass')  
   conn.autocommit = True  
   cursor = conn.cursor()  
 except psycopg2.DatabaseError, e:  
   print 'Error %s' % e    
   sys.exit(1)  
   
   
    
 def since_epoch(date_string):  
   return int(time.mktime(datetime.strptime(date_string, "%Y-%m-%dT%H:%M:%S.%fZ").timetuple()) * 1000)  
    
   
   
 def comments_generator(client, video_id):  
   comment_feed = client.GetYouTubeVideoCommentFeed(video_id=video_id)  
   while comment_feed is not None:  
     for comment in comment_feed.entry:  
       yield comment  
     next_link = comment_feed.GetNextLink()  
     if next_link is None:  
       comment_feed = None  
     else:  
       comment_feed = client.GetYouTubeVideoCommentFeed(next_link.href)  
   
   
   
 def search_and_store_comments(client, keyword, video_id):  
   query = "SELECT MAX(updated) FROM yt_comments WHERE keyword = %s AND video_id = %s"  
   cursor.execute(query, (keyword, video_id))  
   result = cursor.fetchone()  
   if len(result) > 0:  
     last_update = result[0]  
   if last_update is None:  
     last_update = '0'  
   if last_update == '':  
     last_update = '0'  
   for comment in comments_generator(client, video_id):  
     author_name = comment.author[0].name.text.strip()  
     text = comment.content.text.strip()  
     update_time = comment.updated.text.strip()  
     update_time = since_epoch(update_time)  
     if update_time <= int(last_update):  
       break  
     print (keyword, video_id, author_name, text, update_time)  
     try:  
       cursor.execute("INSERT INTO yt_comments (keyword, video_id, author, text, updated) VALUES (%s, %s, %s, %s, %s)", (keyword, video_id, author_name, text, update_time))  
     except psycopg2.DatabaseError, e:  
       print 'Error %s' % e  
     
   
   
   
 def search_yt(keyword):  
   client = gdata.youtube.service.YouTubeService()  
     
   vid_pat = re.compile(r'videos/(.*?)/comments')  
   vid_list = []  
   
   for start_index in range(1, 1000, 50):  
     query = gdata.youtube.service.YouTubeVideoQuery()  
     query.vq = keyword  
     query.max_results = 50  
     query.start_index = start_index  
     query.orderby = 'relevance'  
     print start_index  
     feed = client.YouTubeQuery(query)  
       
     if len(feed.entry) == 0:  
       break  
   
     for entry in feed.entry:  
       if entry.comments is None:  
         continue  
       comment_url = entry.comments.feed_link[0].href            
       result = re.findall(vid_pat, comment_url)  
       if len(result) > 0:  
         video_id = result[0]  
       else:  
         continue  
       if video_id not in vid_list:  
         vid_list.append(video_id)   
       print video_id  
         
     time.sleep(1)  
         
   vid_list = list(set(vid_list))  
   for vid in vid_list:  
     search_and_store_comments(client, keyword, vid)  
   
   
     
 def main():  
   query = "SELECT keyword FROM yt_keywords"  
   cursor.execute(query)  
   result = cursor.fetchall()  
   for item in result:  
     keyword = item[0]  
     print "Searching for keyword:", keyword  
     search_yt(keyword)  
     time.sleep(1)  
       
     
     
 if __name__ == "__main__" :  
   main()  
   print "done"  

Fall in love programming again...

Fall in love programming again...

I have been biased towards C++ ever since I learned it in school. But even after years of using it, I still don't find myself writing C++ programs effortlessly. I tried other languages like C, Java and Visual Basic but found none appealing enough.

In last few months I read a lot about Python. But, I wasn't in any mood to learn a “scripting language”. I am embarrassed to admit that I thought scripting languages like Perl/Python were for kids not real programmers. Also, I hated scripting ever since my undergrad Unix course where I had to write ugly looking, error prone shell scripts.

I started learning Python just to try a book I downloaded - “A Byte of Python” by Swaroop. After reading a few chapters, I was quite impressed. Python worked for me perfectly. Unlike most people, I loved the concept of whitespaces for code blocks. In my belief, It's the perfect way to force people to indent their code.

Also, I am mostly able to write error free Python code in the first go. I don't know whether it's common but I almost always encounter errors on first compile/run in my C++ programs.

Now, after finishing the book, I love Python so much that I have decided to write all new programs in Python. Currently, I am writing a Matrix class to allow matrix manipulation like Matlab. After this, I'll start wxWindows (a GUI Toolkit) and Python Imaging Library.

I must say, If you love (or used to love) programming, give Python a try. It'll remind you of the reason you loved programming originally.

Diluting the role of the IIT JEE

Diluting the role of the IIT JEEThe JEE used to serve India well
Many years ago, high school education in India worked in a twin track system: There were those who studied for the IIT JEE and there was everyone else who didn't. The former studied good books like Resnick/Halliday, which is a college level book elsewhere in the world, solved physics problems from Irodov, etc.
In contrast, studying for the 12th standard ("board examination") tended to emphasise rote memorisation, focusing on trivial questions where you had to plug numbers into a formula, emphasised accuracy of calculation and good handwriting. I vividly remember a textbook for 11th class physics used in Maharashtra, which said that Newton's second law did not apply for living things and powered vehicles. The thoughtful author must have wondered how a stationary cat started walking without the action of an external unbalanced force, and resolved the problem by limiting the footprint of Newton's second law. The less time that kids spend in studying for board examinations, the better.
I used to be optimistic that the footprint of the enhanced curriculum, and complexity of examination questions, lay far beyond the tiny number of people who entered IIT. Even if only 2,000 kids entered IIT, if 40,000 of them studied for the JEE, it gave them world class capabilities at high school. In each cohort, we got 40,000 people who were very good by world standards. In a country with pervasively low capabilities, it was very useful having this slice of high inequality of knowledge, for it gave a group of people who were able to learn modern technology, connect to globalisation, and create firms which generate a lot of high-paying jobs. It is fashionable to complain about inequality of knowledge, but given that you are in a LDC with a very low mean, would you really rather have very low variance??
With this old configuration, we also got a nice tool for inter-generational class mobility. The middle class got their kids into IIT, and almost all these graduated into upper class by the time they were 30.
More generally, a lot of countries have found that high stakes examinations are a good thing. High stakes examinations push the work ethic, grow the ability of young people to work hard in a sustained manner with high concentration, ensure foundations of mathematics and science, and encourage a meritocracy. They create a self-selected elite of young people who are not immersed in and defined by mass culture. All these are good things.
Problems of the JEE
I used to think like this for a long time. I have reluctantly been persuaded, over recent years, that the JEE isn't working so well.
Too many young people are studying for the examination and not the subject. The obsessive focus upon coaching classes is producing a one-dimensional personality which isn't so well suited to entering college. In the 1980s, the most interesting students in IIT were thinking people who read books, knew a lot about the world, and could also solve monkeys on pulleys. With brutal competition, and the coaching classes phenomenon, too often, all that's left today is the monkeys on pulleys. There is a certain kind of parent who is willing to have a child go live in Kota at age 15. This screened out many families from the race.
Economists know about this phenomenon in agency theory. High-powered incentives are a problem because the agent only focuses on the incentive and tends to cut corners (or worse) on everything that's not mentioned in the incentive contract. Andrade and Castro bring this generic idea in agency theory into the question of examinations, and find similar effects.
In the 1980s, there was substantial diversity of background, experiences and class amongst the students. This was a good thing, since students would then pick up the culture of people unlike them. In recent years, it appears that there is much greater homogeneity of background, experiences and class. The extent to which the person gets transformed in the four years has, as a consequence, gone down. When very few children of the elite go to IIT, this reduces access to the knowledge and networks of the elite for everyone who goes to IIT. This has reduced the ability of IIT to generate inter-generational class mobility.
Jishnu Das and Tristan Zajonc have found a nice bump in the upper tail of the distribution of skills in India. The pessimist sees this as being about class or caste: certain families bring up kids who know more. The optimist in me used to think this was the bunch studying for the IIT entrance. Also see Geniuses and economic development on the importance of the upper tail of the skills distribution.
It is increasingly difficult to be optimistic about how this is going. Narendra Karmarkar graduated from IITB in 1978, and went on to do truly important work in 1984. My optimism about the IITs peaked in 1984. Phenomena like Narendra Karmarkar should have scaled up manifold in the following years. This has not happened. In the 1980s, I used to think that by 2010, we'd have atleast one Nobel laureate from the IITs. That has not happened. This tells us that the IITs are not delivering on their early promise; things haven't worked out well in the following years. While the IITs suffer from many problems, I think the JEE is also a part of the problem.
One of the most disappointing features of the recent OECD PISA evidence was the absence of this bump in the upper tail. This new evidence shows a scary world of low inequality of skill, of a country with a terrible mean and no upper tail of an elite that can power the country out of mass poverty. I would conjecture two potential explanations for what has been found. One, it could be the case that this testing was done at age 15, at which time not much of the IIT JEE studying has as yet taken place; we're only picking up the victims of board examinations. Alternatively, it could be the case that studying for the IIT JEE is distorted by the coaching class phenomenon, and is not producing good knowledge.
But the solution being offered doesn't seem to be the right one
There are two views on how these problems can be solved. The first alternative is to shift away from admissions based on a high stakes examination. Universities in the US screen applicants on many parameters, so this is generally thought to be better. But when we look back in history, universities in the US used to focus primarily on academic performance only, until a glut of Jews showed up in Harvard. The shift to asking for `well rounded personalities' was a tool by the dominant anti-semetic elite to screen out Jewish kids who did not play football. So we should be cautious in respecting the undergraduate admissions process in the US. It is also important to remember that the quality of kids starting college in the US is quite weak by world standards. There are other countries (e.g. Japan) where large scale high-stakes examinations are used for university admissions, with much success.
I feel that the core problem that we have in India is just too few seats, which has generated a ridiculous extent of competition and distorted behaviour on the part of the kids. The solution lies in solving the policy problems in higher education, so that a large number of kids are taken into world class institutions every year. E.g. adding undergraduate programs at I I Sc, with recruitment through the JEE, was a move in the right direction. We need to grow the size of the entrant class in universities in India, that figure in the Times Higher Education Supplement ranking, by 25-fold. At present, we have only one university in that list - IIT Bombay.

Kapil Sibal is offering neither of the two solutions above: we are not being offered a modified admissions process based on looking at a fuller picture of the child, and we are not being offered a Japanese scale world of high stakes examinations with a lot of seats in world class universities. What we're being offered is a scaling down of the role of the IIT JEE. A greater role for the 12th standard examination is just a recipe to emphasise rote memorisation, focusing on trivial questions where you had to plug numbers into a formula, emphasising accuracy of calculation and good handwriting. This seems wrong to me.

LIFE IN HOSTEL:

"Haan ye Umango se Phula hua Seena hai,

Lamhaa ye Maanga nahin, ise Humne Chheena hai!"

Hello Freshers!

The legacy of DTU, Mech-C, Friends, Bunks, etc. is all known to you till now. I would be highlighting a different dimension of college life, i.e. Life in Hostels! This article is certainly not restricted to the hostellers, but every first year student! Yes, the non-hostellers would not be spending their 24 hours in the hostel, but would be found there quite often, playing cards or TT, copying assignments (:-D), finishing practical files, blah, blah.

Hostel, often known as the 2^nd Home can be initially a very lonely & uncomfortable place. You will not know students around you, well not even your roomies. But, each one of you will be travelling in the same boat. New friends will be made, mess food will be ignored, & the lonely initial nights (usually 2-3) will be replaced by adventure & fun of a new kind altogether.Boys will be comparatively luckier than the girls because they can explore this fantabulous campus under moonlight. For majority of guys (:-D), it would surely not be romantic but full of joy & adventure. Be careful of seniors, because you might have an interview session much before than you expected. Remembering my good times, once we 5 friends finished with the campus exploration, we used to visit OAT every night. Silent & soothing; surely, one of the best & unforgettable moments I have lived.

You will be watching movies whole night, gossiping, chatting with the guy/girl you found interesting in college (;-)). Few would turn to Romeo & Juliet and the remaining would continue watching all the movies & games released till date. CS would get a new full form, i.e. Counter Strike (:-D) & Computer Science will soon be forgotten. Maggi Baba ki Maggi & chai, Night Canteen ke Paranthe, Hostel Room ki Mehfil would give you your life’s best moments where you will be all free, & living your life the way you want to live. There will be students from different parts of the nation. Some would spend all their time chilling out, some would study 24*7, & some would try to get a balance between the two. We get 1 life, 1 chance, so I would advise not to miss out on the fun part, but alongside keep putting in regular efforts towards your studies.

Last but not least, hostellers are loaded with freedom & advantages. But never forget, “Freedom always accompanies Responsibility”. Make full use of the advantages, fight out the negatives & never miss out on Fun (:-D)

Have a splendid college life ahead J

Cheers,

- PRAKHAR THAPAK

NOW

This section of the website contains information about the life at the DCE campus. Containing information about the technical societies, project teams, cultural societies, hobby groups, student initiatives amongst others, it tends to highlight those factes of the DCE life that lend to it its vibrant character.

Please go through the various sub-links of the 'Campus Life' navigator on your left for indepth information about the camous life at Delhi College of Engineering.

Community - Students

The convocation ceremony Every institution is known by the achievements of its students. Here at DTU, we are proud to state that some of the best brains in the country pursuing their programmes of interest and in the meanwhile excelling at whatever they initiate. Be it the academics, the project teams, the research activities, the technical competetions, the managerial skills or even extra-curricular activities, DTU's students have brought fame to the institution on the global platform. Community - Faculty The front view of the Department of Electrical Engineering This section lists information about the renowned faculty of Delhi College of Engineering. Spanning 12 departments, DCE takes pride in its faculty who have some of the biggest achievements to their name. Faculty of Department of Applied Chemistry & Polymer Technology Faculty of Department of Applied Physics Faculty of Department of Applied Mathematics Faculty of Department of Biotechnology Engineering Faculty of Department of Civil Engineering Faculty of Department of Computer Engineering Faculty of Delhi School of Management Faculty of Department of Environmental Engineering Faculty of Department of Electrical Engineering Faculty of Department of Electronics and Communication Engineering Faculty of Department of Humanities Faculty of Department of Information Technology Faculty of Department of Mechanical & Production Engineering Faculty of Department of Training and Placement

Community - Deans

The front view of the building of the faculty of technology. Dean, Academics Prof. S. K. Garg Dean, Industrial Research and Development Prof. S. Maji Dean, International Affairs Dr. S. K. Singh Dean, Student Welfare Prof. H C Taneja Dean, Continuing Education Prof. O. P. Verma Life at DCE - Project Teams Delhi College of Engineering lays a great emphasis on the importance of student project teams. The following teams are part of the R&D process that is integral to DCE. Unmanned Aircraft Systems (UAS) Solaris Super Mileage Vehicle (SMV) Resistance, Mini Baja   Defianz Racing Chandra Rath Hybrid Car Autonomous Underwater Vehicle(AUV) Moon Buggy Autonomous Urban Robot (AUR) Life at DCE - Cultural Societies Delhi College of Engineering has established the following cultural societies. Madhurima - The Music Society Pratibimb - The Dramatics Society Kalakriti - The Creative Arts Society Dhoom - The Dance & Choreography Society Furore - The Live Events Society SPIC-MACAY Life at DCE - Sports and Games Physical education & sports play vital role in achieving the aims and objectives of Education. The students of Delhi College of Engineering are provided with excellent facilities and encouraged to take part in the tournaments held in and around N.C.R. Delhi, , particulars, engineering institutions. Delhi College of Engineering is having 450 m. track, ground for Football, Hockey, Cricket, two courts for Volely ball, two courts for basketball, three courts for Tennis Iand five courts for Badminton, Table Tennis rooms, Chess Rooms, Carrom Rooms and Gyms are also available in the each hostel of the college campus. With the view to recognize upcoming talented sportsman and sports woman in the college, the Sports Council organizes sports festival ARENA. The festival witnesses the large participation of boys and girls which included atheletics, badminston, table tennis, basketball, carrom, chess, cricket, tennis and volleyball. Prizes and certificates were awarded to the winners. The sports year also involves the organization of the Inter Hostel Tournaments in various games i.e. badminton, basket ball, carrom, chess, cricket, football, table tennis, volleyball and tennis. The main cricket field and atheletics track in the sports complex. The volleyball, basketbal,tennis courts and football field in the sports complex. The main football field adjacent to the Director's residence.