It is that time of the year again, where the anxious gullible is fed with enough printed material soaked in vague statistics based on “national poll” surveys that either conclude sometimes correctly with wrong reasons or conclude wrongly with correct reasons or is plain inconclusive and correctly so, because of the method of the survey.
One such survey has already been given the treatment by Prof. Madhukar Shukla of XLRI in his blog alternate perspective. What follows is not as hilarious as that one, but only a sober set of mundane doubts that I have.
I came to know of Science of the Times, the recent Outlook magazine survey for top 100 engineering colleges and top 25 medical colleges through a post in the nanopolitan, a blog by Dr. Abinandhan from IISc. By reading the cover story and the methodology of survey, one can form one’s opinion without having to read this post any further for my opinion.
From the methodology one could learn that
Cfore conducted the study in two phases. In the first, a pilot survey was conducted among students, faculty members and industry representatives to finalize the main parameters. Based on this, it identified five factors�intellectual capital, infrastructure and facilities, pedagogic systems and processes, industry interface and placements.
What is “intellectual capital” in the above survey method? How is it objectively defined and quantified for this survey to rank a particular institute?
If it is the “knowledge content” of the faculty of that institute, which could be measured by how many of them hold Ph. D. degrees, then as faculties of most of these engineering institutes would agree, a Ph. D. is not necessary for teaching the UG courses. I myself have been a faculty in one such engineering institute for three years, before my Ph. D. degree, but with reasonable success – I know this because my increments were based then on my teacher evaluation. So, how are institutions that actually have good UG faculty without Ph. D.s treated in this survey?
Next, let us say the “infrastructure and facilities” are measured by some standardized scale. One institute can prescribe a good syllabus for the UG program and may lack the infrastructure (lab equipment and machines, computers, classrooms with proper teaching aids etc.) to successfully impart the syllabus to the students. So it may not get a high rank in this category. Fair enough. But there are institutions that deliberately reduce the syllabus. For instance, a specific experiment may be supplemented with another simpler one or altogether cancelled to suit the (non)available infrastructure and (lack of) facilities. But on paper, this would mean that the infrastructure of that institute is “adequate” to teach the “prescribed” syllabus. How is this factored into this survey?
If “pedagogic systems and processes” include syllabus content and how it is taught (hopefully), how are these things quantified objectively?
The success of an institution lies solely with its output, in this case, UG degree holders educated in engineering. If the UG graduates are “successful” then the institute is successful. The only glitch is how to define success for the UG graduates. One measure is to find out whether they get a job after they graduate (or even before they do). In this context, I think the “placements” section is the only place where the actual worth of an institute is being measured. But this depends on various factors including past performance of an institute, the relevance of a particular UG program for that year’s job market, the job market itself for that year, etc. How are these factored into the survey?
Proceeding with the methodology of the survey we further learn that
In the second phase, a semi-structured questionnaire was designed and given to senior faculty members in various institutes who had more than ten years of teaching experience in different states.
Firstly, what is special about this “senior faculty” population? Has any statistics been done on how many of those “senior faculty” consulted, actually have taught UG program courses for “more than 10 years”? On the other hand, why not the opinions of “junior” faculty, who actually teach most of the UG programs in most of these institutes need not be considered?
Secondly, what about institutions with no “senior faculty” as defined by this survey, but are still producing UG graduates with reasonable “success” (they get jobs)?
Thirdly, what is “semi-structured questionnaire”? Does this mean the other remaining “semi” is “unstructured”? Why not simply make it known to the public, what the questions asked in this questionnaire are? If it is too detailed, it can be put in the internet.
Further, the methodology suggests
The respondents were asked to give weightages to different parameters and evaluate the institutes that they were aware of on a ten-point scale. The rating that the respondents gave to their own institutes was not considered.
“The respondents were asked to…..evaluate the institutes that they were aware of…” (bold-font mine)
Wait a minute: Does this mean if some institutes are not heard of or eluded the memory of the “senior faculties” at the time of the survey, they didn’t make it into the survey board at all? This is, a wee bit (only a wee bit, I assure you), confusing for me.
If I am the “senior faculty”, and since the rating that I give for my institute is not considered anyway, I could as well consciously put down any “rival institute” I chose to, by not having them in my top 10 at least. Before jumping at me that I am claiming all “senior faculty” are liars, just answer my implied question about the survey of how it avoids “human fudge factors” in an objective analysis.
Proceeding with this argument, if I were not from the IITs, I might just choose the IITs as top institutes rather than the non-IITs, since, I can safely assume that everyone else in my situation will do the same. It is no surprise then, that the IITs top the list.
The hierarchy amongst them (IITs) in the top list could as well be explained by the random perturbation in the opinions of the consulted senior faculties among the IITs itself, as the “senior faculty” from most of the rest of the institutes are going to anyway point IITs as the best in India (and so essentially doesn’t contribute to the survey to determine the top 10 at least). This argument could be proved if the survey is taken again exactly with the same population. Chances are high that there could be a redistribution in the first five, but all of them will again be IITs.
Finally, the methodology states
In all, 208 faculty members participated in the survey. The sum total of average rating score that each institute got against different parameters was used to rank the institutes. Only the engineering colleges that offered under-graduate programmes and that were evaluated by at least 10 senior faculty members got into the final list.
The cover story itself points out “there are 1,346 engineering colleges approved by the All India Council for Technical Education”. If one were to consider only 10 “senior faculty” in each of these institutes, their total population is 13460, out of which 208 is about 1.5 percentage (if it is only 1 “senior faculty” per engineering college then 208 is about 15.5 percentage and my question still is valid). What scientific statistical method is employed to pick 208 as the representative “statistically significant” group to conduct this survey?
If a magazine in a few pages cannot do a survey with all of these details in place, then why should one believe the results of the survey alone?
On the other hand, perhaps all of these questions are taken care of by the Cfore group, which conducted the survey. May be they or Outlook magazine can release the appropriate data and details in the internet so that credibility is given to their commendable efforts to provide the public with useful information and silence nit pickers and nosy parkers with some semblance of common sense.
As I am in a profession where as an academic exercise my very existence is sometimes suspected, I am unable to watch the nebulous clouds of reason serenade past my youthful and immature hot-head. Hence, I made the above post with questions about the method of survey (and only that). If most of them have been or will be answered by Cfore or Outlook, no, need not be to my satisfaction, but to the satisfaction of any objective standard, I can start rejoicing.
After all, the survey reports my department to be the top in India (and I believe in that, with or without this survey).
Tags: academics, aicte, cfore, education, engineering ranking, e-school ranking, iit, outlook, survey, top Indian engineering colleges