claude shannon, entrophy, id3 alogrithm, information gain.

ID3 Algorithm:
ID3 invented by Ross Quinlan used to generate a tree from a dataset. its precursor of C4.5 algorithm, used in machine learning and NLP
domains.
it begins with the original set S  as the root node. IT calculates the entrophy  H(S) (or information gain IG(S)) before, it iterates through
every unused attributes of the set S on each iteration of algorithm. It n selects the smallest entrophy (or largest information gain)value.
the set S is then spit by the selected attribute (e.g age is less than 50, age is between 50 and 100, age is greater than 100 ) to produce
subsets of the data. The algorithm continues to recurse on each subset, considering only attributes never selected before. 
INformation Gain in decision tree:
it is used as mutual information, which is the expected value of the kullback leibler divergance of the univariate probability distribution
of one variable from the conditional distribution of its variable given the other one.
In general terms, the expected  information gain is the change in the information entrophy H from a prior state that takes some information as given:
IG(T,a)=H(T)-H(T|a).
Entrophy:
it is defined as the average amount of data produced by the stocastic(random) source of data.
Claude Shannon:(april 30 1916 ---- 2001)
american mathematician, electrical engineer and cryprtographer known as father of information theory. noted for having founded information theory with a
landmark paper, A Mathematical Theory of communication, published on 1948, he is equally known for founding 
 digital circuit thoey in 1937.

Comments

Post a Comment

Popular posts from this blog

Aspect Ratio

Image
What is Aspect Ration? The height and width on an image is referred to as the aspect ratio. It defines its overall shape, and it is usually shown as W:H where W is width and H is height. The most common aspect ratio today is 16:9, which means that if the width is divided into 16 equal parts, the height of the TV or picture should be 9 parts.   16:9 works great for TVs since that is the format modern TV shows are delivered on, but most movies are made using the cinema standard, which is 21:9. 21:9 is much wider, so parts of the screen need to be filled with black bars above and below the image in order to fit most TVs. These horizontal bars are called ''letterboxes''. Similar to movies, TV shows used to be made using a 4:3 aspect ratio, which is a lot more square than current TVs(this is why 16:9 is often called a widescreen aspect ratio). To fit modern TVs, vertical black bars or 'pillarboxing' is used. We've listed the most common aspect ratios in the t...
Read more

Why People attend Higher education institutions

Image
  Human from all over the world go to educational institutes for several reasons. There could be numerous different reasons behind each individual’s decision of attending higher education.   Generally, humans make decisions based on logic or emotions. In my opinion, there are two basic reasons due to which an individual would acquire higher studies i.e. either need or interest. There could be thousands of needs that an individual wants to fulfill by getting a scholarly degree the most basic one is the financial needs. Most people get scholarly degrees to get a job that fulfills their financial needs but there could be thousands of other needs that a person might be able to fulfill by achieving a degree for instance, one could need a degree to achieve the standards set by the society. An example of this can be   the people from developing countries like India and Pakistan which are now shifting towards getting higher education to achieve the standards set by the western ...
Read more

Quarantine affecting different people

Image
Covid-19 itself has a huge impact on everyone psychologically, but quarantine has an impact on everyone differently according to their personalities. But mostly, it affected people badly in different ways, especially financially. Many people lost their jobs and source of income.   When quarantine began, it looked like it is heaven for all the introverts out there yet extremely difficult for extroverts. Introverts are people who do not like to socialize a lot. For them, quarantine was like a dream coming to reality because they do not have to socialize much and prefers to spend time alone or with some people whom they know intimately. Introverts in quarantine are more energized than ever. Being introverted does not mean you do not like people, it means you just do not like to talk because it just means that doing it de-energizes you. They get moody and irritated when they do not get enough alone time. So quarantine was a great advantage for introverts to get away from crowds and int...
Read more