Wikipedia Pull Data from table sample

I was researching some data sets and had the need to pull population data from Wikipedia. I looked at several articles on doing this and they all seemed either very basic or not current. I had already done an article on the Wikipedia Python package so I decided to try to write one on pulling table data out of a Wikipedia page. The packages I used were urllib and Beautiful Soup. I could have done this with other packages (requests comes to mind) but this is very simple and it works. Many Wikipedia pages offer tables to display data and most of them happen to be the only tables on the page or have unique IDs. Unfortunately, on the "List_of_states_and_territories_of_the_United_States_by_population" page on Wikipedia, There is a table for state rankings and another for rankings by region as well. So I would need to only pull table data for the specific table I needed, which was not too hard.

So to start, how do you pull data from a Wikipedia page? urllib works for me so that's what I used:

So we import urlopen from urllib.request and beautiful soup from bs4. Setup the url and then open and read in the content. The next step is to setup your data with BeautifulSoup. Quick note: I have hardcoded some things and removed the code from the class it was in for simplicity sake. The next code snippet reads through the Beautiful soup(BS) dataset and parses the data for me.

for row in tableData.select('table.wikitable.sortable tbody tr'):
            rowCount += 1
            if done:
                break
            if rowCount >= 3:  # data_starts at row 3:
                colCount = 0
                for col in row.findAll('td'):
                    colCount += 1
                    if colCount == 1 and '—' in col.get_text():  # This ends the Pop data I want
                        done = True
                        break
                    if colCount == 3: # state is in column 3
                        state_data[col.get_text()] = ['None', 'None', 'None']
                        state_temp = col.get_text()
                    elif colCount == 4:  # column where 2019 population estimate is
                        state_data[state_temp][0] = (col.get_text())
                    elif colCount == 5:  # column where 2010 census population is
                        state_data[state_temp][1] = (col.get_text())
                    elif colCount == 12: # column where US population percent is
                        state_data[state_temp][2] = (col.get_text())

That first line is where most of the magic happens. Most tables on the Wikipedia site use the table classes, wikitable and sortable. BS has a great interface called select where you can use HTML tags (table in this case) and then further find the exact table you want using css tags, so that explains the .wikitable.sortable . Then we tell BS that we want to further filter our page table with the tbody and tr tags that are found within the table we want. The rest is just simplified for loop to loop through the table rows and pull out the State name and population statistics I needed. The result is the data shown below.

Here are the results for U.S Population by State:

State	2019 Population Estimate	2010 Census Population	Percent of US Population
California	39,512,223	37,253,956	702,885
Texas	28,995,881	25,145,561	698,503
Florida	21,477,737	18,801,310	696,468
New York	19,453,561	19,378,102	717,707
Pennsylvania	12,801,989	12,702,379	705,715
Illinois	12,671,821	12,830,632	712,864
Ohio	11,689,100	11,536,504	721,032
Georgia	10,617,423	9,687,653	691,975
North Carolina	10,488,084	9,535,483	733,498
Michigan	9,986,857	9,883,640	705,974
New Jersey	8,882,190	8,791,894	732,658
Virginia	8,535,519	8,001,024	727,366
Washington	7,614,893	6,724,540	672,454
Arizona	7,278,717	6,392,017	710,224
Massachusetts	6,892,503	6,547,629	727,514
Tennessee	6,829,174	6,346,105	705,123
Indiana	6,732,219	6,483,802	720,422
Missouri	6,137,428	5,988,927	748,615
Maryland	6,045,680	5,773,552	721,694
Wisconsin	5,822,434	5,686,986	710,873
Colorado	5,758,736	5,029,196	720,704
Minnesota	5,639,632	5,303,925	662,991
South Carolina	5,148,714	4,625,364	660,766
Alabama	4,903,185	4,779,736	682,819
Louisiana	4,648,794	4,533,372	755,562
Kentucky	4,467,673	4,339,367	723,228
Oregon	4,217,737	3,831,074	766,215
Oklahoma	3,956,971	3,751,351	750,270
Connecticut	3,565,287	3,574,097	714,824
Utah	3,205,958	2,763,885	690,972
Puerto Rico	3,193,694	3,725,789	3,725,789
Iowa	3,155,070	3,046,355	761,717
Nevada	3,080,156	2,700,551	675,173
Arkansas	3,017,804	2,915,918	728,990
Mississippi	2,976,149	2,967,297	742,026
Kansas	2,913,314	2,853,118	713,280
New Mexico	2,096,829	2,059,179	686,393
Nebraska	1,934,408	1,826,341	608,780
West Virginia	1,792,147	1,852,994	617,670
Idaho	1,787,065	1,567,582	783,826
Hawaii	1,415,872	1,360,301	680,151
New Hampshire	1,359,711	1,316,470	658,233
Maine	1,344,212	1,328,361	664,181
Montana	1,068,778	989,415	989,417
Rhode Island	1,059,361	1,052,567	526,466
Delaware	973,764	897,934	897,934
South Dakota	884,659	814,180	814,180
North Dakota	762,062	672,591	672,591
Alaska	731,545	710,231	710,231
District of Columbia	705,749	601,723	—
Vermont	623,989	625,741	625,741
Wyoming	578,759	563,626	563,626
Guam	168,485	159,358	—
U.S. Virgin Islands	106,235	106,405	—
Northern Mariana Islands	51,433	53,883	—
American Samoa	49,437	55,519	—
Back to top	Total States and Territories found: 56