Lecture 3: Big Ideas: Data Collection, Data Questions, Data Formats

• Announcements:
  • Discord #q-and-a is active and may have answers for you!
  • Also #rtyi, where we can post links to course-relevant articles, etc.
  • Lab 1 due Thursday at 10pm. Remote instructions have some new clarifications and are now linked from the lab writeup as well as the syllabus.
• Themes from the start of quarter survey
  • Experience levels: very mixed! Let's try to use this as an asset.
  • Concerns - new course, technology unfamiliarity (me too!)
  • Remote aversion: generally 2-5
    • Just being here is the biggest thing - focus, interactions, etc.
  • Overall: lots of excitement! This is gonna be super fun!
• Goals:
  • Think about who collects data, who makes it available, and know about some of the places you might find data available for download. (Skiena 3.2)
  • Asking questions: Think about what makes a good data science question and practice formulating some of your own. (Skiena 1.2)
  • Know the basics of some of the most common data formats, including CSV, JSON, XML, (SQL). (Skiena 3.1.2)

Data Collection

Two minute think: who (or which entities) collect data? Which ones might share that data with you?

Ideas from the class:

• Scott
• Researchers
• Government
• Academics
• Companies
• Websites
• Media / newspapers
• Stores
• Laboratories
• Observatories
• Individuals (enthusiasts)
• Citizen scientists
• Health care industry
• Banks / financial institutions
• Data scientists
• Data journalists
• People who write books
• People who wrote books / tablets / ...
• Web crawlers

A few ideas:

• Companies, sometimes (privacy/liability/trade secrets)
  • Google ngrams: https://books.google.com/ngrams
  • API - "application programming interfaces" - a way to programmatically ask for data. Many companies, incluing:
    • Twitter
    • New York Times
    • Facebook
    • Uber
• Governments
  • https://www.data.gov/
  • https://data.wa.gov/
  • https://www.whatcomcounty.us/716/Data
  • https://www.google.com/search?q=city+of+bellingham+data
  • https://data.seattle.gov/
• Scientists / academics
  • Have to search around - rarely available in centralized repositories; often not available.
    • https://scholar.google.com/ is good for finding papers related to a subject
• Other data sciency people
  • Data science/ML contests
    • https://www.kaggle.com/datasets
  • Data journalists:
    • https://data.fivethirtyeight.com/
    • https://open.nytimes.com/data/home
• You!
  • Any tech you have control over, you can log stuff!
    • smartphone, fitness tracker, ...
  • some online services let you download the data they have on you:
    • e.g.: https://support.strava.com/hc/en-us/articles/216918437-Exporting-your-Data-and-Bulk-Export#Bulk
  • Web scraping - later in this course

Asking interesting data questions

• Consider a couple example datasets:

  • IMDB: All things movies. https://www.imdb.com/interfaces/
    • Films: title, duration, genre tags, date, cast, crew, user ratings, critic ratings, ...
    • People (actors, directors, writers, producers, crew): appearances/credits, birth/death dates, height, awards, ...
  • Boston Bike Share data: https://www.bluebikes.com/system-data
    • Trips: Trip Duration, Start Time and Date, Stop Time and Date, Start Station Name & ID, End Station Name & ID, Bike ID, User Type (Casual = Single Trip or Day Pass user; Member = Annual or Monthly Member), Birth Year, Gender (self-reported by member)
    • Stations: ID, Name, GPS coordinates, # docks

• Think-pair-share: Come up wtih one interesting question you might want to answer with each of these datasets.

Ideas from the class:

• IMDB:

  • are movie crews getting bigger or smaller over time?
  • do certain actors/directors/etc tend towards longer or shorter films?
  • effect of (one-liners??) on ratings
  • does cameraperson affect critic ratings?

• Bikes:

  • which stations/routes frequented by annual vs causal riders
  • annual vs casual effect on time given the same route, or longer routes?
  • use distributed across bike
  • do members get their money's worth out of the annual memberships?

Insight: Datasets can often answer questions that the data isn't directly about.

• Example: baseball stats (http://www.baseball-reference.com) has details about ~20,000 major league baseball players over the last 150 years. This data includes handedness and birth/death dates, so we can study a possible link between handedness and lifespan.

Data formats

• CSV (our bread and butter)
  • example: avengers.csv, via https://github.com/fivethirtyeight/data/tree/master/avengers
  • Reading a CSV file in Python, 3 ways: https://www.youtube.com/watch?v=fbl8fMQ9tQM
  • See the arguments to pd.read_csv to see some of the craziness you can find in CSV files and some of the capabilities of pandas in reading them.
  • Variations: separators, quoting, headers or not, ...

Let's look at the raw contents of the avengers.csv file from Lab 1:

import requests

response = requests.get("https://fw.cs.wwu.edu/~wehrwes/courses/data311_21f/data/avengers/avengers.csv")
response.text

'URL,Name/Alias,Appearances,Current?,Gender,Probationary Introl,Full/Reserve Avengers Intro,Year,Years since joining,Honorary,Death1,Return1,Death2,Return2,Death3,Return3,Death4,Return4,Death5,Return5,Notes\rhttp://marvel.wikia.com/Henry_Pym_(Earth-616),"Henry Jonathan ""Hank"" Pym",1269,YES,MALE,,Sep-63,1963,52,Full,YES,NO,,,,,,,,,Merged with Ultron in Rage of Ultron Vol. 1. A funeral was held. \rhttp://marvel.wikia.com/Janet_van_Dyne_(Earth-616),Janet van Dyne,1165,YES,FEMALE,,Sep-63,1963,52,Full,YES,YES,,,,,,,,,Dies in Secret Invasion V1:I8. Actually was sent tto Microverse later recovered\rhttp://marvel.wikia.com/Anthony_Stark_(Earth-616),"Anthony Edward ""Tony"" Stark",3068,YES,MALE,,Sep-63,1963,52,Full,YES,YES,,,,,,,,,"Death: ""Later while under the influence of Immortus Stark committed a number of horrible acts and was killed.\'  This set up young Tony. Franklin Richards later brought him back"\rhttp://marvel.wikia.com/Robert_Bruce_Banner_(Earth-616),Robert Bruce Banner,2089,YES,MALE,,Sep-63,1963,52,Full,YES,YES,,,,,,,,,"Dies in Ghosts of the Future arc. However ""he had actually used a hidden Pantheon base to survive"""\rhttp://marvel.wikia.com/Thor_Odinson_(Earth-616),Thor Odinson,2402,YES,MALE,,Sep-63,1963,52,Full,YES,YES,YES,NO,,,,,,,Dies in Fear Itself brought back because that\'s kind of the whole point. Second death in Time Runs Out has not yet returned\rhttp://marvel.wikia.com/Richard_Jones_(Earth-616),Richard Milhouse Jones,612,YES,MALE,,Sep-63,1963,52,Honorary,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Steven_Rogers_(Earth-616),Steven Rogers,3458,YES,MALE,,Mar-64,1964,51,Full,YES,YES,,,,,,,,,Dies at the end of Civil War. Later comes back. \rhttp://marvel.wikia.com/Clint_Barton_(Earth-616),Clinton Francis Barton,1456,YES,MALE,,May-65,1965,50,Full,YES,YES,YES,YES,,,,,,,Dies in exploding Kree ship in Averngers Vol. 1  Issue 502. Brought back by Scarlet Witch. Dies again in House of M Vol 1 Issue 7. Is later brought back. \rhttp://marvel.wikia.com/Pietro_Maximoff_(Earth-616),Pietro Maximoff,769,YES,MALE,,May-65,1965,50,Full,YES,YES,,,,,,,,,Dies in House of M Vol 1 Issue 7. Later comes back\rhttp://marvel.wikia.com/Wanda_Maximoff_(Earth-616),Wanda Maximoff,1214,YES,FEMALE,,May-65,1965,50,Full,YES,YES,,,,,,,,,Dies in Uncanny_Avengers_Vol_1_14. Later comes back\rhttp://marvel.wikia.com/Jacques_Duquesne_(Earth-616),Jacques Duquesne,115,NO,MALE,,Sep-65,1965,50,Full,YES,YES,,,,,,,,,Dies in Avengers_Vol_1_130. Brought back by the Chaos King\rhttp://marvel.wikia.com/Hercules_(Earth-616),Heracles,741,YES,MALE,,Oct-67,1967,48,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/T%27Challa_(Earth-616),T\'Challa,780,NO,MALE,,May-68,1968,47,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Vision_(Earth-616),Victor Shade (alias),1036,YES,MALE,,Nov-68,1968,47,Full,YES,YES,,,,,,,,,Dies in Avengers_Vol_1_500. Is eventually rebuilt. \rhttp://marvel.wikia.com/Dane_Whitman,Dane Whitman,482,NO,MALE,,Dec-69,1969,46,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Natalia_Romanova_(Earth-616)#,Natalia Alianovna Romanova,1112,YES,FEMALE,,May-73,1973,42,Full,YES,YES,,,,,,,,,Killed by The Hand. Later revived with The Stone of the Chaste. \rhttp://marvel.wikia.com/Mantis_(Earth-616)#,Brandt,160,NO,FEMALE,,Aug-73,1973,42,Full,YES,YES,,,,,,,,,"Dies in Silver_Surfer_Vol_3_3. Actually ""fragments of her essence were spread out all across the universe"""\rhttp://marvel.wikia.com/Henry_McCoy_(Earth-616)#,Henry P. McCoy,1886,NO,MALE,Jul-75,Sep-76,1976,39,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Heather_Douglas_(Earth-616)#,Heather Douglas,332,NO,FEMALE,Jul-75,Sep-76,1976,39,Full,YES,YES,YES,YES,,,,,,,"Dies in Defenders_Vol_1_152. Later \'Obtained a new body.\' Dies in Annihilation:_Conquest_Vol_1_3. Comes back as ""As before"" also was once named Madame MacEvil because LOL. "\rhttp://marvel.wikia.com/Patricia_Walker_(Earth-616)#,Patsy Walker,557,NO,FEMALE,Nov-75,Sep-76,1976,39,Full,YES,YES,,,,,,,,,Died during time on Defenders. Eventually ressurected by villian Grim Reaper along with Mockingbird to fight the Avengers. \rhttp://marvel.wikia.com/Matthew_Hawk_(Earth-616)#,Matthew Liebowitz (birth name),197,NO,MALE,,Aug-75,1975,40,Full,YES,YES,YES,NO,,,,,,,Died during Return_to_the_Old_West. Resurrected by She Hulk in negotiation with Time Variance Authority. As a time traveller he died in 1938. Did not return\rhttp://marvel.wikia.com/Robert_Frank_(Earth-616)#,Robert L. Frank Sr.,106,NO,MALE,Apr-77,Jul-78,1978,37,Full,YES,NO,,,,,,,,,Died in Vision_and_the_Scarlet_Witch_Vol_1_2. Did not return however he was eventually cloned but the clone was killed immediately. \rhttp://marvel.wikia.com/Simon_Williams_(Earth-616)#,Simon Williams,692,YES,MALE,Apr-77,Mar-79,1979,36,Full,YES,YES,,,,,,,,,Died in Avengers_Vol_1_9. Actually just in a death-like coma. \rhttp://marvel.wikia.com/Yondu_Udonta_(Earth-691)#,Yondu Udonta,109,NO,MALE,,Feb-78,1978,37,Honorary,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Martinex_T%27Naga_(Earth-691)#,Martinex T\'Naga,100,NO,MALE,,Feb-78,1978,37,Honorary,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Charlie-27_(Earth-691)#,Charlie-27,132,NO,MALE,,Feb-78,1978,37,Honorary,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Nicholette_Gold_(Earth-691)#,Nicholette Gold,108,NO,FEMALE,,Feb-78,1978,37,Honorary,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Stakar_Ogord_(Earth-691)#,Stakar,100,NO,MALE,,Feb-78,1978,37,Honorary,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Vance_Astro_(Earth-691)#,Vance Astrovik,156,NO,MALE,,Feb-78,1978,37,Honorary,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Mar-Vell_(Earth-616)#,Mar-Vell,254,NO,MALE,,Jul-78,1978,37,Full,YES,YES,YES,YES,YES,NO,,,,,The bad penny of the Marvel universe. First killed in Secret invasion. Revived during the Chaos war. Died again during aformentioned chaos war. Resurrected by the Kree Empire using theæM\'Kraan Crystal. Third death during Avengers vs. X-Men. Has not return\rhttp://marvel.wikia.com/Carol_Danvers_(Earth-616)#,Carol Susan Jane Danvers,935,YES,FEMALE,,Apr-79,1979,36,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Samuel_Wilson_(Earth-616)#,Samuel Thomas Wilson,576,YES,MALE,,Jun-79,1979,36,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Jocasta_(Earth-616)#,Jocasta,141,YES,FEMALE,Jul-80,Nov-88,1988,27,Full,YES,YES,YES,YES,YES,YES,YES,YES,YES,YES,"From her article: Death1: ""Defeated Ultron and reversed the process leaving Jocasta a mindless husk."" Return 1: ""Ultron later revived Jocasta with a remote link activating the mental residue the Wasp left behind"" Death 2: ""Sacrificing herself to try to ki"\rhttp://marvel.wikia.com/Greer_Nelson_(Earth-616)#,Greer Grant Nelson,355,YES,FEMALE,,Sep-81,1981,34,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Jennifer_Walters_(Earth-616)#,Jennifer Walters,933,YES,FEMALE,,Jul-82,1982,33,Full,YES,YES,,,,,,,,,Dies during Red Hulk Saga. Returns when Lyra infiltrates Intelligenica and finds her in statis\rhttp://marvel.wikia.com/Monica_Rambeau_(Earth-616)#,Monica Rambeau,348,YES,FEMALE,Jan-83,May-83,1983,32,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Eros_(Earth-616)#,Eros,206,NO,MALE,Jun-83,May-84,1984,31,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/James_Rhodes_(Earth-616)#,James R. Rhodes,533,NO,MALE,May-84,Sep-84,1984,31,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Barbara_Morse_(Earth-616)#,Barbara Barton (nee Morse),374,NO,FEMALE,Jun-84,Sep-84,1984,31,Full,YES,YES,,,,,,,,,Killed by Mephisto. After Secret Invasion it turns out she was actually being impersonated by a Skrull the whole time and was alive and well. \rhttp://marvel.wikia.com/Moira_Brandon_(Earth-616)#,Moira Brandon,2,NO,FEMALE,,Nov-93,1993,22,Honorary,YES,NO,,,,,,,,,Died in her second appearance earns honorary Avengers status doing so. Stays dead. \rhttp://marvel.wikia.com/Benjamin_Grimm_(Earth-616)#,Benjamin Jacob Grimm,2305,NO,MALE,,Jun-86,1986,29,Full,YES,YES,,,,,,,,,Once killed during a battle with Doctor Doom.\' Brought back by the FF when they literally went to Heaven to get him. \rhttp://marvel.wikia.com/Bonita_Juarez_(Earth-616)#,Bonita Juarez,83,NO,FEMALE,,Sep-87,1987,28,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Marc_Spector_(Earth-616)#,Marc Spector,402,NO,MALE,Sep-87,Jun-88,1988,27,Full,NO,,,,,,,,,,NA but he he did die and return to get his powers prior to joining the Avengers. \rhttp://marvel.wikia.com/John_Walker_(Earth-616)#,John F. Walker,352,NO,MALE,,May-89,1989,26,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Human_Torch_(Android)_(Earth-616)#,Jim Hammond (alias),565,NO,MALE,,Nov-89,1989,26,Full,YES,YES,,,,,,,,,"Dies during New Invaders. Was ""was revived as a living weapon the Thinker intended to sell."" Prior to joining the avengers he had died like 2 times in the 40s. "\rhttp://marvel.wikia.com/Miguel_Santos_(Earth-616)#,Miguel Santos,112,NO,MALE,,Apr-91,1991,24,Full,NO,,,,,,,,,,NA: Has not died since joing the Avengers but prior to joining he fought the West Coast avengers and seemingly died. Later turned up under control of a villain\rhttp://marvel.wikia.com/Julia_Carpenter_(Earth-616)#,Julia Carpenter,218,NO,FEMALE,,Sep-92,1992,23,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/2ZP45-9-X-51_(Earth-616)#,X-51,149,NO,MALE,,Jun-92,1992,23,Full,YES,YES,,,,,,,,,Died in Deathlok Vol 2 #5. JK that was actually just a Life Model Decoy\rhttp://marvel.wikia.com/Chris_Powell_(Earth-616)#,Christopher Powell,168,NO,MALE,,May-93,1993,22,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Namor_McKenzie_(Earth-616)#,Namor McKenzie,1561,NO,MALE,,Dec-85,1985,30,Full,YES,YES,,,,,,,,,Namor was presumed killed in the battle with Atlantean Barbarians. Had actually survived and turned up in the South Pacific\rhttp://marvel.wikia.com/Scott_Lang_(Earth-616)#,Scott Edward Harris Lang,217,NO,MALE,Jan-87,3-Feb,2003,12,Full,YES,YES,,,,,,,,,Died in Avengers:_Disassembled. Was brought back in the Young Avengers: Children\'s Crusade arc because time travel but take my word for it it was actually kind of well done. \rhttp://marvel.wikia.com/Anthony_Druid_(Earth-616)#,Anthony Ludgate Druid,158,NO,MALE,,Apr-87,1987,28,Honorary,YES,YES,YES,YES,,,,,,,Shot with a Breathing gun and corpse dumped in trash. Revived by the Grim Reaper. Helped Avengers defeat the Reaper so his spirit could remain at peace. Later completely revived by the Chaos King.\rhttp://marvel.wikia.com/Marrina_Smallwood_(Earth-616)#,Marrina Smallwood,86,NO,FEMALE,,Dec-87,1987,28,Full,YES,YES,YES,YES,,,,,,,Killed by Namor with the Ebony Blade. Returned when The Master put her in a statis tube. Death 2 was when Namor mercy killed her. Returned to life a second time by the Chaos King. \rhttp://marvel.wikia.com/Ravonna_Renslayer_(Earth-6311)#,Ravonna Lexus Renslayer,41,NO,FEMALE,,May-88,1988,27,Full,YES,YES,YES,NO,,,,,,,Killed in Avengers_Vol_1_24. Revived by the Grandmaster. Died in Avengers:_Forever_Vol_1_3.\rhttp://marvel.wikia.com/Rita_DeMara_(Earth-616)#,Rita DeMara,68,NO,FEMALE,,Nov-88,1988,27,Honorary,YES,YES,,,,,,,,,Killed by Iron Man when he was controlled by Immortus. Revived during Chaos war. \rhttp://marvel.wikia.com/Dennis_Dunphy_(Earth-616)#,Dennis Dunphy,70,NO,MALE,,Jan-88,1988,27,Full,YES,YES,YES,NO,,,,,,,"Died in Captain_America_348. Actually ""Dunphy miraculously survived the crash and lived with an Inuit tribe."" Second death when killed by Sharon Carter.  "\rhttp://marvel.wikia.com/Gilgamesh_(Earth-616)#,,61,NO,MALE,,Feb-89,1989,26,Full,YES,YES,,,,,,,,,Killed by Neut who was working for Immortus. Reborn into a new body. \rhttp://marvel.wikia.com/Reed_Richards_(Earth-616)#,Reed Richards,2125,YES,MALE,,Feb-89,1989,26,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Susan_Storm_(Earth-616)#,Susan Richards (nee Storm),1761,NO,MALE,,Feb-89,1989,26,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Wendell_Vaughn_(Earth-616)#,Wendell Elvis Vaughn,293,NO,MALE,,Oct-89,1989,26,Full,YES,YES,,,,,,,,,Killed by the Cosmic Assassin. Actually turned into a being composed of pure energy. \rhttp://marvel.wikia.com/Ashley_Crawford_(Earth-616)#,Ashley Crawford,36,YES,FEMALE,,Jul-89,1989,26,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Dinah_Soar_(Earth-616)#,,22,NO,FEMALE,,Jul-89,1989,26,Full,YES,NO,,,,,,,,,Died in Great_Lakes_Avengers_Vol_1_1. Has not returned\rhttp://marvel.wikia.com/DeMarr_Davis_(Earth-616)#,DeMarr Davis,31,YES,MALE,,Jul-89,1989,26,Full,YES,YES,,,,,,,,,Sacrificed self so that Mr. Immortal could stop the villain Maelstrom. Returned as the Angel of Death. \rhttp://marvel.wikia.com/Val_Ventura_(Earth-616)#,Val Ventura,34,YES,MALE,,Jul-89,1989,26,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Craig_Hollis_(Earth-616)#,Craig Hollis,33,YES,MALE,,Jul-89,1989,26,Full,NO,,,,,,,,,,Christ; where to begin. His superpower is that he cannot die when he dies he immediately comes back. The Wikia inventories 24 deaths. These will be excluded from the analysis. \rhttp://marvel.wikia.com/Gene_Lorrene_(Earth-616)#,Gene Lorrene,4,NO,MALE,,5-Apr,2005,10,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Doreen_Green_(Earth-616)#,Dorreen Green,47,NO,FEMALE,,5-May,2005,10,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Monkey_Joe_(Earth-616)#,,7,NO,MALE,,5-May,2005,10,Full,YES,NO,,,,,,,,,Killed by Leather Boy\rhttp://marvel.wikia.com/Doug_Taggert_(Earth-616)#,Doug Taggert,3,NO,MALE,,5-May,2005,10,Full,YES,NO,,,,,,,,,Accidently killed by Zaran\rhttp://marvel.wikia.com/Tippy-Toe_(Earth-616)#,,16,NO,FEMALE,,5-Jul,2005,10,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Wade_Wilson_(Earth-616)#,Wade Wilson,575,NO,MALE,,7-Sep,2007,8,Full,YES,NO,,,,,,,,,Died during incursion. It\'s not ever particulalry clear if he ever joined an Avengers team besides one issue so his inclusion here is specious at best. \rhttp://marvel.wikia.com/Greg_Willis_(Earth-616)#,Greg Willis,58,NO,MALE,,9-Aug,2009,6,Full,NO,,,,,,,,,,Died on Battleworld and came back prior to joining avengers\rhttp://marvel.wikia.com/Sersi_(Earth-616)#,Circe,237,NO,FEMALE,,Feb-90,1990,25,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Peter_Parker_(Earth-616)#,Peter Benjamin Parker,4333,YES,MALE,,Apr-90,1990,25,Full,YES,YES,YES,YES,,,,,,,Since joining the New Avengers: First death Killed by Morlun. Ressurected in a brand new body. Died in Amazing Spider-Man #700. Eventually mainfested in his body that Octavious stole and then took over again. \rhttp://marvel.wikia.com/Walter_Newell_(Earth-616)#,Walter Newell,126,NO,MALE,,Jul-90,1990,25,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Elvin_Haliday_(Earth-616)#,Elvin Haliday,158,NO,MALE,Feb-91,,1900,115,Probationary,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/William_Baker_(Earth-616)#,William Baker,355,NO,MALE,Feb-91,,1900,115,Probationary,YES,YES,NO,,,,,,,,Died in Identity_Disc_Vol_1_1. Later was revealed to be alive and working along with the Vulture.\rhttp://marvel.wikia.com/Crystalia_Amaquelin_(Earth-616)#,Crystal Amaquelin Maximoff,517,NO,FEMALE,Aug-91,Jan-92,1992,23,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Eric_Masterson_(Earth-616)#,Eric Kevin Masterson,202,NO,MALE,,Jan-92,1992,23,Full,YES,YES,YES,NO,,,,,,,Became posessed died to purge self of corruption. Revived as undead Avenger minion of the Grim Reaper. Returned to the spirit world by the Scarlet Witch. Has since stayed dead. \rhttp://marvel.wikia.com/Philip_Javert_(Earth-921)#,Phillip Javert,31,NO,MALE,,Dec-92,1992,23,Honorary,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Melissa_Darrow_(Earth-9201)#,,28,NO,FEMALE,,Jun-93,1993,22,Honorary,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Deathcry_(Earth-616)#,,50,NO,FEMALE,,Jul-93,1993,22,Honorary,YES,YES,YES,NO,,,,,,, Reduced her to a pile of bones and organsby Vargas. Brought back by Chaos King. Died second time in Chaos_War:_Dead_Avengers_Vol_1_3\rhttp://marvel.wikia.com/Anthony_Stark_(Earth-96020)#,Anthony Edward Stark,27,NO,MALE,,Feb-96,1996,19,Full,YES,NO,,,,,,,,,Merged with 616 Tony Stark\rhttp://marvel.wikia.com/Giuletta_Nefaria_(Masque)_(Earth-616)#,"""Giulietta Nefaria""",18,NO,FEMALE,,Apr-96,1996,19,Honorary,YES,NO,,,,,,,,,Whitney killed was a clone possibly this incarnation of Masque\'\rhttp://marvel.wikia.com/Vance_Astrovik_(Earth-616)#,Vance Astrovik,302,NO,MALE,,May-98,1998,17,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Angelica_Jones_(Earth-616)#,Angelica Jones,330,NO,FEMALE,,May-98,1998,17,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Delroy_Garrett_Jr._(Earth-616)#,Delroy Garrett Jr.,101,NO,MALE,,Apr-00,2000,15,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Maria_de_Guadalupe_Santiago_(Earth-616)#,Maria de Guadalupe Santiago,43,NO,FEMALE,,Jul-00,2000,15,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Jonathan_Hart_(Earth-616)#,Jonathan Hart,126,NO,MALE,,1-Aug,2001,14,Full,YES,YES,YES,YES,,,,,,,First died in a zero energy explosion in space. Body was apparently reanimated by Scarlet Witch. Abruptly exploded killing Scott Lang as well. Second return in Marvel_Zombies_Supreme_Vol_1_2. \rhttp://marvel.wikia.com/Kelsey_Leigh_(Earth-616)#,Kelsey Leigh Shorr,24,NO,FEMALE,,4-Jun,2004,11,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Luke_Cage_(Earth-616)#,Carl Lucas,886,YES,MALE,,5-Mar,2005,10,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Veranke_(Earth-616)#,Veranke,159,NO,FEMALE,,5-Mar,2005,10,Full,YES,NO,,,,,,,,,Killed by Norman Osbourne. Has not returned. \rhttp://marvel.wikia.com/James_Howlett_(Earth-616)#,"James ""Logan"" Howlett",3130,YES,MALE,,5-Jun,2005,10,Full,YES,NO,,,,,,,,,Died in Death_of_Wolverine_Vol_1_4. Has not yet returned\rhttp://marvel.wikia.com/Robert_Reynolds_(Earth-616)#,Robert Reynolds,241,NO,MALE,,5-Oct,2005,10,Full,YES,YES,,,,,,,,,Died during Seige. Brought back to life by the Apocalypse twins as a horseman of death in Uncanny_Avengers_Vol_1_9.\rhttp://marvel.wikia.com/Maya_Lopez_(Earth-616)#,Maya Lopez,67,NO,FEMALE,,5-Nov,2005,10,Full,YES,YES,YES,NO,,,,,,,"First death \'had been murdered in a fight with Elektra.\' ""Revived by Elektra and the Hand with their dark magic."" Died in most recent run of Moon Knight has not yet returned."\rhttp://marvel.wikia.com/Nathaniel_Richards_(Iron_Lad)_(Earth-6311)#,Nathaniel Richards,23,NO,MALE,,5-Apr,2005,10,Full,NO,,,,,,,,,,NA is actually Kang but let\'s just not go there.\rhttp://marvel.wikia.com/Elijah_Bradley_(Earth-616)#,Elijah Bradley,103,NO,MALE,,5-Apr,2005,10,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/William_Kaplan_(Earth-616)#,"William ""Billy"" Kaplan",123,YES,MALE,,5-Apr,2005,10,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Dorrek_VIII_(Earth-616)#,"Dorrek VIII/Theodore ""Teddy"" Altman",110,YES,MALE,,5-Apr,2005,10,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Cassandra_Lang_(Earth-616)#,Cassie Lang,160,NO,FEMALE,,5-Jun,2005,10,Full,YES,YES,,,,,,,,,Dies in Young Avengers: The Childrens Crusade when she is killed by Doom. Eventually reborn years later. \rhttp://marvel.wikia.com/Katherine_Bishop_(Earth-616)#,"Katherine ""Kate"" Bishop",132,YES,FEMALE,,5-Jun,2005,10,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Vision_(Jonas)_(Earth-616)#,Alias: Jonas,121,NO,MALE,,6-Feb,2006,9,Full,YES,NO,,,,,,,,,Dies in Young Avengers The Children\'s Crusade\rhttp://marvel.wikia.com/Thomas_Shepherd_(Earth-616)#,"Thomas ""Tommy"" Shepherd",59,YES,MALE,,6-Jun,2006,9,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Daniel_Rand_(Earth-616)#,Daniel Thomas Rand K\'ai,629,NO,MALE,,7-Feb,2007,8,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Stephen_Strange_(Earth-616)#,Doctor Stephen Vincent Strange,1324,NO,MALE,,7-Feb,2007,8,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Ares_(Earth-616)#,Ares,236,NO,MALE,,7-Mar,2007,8,Full,YES,YES,YES,NO,,,,,,,Ripped in half by the Sentry during Seige. Revived during Chaos King. Then killedby Chaos King. \rhttp://marvel.wikia.com/James_Buchanan_Barnes_(Earth-616)#,James Buchanan Barnes,663,NO,MALE,,8-Dec,2008,7,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Jessica_Drew_(Earth-616)#,Jessica Miriam Drew,525,YES,FEMALE,,8-Dec,2008,7,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Jessica_Jones_(Earth-616)#,Jessica Jones,205,YES,FEMALE,,10-Aug,2010,5,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Amadeus_Cho_(Earth-616)#,Amadeus Cho,108,NO,MALE,,9-Dec,2009,6,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Maria_Hill_(Earth-616)#,Maria Hill,359,YES,FEMALE,,10-May,2010,5,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Robert_Baldwin_(Earth-616)#,Robbie Baldwin,299,NO,MALE,,10-Jun,2010,5,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Sharon_Carter_(Earth-616)#,Sharon Carter,333,NO,FEMALE,,10-May,2010,5,Full,YES,YES,,,,,,,,,Died in Captain_America_Vol_7_10. It\'s not clear how she survived but yeah she totally did. \rhttp://marvel.wikia.com/Eric_O%27Grady_(Earth-616)#,Eric O\'Grady,88,NO,MALE,,10-May,2010,5,Full,YES,NO,,,,,,,,,Died in Secret_Avengers_Vol_1_22. \rhttp://marvel.wikia.com/Brunnhilde_(Earth-616)#,Brunnhilde,369,NO,FEMALE,,10-May,2010,5,Full,NO,,,,,,,,,,NA but died/came back during Ragnarok prior to becoming an Avenger\rhttp://marvel.wikia.com/Richard_Rider_(Earth-616)#,Richard Rider,380,NO,MALE,,10-May,2010,5,Full,YES,NO,,,,,,,,,Died in Guardians_of_the_Galaxy_Vol_3_20.\rhttp://marvel.wikia.com/Brian_Braddock_(Earth-616)#,Brian Braddock,545,NO,MALE,,10-May,2010,5,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Dennis_Sykes_(Earth-616)#,Dennis Sykes,6,NO,MALE,,10-Sep,2010,5,Full,YES,NO,,,,,,,,,Died in Heroic_Age:_One_Month_to_Live_Vol_1_5 stayed dead. \rhttp://marvel.wikia.com/Noh-Varr_(Earth-616)#,Noh-Varr,126,YES,MALE,,10-Jun,2010,5,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Thaddeus_Ross_(Earth-616)#,Thaddeus Ross,417,NO,MALE,,11-Jun,2011,4,Full,YES,YES,,,,,,,,,Died in Circle of Four arc. Made a deal with Mephisto and came back. \rhttp://marvel.wikia.com/John_Aman_(Earth-616)#,John Aman,31,YES,MALE,,10-Dec,2010,5,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Shang-Chi_(Earth-616)#,Shang-Chi,310,YES,MALE,,11-Apr,2011,4,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/James_Santini_(Earth-616)#,James Santini,40,YES,MALE,,,1900,115,Academy,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Emery_Schaub_(Earth-616)#,Emery Schaub,26,YES,MALE,,,1900,115,Academy,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Jeanne_Foucault_(Earth-616)#,Jeanne Foucault,63,YES,FEMALE,,10-Aug,2010,5,Academy,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Fiona_(Inhuman)_(Earth-616)#,Fiona,2,YES,FEMALE,,,1900,115,Academy,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Jennifer_Takeda_(Earth-616)#,Jennifer Takeda,73,YES,FEMALE,,10-Aug,2010,5,Academy,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Hollow_(Earth-616)#,Yvette,22,YES,FEMALE,,,1900,115,Academy,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Julie_Power_(Earth-616)#,Julie Power,153,YES,FEMALE,,,1900,115,Academy,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Alani_Ryan_(Earth-616)#,Alani Ryan,73,YES,FEMALE,,,1900,115,Academy,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Ken_Mack_(Earth-616)#,Ken Mack,59,NO,MALE,,10-Aug,2010,5,Academy,YES,NO,,,,,,,,,Died in Murderworld during Avengers Arena\rhttp://marvel.wikia.com/Humberto_Lopez_(Earth-616)#,Humberto Lopez,66,YES,MALE,,10-Aug,2010,5,Academy,YES,NO,,,,,,,,,Died in Murderworld during Avengers Arena\rhttp://marvel.wikia.com/Johnathon_Gallo_(Earth-616)#,Johnny Gallo,43,YES,MALE,,,1900,115,Academy,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Lyra_(Earth-8009)#,Lyra,55,YES,FEMALE,,,1900,115,Academy,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Anya_Corazon_(Earth-616)#,Anya Corazon,108,YES,FEMALE,,,1900,115,Academy,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Brandon_Sharpe_(Earth-616)#,Brandon Sharpe,64,YES,MALE,,10-Aug,2010,5,Academy,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Kevin_Masterson_(Earth-616)#,Kevin Masterson,62,YES,MALE,,,1900,115,Academy,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Michiko_Musashi_(Earth-616)#,Michiko Musashi,94,YES,FEMALE,,,1900,115,Academy,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Takashi_Matsuya_(Earth-616)#,Taki Matsuya,18,YES,MALE,,,1900,115,Academy,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Daisy_Johnson_(Earth-616)#,Daisy Johnson,81,NO,FEMALE,,12-Jan,2012,3,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Ororo_Munroe_(Earth-616)#,Ororo Munroe,1598,NO,FEMALE,,12-Jan,2012,3,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Matthew_Murdock_(Earth-616)#,Matt Murdock,1375,NO,MALE,,11-Nov,2011,4,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Eugene_Thompson_(Earth-616)#,Flash Thompson,746,YES,MALE,,12-Apr,2012,3,Honorary,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Otto_Octavius_(Earth-616)#,Otto Octavius,561,NO,MALE,,13-Jan,2013,2,Full,YES,NO,,,,,,,,,Died in Superior_Spider-Man_Vol_1_30. \rhttp://marvel.wikia.com/Alexander_Summers_(Earth-616)#,Alex Summers,592,YES,MALE,,12-Dec,2012,3,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Samuel_Guthrie_(Earth-616)#,Samuel Guthrie,679,YES,MALE,,13-Feb,2013,2,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Roberto_da_Costa_(Earth-616)#,Roberto da Costa,491,YES,MALE,,13-Feb,2013,2,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Eden_Fesi_(Earth-616)#,Eden Fesi,65,YES,MALE,,13-Feb,2013,2,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Captain_Universe_(Earth-616)#,,55,YES,MALE,,13-Feb,2013,2,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Isabel_Kane_(Earth-616)#,Izzy Kane,44,YES,FEMALE,,13-Feb,2013,2,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Marcus_Milton_(Earth-13034)#,Marcus Milton,65,YES,MALE,,13-Feb,2013,2,Full,YES,NO,,,,,,,,,Died in New_Avengers_Vol_3_32. \rhttp://marvel.wikia.com/Rogue_(Anna_Marie)_(Earth-616)#,Anna Marie,877,YES,FEMALE,,13-Apr,2013,2,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Shiro_Yoshida_(Earth-616)#,Shiro Yoshida,176,YES,MALE,,13-May,2013,2,Full,YES,YES,,,,,,,,,Died in Uncanny_Avengers_Vol_1_22. Returned when \'he was able to use his power to transform his body into a state of pure energy \'\rhttp://marvel.wikia.com/Ex_Nihilo_(Earth-616)#,,24,YES,MALE,,13-Oct,2013,2,Full,YES,NO,,,,,,,,,Died in New_Avengers_Vol_3_32\rhttp://marvel.wikia.com/Abyss_(Ex_Nihilo%27s)_(Earth-616)#,,25,YES,FEMALE,,13-Oct,2013,2,Full,YES,NO,,,,,,,,,Died in New_Avengers_Vol_3_32\rhttp://marvel.wikia.com/Nightmask_(Earth-616)#,Adam,35,YES,MALE,,13-Oct,2013,2,Full,YES,NO,,,,,,,,,Died in New_Avengers_Vol_3_32\rhttp://marvel.wikia.com/Kevin_Connor_(Earth-616)#,Kevin Kale Connor,44,YES,MALE,,13-Oct,2013,2,Full,YES,NO,,,,,,,,,Died in New_Avengers_Vol_3_32\rhttp://marvel.wikia.com/Sam_Alexander_(Earth-616)#,Sam Alexander,78,YES,MALE,,15-Feb,2015,0,Honorary,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/America_Chavez_(Earth-616)#,America Chavez,22,YES,FEMALE,,13-Jul,2013,2,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Loki_Laufeyson_(Ikol)_(Earth-616)#,Loki Laufeyson,77,NO,MALE,,13-Jul,2013,2,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/David_Alleyne_(Earth-616)#,David Alleyne,115,YES,MALE,,13-Sep,2013,2,Full,NO,,,,,,,,,,NA\r"http://marvel.wikia.com/Nicholas_Fury,_Jr._(Earth-616)#","Nicholas Fury, Jr., Marcus Johnson",77,YES,MALE,,13-Apr,2013,2,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Phillip_Coulson_(Earth-616)#,Phillip Coulson,69,YES,MALE,,13-Apr,2013,2,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Tony_Masters_(Earth-616)#,Tony Masters,173,NO,MALE,,13-May,2013,2,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Victor_Mancha_(Earth-616)#,Victor Mancha,75,YES,MALE,,13-Sep,2013,2,Full,YES,YES,,,,,,,,,Died in Avengers_A.I._Vol_1_4. Returned in Avengers_A.I._Vol_1_9. \rhttp://marvel.wikia.com/Monica_Chang_(Earth-616)#,Monica Chang,12,YES,FEMALE,,13-Sep,2013,2,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Doombot_(Avenger)_(Earth-616)#,,14,YES,MALE,,13-Sep,2013,2,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Alexis_(Earth-616)#,Alexis,13,YES,FEMALE,,13-Nov,2013,2,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Eric_Brooks_(Earth-616)#,Eric Brooks,198,YES,MALE,,13-Nov,2013,2,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Adam_Brashear_(Earth-616)#,Adam Brashear,29,YES,MALE,,14-Jan,2014,1,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Victor_Alvarez_(Earth-616)#,Victor Alvarez,45,YES,MALE,,14-Jan,2014,1,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Ava_Ayala_(Earth-616)#,Ava Ayala,49,YES,FEMALE,,14-Jan,2014,1,Full,NO,,,,,,,,,,NA\rhttp://marvel.wikia.com/Kaluu_(Earth-616)#,Kaluu,35,YES,MALE,,15-Jan,2015,0,Full,NO,,,,,,,,,,NA\n'

• JSON (popular, especially for data with more nuanced structure)
  • example: SEA_building_energy.json, via https://www.kaggle.com/city-of-seattle/sea-building-energy-benchmarking/version/8
• XML - highly structured, like JSON, more rigid; easier to specify schemas; decreasing in popularity
• (SQL) - traditional databases; generally not human-readable, but exportable to JSON or CSV