The Open Data Layer: Free APIs With Surprising Depth

Before you sign up for another data API subscription, spend an hour in the places most engineers never look.

The FDA publishes every drug adverse event report filed since 2004 as a free, structured, queryable API. The Met Museum serves high-resolution images and metadata for 490,000 artworks with no key required. NOAA has global climate records going back 150 years. None of this costs anything.

The data was never scarce. The attention was.

The Institutional Data Layer

Governments and academic institutions have been publishing structured data for decades. Not as a product strategy. As a byproduct of doing their jobs.

The data exists because the FDA has a statutory obligation to track adverse events. NOAA has to track weather. The Census Bureau has to count people. The research papers indexed by PubMed were funded with public money and required to be accessible. None of these organizations built an API to compete in a market. They built one because the data was already there and making it accessible was the right thing to do.

That origin is what makes it valuable. It is not crowd-sourced, not scraped, not synthetic. It is primary-source data produced by organizations with legal obligations to collect and maintain it accurately.

The scale is also not what you would expect. data.gov indexes over 526,000 datasets across every federal agency. The World Bank publishes development indicators covering 200+ countries going back decades. CrossRef maintains metadata for 130 million scholarly publications, each with a DOI, citation links, author data, and journal provenance. You can query all of it with a GET request and no API key.

Five Boring Sources With Surprising Depth

Most engineers reach for whatever comes up first in a search for a given data category. The sources below rarely come up first. They should.

OpenFDA

The FDA's open API covers drug adverse events, medical device malfunction reports, food recalls, and drug labeling. Every adverse event report a clinician or patient has filed about a medication since 2004 is in there. The database has over 20 million records. You can query by drug name, patient demographics, reaction type, or reporter country. No key, no account, no terms negotiation.

The device malfunctions endpoint is particularly interesting. Every report of a pacemaker, infusion pump, or surgical robot failing in the field is in there, with manufacturer response and resolution data. This is the data that drives regulatory decisions and it is free.

USDA FoodData Central

The USDA maintains the most comprehensive food composition database in existence. Chemical analysis data for over 600,000 foods: amino acids, fatty acids, vitamins, minerals, phytochemicals, everything measured in a lab. The API is free, the data is methodologically documented, and the records go back to foundation-level nutrient surveys from the 1970s.

If you have ever needed food data and ended up paying for a nutrition API, you were probably paying for a wrapper around this dataset.

The Met Museum API

The Metropolitan Museum of Art in New York serves metadata and high-resolution images for over 490,000 works in its collection. No authentication. No rate limit published anywhere in the documentation. Department IDs, artist provenance, accession dates, classification, culture, medium, dimensions, image URLs all included in each record.

The Harvard Art Museums API and the Rijksmuseum API follow the same pattern. Major institutions have quietly built excellent APIs that almost nobody outside the digital humanities community seems to know about.

Open-Meteo

A European weather API with no authentication, no API key, and no documented usage limits. Hourly weather forecasts for any coordinate on earth, historical data back to 1940, soil temperature, ocean wave data, flood modeling outputs. The models it runs on are the same ensemble models used by national weather services.

Most developers I have talked to reach for OpenWeather or a similar service first. Open-Meteo is more capable, completely free for non-commercial use, and has no friction to start.

PubMed and CrossRef

PubMed indexes over 36 million biomedical citations with abstracts, MeSH subject terms, author affiliations, funding sources, and citation links. The NIH has been building this database since 1996. The Entrez API gives you programmatic access to all of it.

CrossRef adds another 130 million scholarly DOI records across all disciplines. You can pull metadata for any paper with a DOI: title, authors, journal, publication date, references, citation count. This is the infrastructure that Semantic Scholar, Elicit, and most AI research tools are built on.

The Sources Everyone Knows but Underuses

Some of the deepest free data sources are famous but used shallowly. Most developers hit them for toy projects and move on without exploring what they actually contain.

Wikipedia's Action API is not just for fetching article text. It exposes page links, backlinks, language variants, revision history, talk page structure, editor contribution data, and the full Wikidata entity graph underneath it. Wikidata alone has 110 million structured items with multilingual labels, aliases, and typed relationships between entities.

NASA has 20+ public API endpoints: the Astronomy Picture of the Day, Mars Rover photos, Earth Observatory imagery, near-Earth object tracking, satellite telemetry from the Earthdata system, and the full NASA image and video library. Most developers have used APOD. Very few have explored the Earth observing system APIs, which give access to satellite imagery and atmospheric measurements updated daily.

The HackerNews API is a real-time firehose via Firebase: every post, comment, score, and timestamp since 2006, with live updates. If you want to understand how a technology topic has evolved over the past decade in the technical community, this is a usable primary source.

What This Unlocks

None of this is new. OpenFDA launched in 2012. The Met API launched in 2018. NOAA has been publishing climate data freely since the early days of the public internet. PubMed has been queryable via API since 2004.

The reason engineers reach for paid subscriptions first is partly habit, partly discoverability. Institutional APIs do not have marketing budgets. They do not rank well for generic searches like "weather API" or "food data API." The signup friction is lower for commercial services because they have invested in that experience.

But the data quality, depth, and permanence of institutional sources is usually higher. The FDA is not pivoting its adverse event database. NOAA is not deprecating its climate archive. These APIs have a different risk profile than a startup's free tier.

If you are building something that requires external data, map the institutional layer first. The free data layer is denser than most people think. Start there.