Python
Finding the Most Liquid Equity ETFs in each Category using Python
FabTrader
Article overview
Not all ETFs are created equal — especially when it comes to liquidity. While NSE provides a full ETF list, identifying the most actively traded ETF for a specific index requires quite a bit of manual cleanup. This small Python utility automates the process by filtering the ETF universe, mapping ETFs to their benchmarks, and surfacing the most liquid ETF in each category.
If you trade or invest in ETFs in India, you’ve probably run into this problem.
You want the most liquid ETF for a particular index — say Nifty 50, Nifty Next 50, Bank Nifty, or Midcap. Sounds simple, right? Not really.
When you download the ETF list from the NSE website, a few challenges appear immediately:
- The list contains all kinds of ETFs — equity, gold, silver, debt, liquid funds, IPO ETFs, etc.
- Many ETFs don’t have a clean category tag, making classification tricky.
- Some indices have multiple ETFs tracking the same benchmark, but liquidity varies widely.
- Manually cleaning and sorting this data becomes tedious and error-prone.
So I built a small Python utility to automate this process.
The goal is simple:
Automatically identify the most liquid equity ETF for each index category.
How the Script Works
The utility follows a simple pipeline.
1. Fetch ETF Data Directly from NSE
The script pulls the live ETF master list directly from the NSE API and loads it into a pandas dataframe. This gives us:
- ETF Symbol
- Underlying Index
- Price
- Volume
2. Filter Only Meaningful ETFs
Next, the script removes unwanted entries:
- ETFs with very low trading volume
- Debt / bond / G-sec / liquid ETFs
- Some edge-case ETFs that don’t fit the use case
This ensures we focus only on actively traded equity ETFs.
3. Estimate Liquidity
Instead of just using volume, the script calculates a simple turnover proxy:
Weight = Price × Volume
This helps rank ETFs based on actual traded value, not just share count.
4. Automatically Identify ETF Categories
This is the trickiest part. ETF names and underlying indices are often inconsistent, so the script uses a cross-reference dictionary to map underlying index names to standard categories such as:
- NIFTY 50
- NIFTY NEXT 50
- MIDCAP
- SMALLCAP
- BANK NIFTY
- PSU
- METAL
- PHARMA
- IT
- etc.
This mapping allows the script to automatically classify ETFs even when the naming isn’t clean.
5. Pick the Most Liquid ETF per Category
Finally, the script:
- Groups ETFs by category
- Ranks them by turnover (Price × Volume)
- Picks the top ETF in each category
The result is a clean list of the most liquid ETF for each benchmark. Exactly what we want.
Why This is Useful
This small utility can help with:
- ETF universe creation for trading strategies
- Algo trading instruments selection
- Avoiding illiquid ETFs
- Quickly identifying the dominant ETF for each index
Instead of manually checking dozens of ETFs, the script gives you a clean, ready-to-use list in seconds.
The Code
Below is the full Python script that performs the entire workflow.
# -------------------------------------------------------------------------
# FabTrader Algorithmic Trading - Tutorials
# -------------------------------------------------------------------------
# CONTACT:
# - Website: https://fabtrader.in
# - Email: [email protected]
#
# Usage: Educational Purposes & training use only. Not for commercial redistribution.
# ------------------------------------------------------------------------
import pandas as pd
import requests
pd.set_option("display.max_rows", None, "display.max_columns", None)
# Function that fetches ETF master from NSE website and filters only equity based etfs
df = pd.DataFrame()
try:
headers = {
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/85.0.4183.83 Safari/537.36',
'Upgrade-Insecure-Requests': "1",
"DNT": "1",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*,q=0.8",
'Accept-Language': 'en-US,en;q=0.9',
'Accept-Encoding': 'gzip, deflate',
'Connection': 'keep-alive'
}
session = requests.Session()
session.get("http://nseindia.com", headers=headers)
cookies = session.cookies.get_dict()
# print("Starting to download ETF List from NSE")
ref_url = 'https://www.nseindia.com/market-data/exchange-traded-funds-etf'
ref = requests.get(ref_url, headers=headers)
url = 'https://www.nseindia.com/api/etf'
response = session.get(url, headers=headers, cookies=ref.cookies.get_dict())
data = response.json() # Convert response to JSON
# Convert JSON data to a DataFrame
df = pd.DataFrame(data['data']) # Extract the main data list
if not df.empty:
df = df[['symbol', 'assets', 'open', 'high', 'low', 'ltP', 'qty']]
df.columns = ['ETF', 'Underlying', 'Open', 'High', 'Low', 'Close', 'Volume']
except Exception as e:
print("Error while extracting ETF List from NSE")
print(e)
# Process the downloaded NSE file
try:
if not df.empty:
df = df[df['Volume'] != '-'] # remove invalid rows from datasets
# ensure numeric conversion (invalid parsing becomes NaN)
df['Close'] = pd.to_numeric(df['Close'], errors='coerce')
df['Volume'] = pd.to_numeric(df['Volume'], errors='coerce')
# create new column for Weigh / Turnover
df['Weight'] = df['Close'] * df['Volume']
# Filter #1 : Filter all etfs that have less than 10K volume
df = df[df['Volume'] >= 10000]
# Filter #2 : Remove all debt/ipo related ETFs
filter_string = [
'Bond', 'GSEC', 'G-Sec', 'GILT', 'LIQ', 'Liquid', 'IPO' # Remove debt and other non-equity based etfs from list
]
for scanString in filter_string:
df = df[~df.Underlying.str.contains('|'.join([scanString]))]
df = df[~df.ETF.str.contains('|'.join([scanString]))]
# Filter #3 : Remove one-off / unwanted etfs from list
filter_etfs = [
'GROWWNET', 'INTERNET', 'SHARIABEES', 'GROWWNXT50'
]
df = df[~df['ETF'].isin(filter_etfs)]
# Cross-reference dictionary (search string -> value)
cross_ref = {
"Midcap 150": "MIDCAP",
"Nifty 100": "NIFTY 100",
"MIDCAP 100": "MIDCAP",
"Nifty 200": "NIFTY 200",
"Nifty 500": "NIFTY 500",
"Nifty 50": "NIFTY 50",
"Silver": "SILVER",
"Gold": "GOLD",
"S&P": "S&P",
"Next 50": "NIFTY NEXT 50",
"Junior": "NIFTY NEXT 50",
"Nifty50": "NIFTY 50",
"Nifty200": "NIFTY 200",
"Nifty Total Market Index": "NIFTY 500",
"Private Bank": "PRIVATE BANK",
"Oil & Gas": "OIL & GAS",
"PSU": "PSU",
"Metal": "METAL",
"MNC": "MNC",
"Consumption": "CONSUMPTION",
"Healthcare": "HEALTHCARE",
"Financial Services": "FIN SERVICES",
"FMCG": "FMCG",
"EV": "EV",
"Nifty Bank": "BANK NIFTY",
"Auto": "AUTO",
"Low-Volatility 30 ": "LOW VOL 30",
"Momentum 50": "MOMENTUM 50",
"Nasdaq": "NASDAQ",
"NYSE": "NASDAQ",
"Alpha 50": "NIFTY 50",
"Smallcap 250": "SMALLCAP",
"Realty": "REALTY",
"MidSmallcap400": "MID SMALL CAP",
"Low Vol 30": "LOW VOL 30",
"Infrastructure": "INFRA",
"Hang Seng": "HANG SENG",
"Midcap 150": "MIDCAP",
"CPSE": "PSU",
"PSE": "PSU",
"Commodity": "GOLD",
"Power": "POWER",
"Defence": "DEFENCE",
"Pharma": "PHARMA",
" IT": "IT",
"SENSEX": "SENSEX",
"Nifty Top 10 Equal Weight": "NIFTY 50",
"Insurance": "INSURANCE",
"Manufacturing": "MANUFACTURING",
"Digital": "DIGITAL",
"Capital Market": "NIFTY 500",
"NIFTY Growth": "NIFTY 500",
"50 TRI": "NIFTY 50",
"Nifty Top 15 Equal Weight": "NIFTY 50",
"BSE 200": "NIFTY 200",
"ESG": "ESG",
"Commodities": "GOLD",
"Infra": "INFRA",
"NIFTY100": "NIFTY 100",
"Tourism": "TOURISM",
"Midcap 50": "MIDCAP",
"Top 20 Equal Weight": "NIFTY 50",
"MSCI": "LARGE MID CAP",
"Nifty Smallcap 100 Index": "SMALLCAP",
"Nifty Index 50 Index": "NIFTY 50",
"Nifty Chemicals Index (TRI)": "CHEMICAL",
"Nifty Chemicals Index - TRI": "CHEMICAL",
"Nifty Energy": "ENERGY",
"Nifty Energy Index": "ENERGY"
}
# Function to map values
def get_benchmark(underlying):
for search_str, value in cross_ref.items():
if search_str.upper() in underlying.upper(): # substring match
return value
print("No associated Category found for underlying. Default value-Unknown-assigned : ", underlying)
# return underlying # fallback to original value if no match
return 'Unknown' # If no match found, mark as 'Unknown', later remove from list
# Apply function to create Category column
df["Category"] = df["Underlying"].apply(get_benchmark)
# Remove ETFs that do not have a valid Category
df = df[~df['Category'].isin(['Unknown'])]
# Retain only the top most ETFs in each underlying category based on highest weight
df = df.sort_values(['Category', 'Weight'], ascending=False).groupby('Category').head(1)
df.sort_values(by="Weight", ascending=False, inplace=True)
df.reset_index(inplace=True, drop=True)
# Retain only the top 30 ETFs from the final list based on highest weight / turnover
# df = df.head(35)
print(df)
eligible_etfs = df["ETF"].tolist()
print(eligible_etfs)
else:
print("Extracted NSE ETF File is empty")
except Exception as e:
print("Error while processing NSE ETF file")
print(e)
If you're building trading systems, ETF strategies, or screening tools, this small utility can save a surprising amount of time.
Sometimes the best tools are the simple ones that remove friction from everyday tasks.
Hope this helps.
More from Python
The Market’s Coiled Spring — Building the Momentum Squeeze Indicator in Python
Volatility doesn’t expand randomly — it contracts first. The Momentum Squeeze Indicator, popularized by LazyBear, is built on this simple but powerful...
Price Consolidation Boxes: Ranges, Breakouts, and Retests Using Python
Markets don’t trend most of the time — they pause, compress, and consolidate. Before every meaningful move up or down, price typically...
Learn Web Scraping with Python – A Practical Guide for Traders
Web scraping is an essential skill for traders and Python developers who want full control over their market data. In this practical...