This tutorial explains how to fetch logs in cloudflare using its GraphQL API.
We generally check logs to analyze security threats and performance issues. For example, you can identify the IP addresses of users who are performing web scraping on your website.
You need an API key and a zone ID to authenticate and use the API. You can follow the instructions below to get API Key and Zone ID.
Instructions : You can find your API key in the Cloudflare dashboard by clicking on profile icon (top-right) > My Profile > API Tokens. Either you can use global API key or you can create a custom API token with specific permissions. The Zone ID is available on the Overview page of your site in Cloudflare.
The following python code returns the logs of your website from the past 24 hours and save it to Excel file in your current working directory.
import requests
from datetime import datetime, timedelta, timezone
import pandas as pd
from tzlocal import get_localzone
end_hour = datetime.now(timezone.utc)
start_hour = end_hour - timedelta(hours=24)
# Format with explicit UTC designation
def iso_format(dt):
return dt.strftime("%Y-%m-%dT%H:%M:%SZ")
# Define the Cloudflare API credentials
API_KEY = "xxxxxxxxxxxxx"
ZONE_ID = "xxxxxxxxxxxxxx"
API_EMAIL = "xxxxxxxxx@xxxx.com"
# Set up headers
headers = {
"X-Auth-Email": API_EMAIL,
"X-Auth-Key": API_KEY,
"Content-Type": "application/json"
}
# Define the new GraphQL query with the required fields
query = """
query GetAnalytics($zoneTag: String!, $datetime: String!) {
viewer {
zones(filter: { zoneTag: $zoneTag }) {
httpRequestsAdaptive(
filter: $filter
limit: 3000
) {
datetime
cacheStatus
clientASNDescription
clientAsn
clientCountryName
clientDeviceType
clientIP
clientRequestHTTPHost
clientRequestHTTPMethodName
clientRequestHTTPProtocol
clientRequestPath
clientRequestQuery
clientRequestScheme
clientSSLProtocol
edgeResponseStatus
leakedCredentialCheckResult
originResponseDurationMs
requestSource
securityAction
securitySource
userAgent
}
}
}
}
"""
# Create the request body with the variables
payload = {
"query": query,
"variables": {
"zoneTag": ZONE_ID,
"filter": {
"datetime_geq": iso_format(start_hour),
"datetime_leq": iso_format(end_hour)
}
}
}
# Make the API request
response = requests.post(
url="https://api.cloudflare.com/client/v4/graphql",
headers=headers,
json=payload
)
# Process the response
if response.status_code >= 400:
raise Exception(f"Error: {response.status_code}, {response.text}")
data = response.json()
http_requests = data['data']['viewer']['zones'][0]['httpRequestsAdaptive']
# Convert the data into a pandas DataFrame
df = pd.json_normalize(http_requests)
df['datetime'] = pd.to_datetime(df['datetime']).dt.tz_localize(None)
local_timezone = get_localzone()
df['datetime'] = df['datetime'].dt.tz_localize('UTC').dt.tz_convert(local_timezone).dt.tz_localize(None)
df = df.sort_values(by='datetime', ascending=False)
# Write the DataFrame to an Excel file
df.to_excel('cloudflare_logs.xlsx', index=False)
It returns the following 21 columns in an Excel file named "cloudflare_logs".
Deepanshu founded ListenData with a simple objective - Make analytics easy to understand and follow. He has over 10 years of experience in data science. During his tenure, he worked with global clients in various domains like Banking, Insurance, Private Equity, Telecom and HR.
Share Share Tweet