Capture AI crawlers on AWS CloudFront.

CloudFront real-time logs go to a Kinesis Data Stream, and a small Lambda function forwards page requests to Virent. Real-time delivery is best-effort; keep standard access logs for forensic reconciliation.

You need your site ID and server secret from Settings → Connection (see Hosting integrations). In the dashboard, Connect analytics → Hosting integrations → AWS CloudFront shows these steps with your site ID, domain and secret filled in. Below, replace <your-site-id>, <paste-your-server-secret> and example.com with your own values.

Configure AWS CloudFront so example.com sends reliable AI crawler request telemetry to Virent.

Site ID: <your-site-id>
Use the full Virent server secret from the Connect dialog. Do not use the masked preview "vha_sk_••••" as the real value.

Use the current provider documentation: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/real-time-logs.html

Steps:
1. Create the Kinesis destination: Create a Kinesis Data Stream in the same AWS Region where you will create the CloudFront real-time log configuration. Grant CloudFront permission to write to it.

2. Add the Lambda consumer: Create a Node.js 22 Lambda subscribed to the stream. Set VIRENT_INGEST_SECRET to the revealed connection secret and allow outbound HTTPS to https://virent.app/api/ingest/http.

const INGEST = "https://virent.app/api/ingest/http"
const SITE_ID = "<your-site-id>"

export const handler = async (event) => {
  const events = event.Records.flatMap(({ kinesis }) =>
    Buffer.from(kinesis.data, "base64")
      .toString("utf8")
      .trim()
      .split("\n")
      .filter(Boolean)
      .map((line) => line.split("\t"))
      .filter((fields) => ["GET", "HEAD"].includes(fields[3] || ""))
      .map((fields) => ({
        timestamp: new Date(Number(fields[0]) * 1000).toISOString(),
        client_ip: fields[1],
        status_code: Number(fields[2]),
        request_method: fields[3],
        hostname: fields[4],
        request_path: fields[5],
        query_string: null,
        user_agent: fields[7],
        referrer: null,
        content_type: "",
      }))
  )

  for (let offset = 0; offset < events.length; offset += 1000) {
    const response = await fetch(INGEST, {
      method: "POST",
      headers: {
        "content-type": "application/json",
        "x-api-key": process.env.VIRENT_INGEST_SECRET,
      },
      body: JSON.stringify(events.slice(offset, offset + 1000)),
    })

    if (!response.ok) throw new Error(`Virent ingest failed: ${response.status}`)
  }

  return { siteId: SITE_ID, submitted: events.length }
}

3. Configure 100% request logging: Create a CloudFront real-time log configuration with a 100% sampling rate and these fields in exactly this order. Associate it with every cache behavior that serves public pages.

timestamp	c-ip	sc-status	cs-method	cs-host	cs-uri-stem	cs-uri-query	cs(User-Agent)	cs(Referer)	x-edge-request-id

4. Activate and verify: Activate the distribution changes, request https://example.com/ with a supported AI crawler user agent, then confirm Virent receives the visit. CloudFront real-time delivery is best-effort, so use standard access logs for forensic reconciliation.

Keep the secret server-side, preserve 100% request coverage where the provider supports sampling, and verify both normal site delivery and a crawler request after activation.

Steps#

1. 01

   Create the Kinesis destination

   Create a Kinesis Data Stream in the same AWS Region where you will create the CloudFront real-time log configuration. Grant CloudFront permission to write to it.

2. 02

   Add the Lambda consumer

   Create a Node.js 22 Lambda subscribed to the stream. Set VIRENT_INGEST_SECRET to the revealed connection secret and allow outbound HTTPS to https://virent.app/api/ingest/http.

   index.mjs

   const INGEST = "https://virent.app/api/ingest/http"const SITE_ID = "<your-site-id>"export const handler = async (event) => {  const events = event.Records.flatMap(({ kinesis }) =>    Buffer.from(kinesis.data, "base64")      .toString("utf8")      .trim()      .split("\n")      .filter(Boolean)      .map((line) => line.split("\t"))      .filter((fields) => ["GET", "HEAD"].includes(fields[3] || ""))      .map((fields) => ({        timestamp: new Date(Number(fields[0]) * 1000).toISOString(),        client_ip: fields[1],        status_code: Number(fields[2]),        request_method: fields[3],        hostname: fields[4],        request_path: fields[5],        query_string: null,        user_agent: fields[7],        referrer: null,        content_type: "",      }))  )  for (let offset = 0; offset < events.length; offset += 1000) {    const response = await fetch(INGEST, {      method: "POST",      headers: {        "content-type": "application/json",        "x-api-key": process.env.VIRENT_INGEST_SECRET,      },      body: JSON.stringify(events.slice(offset, offset + 1000)),    })    if (!response.ok) throw new Error(`Virent ingest failed: ${response.status}`)  }  return { siteId: SITE_ID, submitted: events.length }}

3. 03

   Configure 100% request logging

   Create a CloudFront real-time log configuration with a 100% sampling rate and these fields in exactly this order. Associate it with every cache behavior that serves public pages.

   CloudFront real-time log fields

   timestamp	c-ip	sc-status	cs-method	cs-host	cs-uri-stem	cs-uri-query	cs(User-Agent)	cs(Referer)	x-edge-request-id

4. 04

   Activate and verify

   Activate the distribution changes, request https://example.com/ with a supported AI crawler user agent, then confirm Virent receives the visit. CloudFront real-time delivery is best-effort, so use standard access logs for forensic reconciliation.

Check AWS CloudFront's current documentation if its screens differ from these steps.

Verify#

Send a test crawler request to your site. The setup dialog in Virent shows "Virent is receiving AI crawler traffic" once the first request arrives.