# Capture AI crawlers on AWS CloudFront.

> Connect AWS CloudFront to Virent AI Crawler Analytics. Stream 100% sampled CloudFront request logs through Kinesis and a small Lambda consumer.

Canonical: https://virent.app/docs/integrations/aws-cloudfront
Last updated: 2026-10-19

Stream 100% sampled CloudFront request logs through Kinesis and a small Lambda consumer.

CloudFront real-time logs go to a Kinesis Data Stream, and a small Lambda function forwards page requests to Virent. Real-time delivery is best-effort; keep standard access logs for forensic reconciliation.

You need your site ID and server secret from **Settings → Connection** (see [Hosting integrations](/docs/integrations/overview#before-you-start)). In the dashboard, **Connect analytics → Hosting integrations → AWS CloudFront** shows these steps with your site ID, domain and secret filled in. Below, replace `<your-site-id>`, `<paste-your-server-secret>` and `example.com` with your own values.

### Set up with your coding agent

Paste this prompt into Claude Code, Codex or another coding agent. Replace the site ID and example.com with your own.

```text
Configure AWS CloudFront so example.com sends reliable AI crawler request telemetry to Virent.

Site ID: <your-site-id>
Use the full Virent server secret from the Connect dialog. Do not use the masked preview "vha_sk_••••" as the real value.

Use the current provider documentation: https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/real-time-logs.html

Steps:
1. Create the Kinesis destination: Create a Kinesis Data Stream in the same AWS Region where you will create the CloudFront real-time log configuration. Grant CloudFront permission to write to it.

2. Add the Lambda consumer: Create a Node.js 22 Lambda subscribed to the stream. Set VIRENT_INGEST_SECRET to the revealed connection secret and allow outbound HTTPS to https://virent.app/api/ingest/http.

const INGEST = "https://virent.app/api/ingest/http"
const SITE_ID = "<your-site-id>"

export const handler = async (event) => {
  const events = event.Records.flatMap(({ kinesis }) =>
    Buffer.from(kinesis.data, "base64")
      .toString("utf8")
      .trim()
      .split("\n")
      .filter(Boolean)
      .map((line) => line.split("\t"))
      .filter((fields) => ["GET", "HEAD"].includes(fields[3] || ""))
      .map((fields) => ({
        timestamp: new Date(Number(fields[0]) * 1000).toISOString(),
        client_ip: fields[1],
        status_code: Number(fields[2]),
        request_method: fields[3],
        hostname: fields[4],
        request_path: fields[5],
        query_string: null,
        user_agent: fields[7],
        referrer: null,
        content_type: "",
      }))
  )

  for (let offset = 0; offset < events.length; offset += 1000) {
    const response = await fetch(INGEST, {
      method: "POST",
      headers: {
        "content-type": "application/json",
        "x-api-key": process.env.VIRENT_INGEST_SECRET,
      },
      body: JSON.stringify(events.slice(offset, offset + 1000)),
    })

    if (!response.ok) throw new Error(`Virent ingest failed: ${response.status}`)
  }

  return { siteId: SITE_ID, submitted: events.length }
}

3. Configure 100% request logging: Create a CloudFront real-time log configuration with a 100% sampling rate and these fields in exactly this order. Associate it with every cache behavior that serves public pages.

timestamp	c-ip	sc-status	cs-method	cs-host	cs-uri-stem	cs-uri-query	cs(User-Agent)	cs(Referer)	x-edge-request-id

4. Activate and verify: Activate the distribution changes, request https://example.com/ with a supported AI crawler user agent, then confirm Virent receives the visit. CloudFront real-time delivery is best-effort, so use standard access logs for forensic reconciliation.

Keep the secret server-side, preserve 100% request coverage where the provider supports sampling, and verify both normal site delivery and a crawler request after activation.
```

## Steps

1. **Create the Kinesis destination**: Create a Kinesis Data Stream in the same AWS Region where you will create the CloudFront real-time log configuration. Grant CloudFront permission to write to it.

2. **Add the Lambda consumer**: Create a Node.js 22 Lambda subscribed to the stream. Set VIRENT_INGEST_SECRET to the revealed connection secret and allow outbound HTTPS to https://virent.app/api/ingest/http.

   ```javascript
   const INGEST = "https://virent.app/api/ingest/http"
   const SITE_ID = "<your-site-id>"

   export const handler = async (event) => {
     const events = event.Records.flatMap(({ kinesis }) =>
       Buffer.from(kinesis.data, "base64")
         .toString("utf8")
         .trim()
         .split("\n")
         .filter(Boolean)
         .map((line) => line.split("\t"))
         .filter((fields) => ["GET", "HEAD"].includes(fields[3] || ""))
         .map((fields) => ({
           timestamp: new Date(Number(fields[0]) * 1000).toISOString(),
           client_ip: fields[1],
           status_code: Number(fields[2]),
           request_method: fields[3],
           hostname: fields[4],
           request_path: fields[5],
           query_string: null,
           user_agent: fields[7],
           referrer: null,
           content_type: "",
         }))
     )

     for (let offset = 0; offset < events.length; offset += 1000) {
       const response = await fetch(INGEST, {
         method: "POST",
         headers: {
           "content-type": "application/json",
           "x-api-key": process.env.VIRENT_INGEST_SECRET,
         },
         body: JSON.stringify(events.slice(offset, offset + 1000)),
       })

       if (!response.ok) throw new Error(`Virent ingest failed: ${response.status}`)
     }

     return { siteId: SITE_ID, submitted: events.length }
   }
   ```

3. **Configure 100% request logging**: Create a CloudFront real-time log configuration with a 100% sampling rate and these fields in exactly this order. Associate it with every cache behavior that serves public pages.

   ```text
   timestamp	c-ip	sc-status	cs-method	cs-host	cs-uri-stem	cs-uri-query	cs(User-Agent)	cs(Referer)	x-edge-request-id
   ```

4. **Activate and verify**: Activate the distribution changes, request https://example.com/ with a supported AI crawler user agent, then confirm Virent receives the visit. CloudFront real-time delivery is best-effort, so use standard access logs for forensic reconciliation.

Check AWS CloudFront's [current documentation](https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/real-time-logs.html) if its screens differ from these steps.

## Verify

Send a [test crawler request](/docs/analytics/ai-crawlers#verify) to your site. The setup dialog in Virent shows "Virent is receiving AI crawler traffic" once the first request arrives.