Lane detection is one of those problems that looks simple until you actually try it. Shadows, worn markings, curved roads, rain – the real world fights you at every step. The good news: you can get surprisingly far with classical OpenCV methods, and when those break down, YOLOv8 segmentation picks up the slack.

Here is the fastest path to a working pipeline. We start with the classical approach, then layer in deep learning.

1

pip install opencv-python-headless numpy ultralytics

Quick Start: Edge Detection and Hough Lines

This gets you lane lines on a single frame in under 20 lines of code:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

import cv2
import numpy as np

frame = cv2.imread("road.jpg")
gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
blur = cv2.GaussianBlur(gray, (5, 5), 0)
edges = cv2.Canny(blur, 50, 150)

height, width = edges.shape
mask = np.zeros_like(edges)
polygon = np.array([[
    (int(width * 0.1), height),
    (int(width * 0.45), int(height * 0.6)),
    (int(width * 0.55), int(height * 0.6)),
    (int(width * 0.9), height)
]], np.int32)
cv2.fillPoly(mask, polygon, 255)
roi = cv2.bitwise_and(edges, mask)

lines = cv2.HoughLinesP(roi, rho=1, theta=np.pi / 180, threshold=50,
                         minLineLength=40, maxLineGap=150)

line_image = np.zeros_like(frame)
if lines is not None:
    for line in lines:
        x1, y1, x2, y2 = line[0]
        cv2.line(line_image, (x1, y1), (x2, y2), (0, 255, 0), 5)

result = cv2.addWeighted(frame, 0.8, line_image, 1.0, 0)
cv2.imwrite("lane_detected.jpg", result)

That works. But it produces a mess of overlapping line segments instead of clean lane boundaries. Let us fix that.

Building the Classical Pipeline

The classical approach has five stages: grayscale conversion, Gaussian blur, Canny edge detection, region-of-interest masking, and Hough line detection. We already saw these above. The missing piece is turning raw Hough segments into two clean lane lines.

Separating Left and Right Lanes

Each Hough segment has a slope. Negative slope (in image coordinates where y increases downward) means left lane, positive means right lane. Filter out nearly horizontal lines – those are noise.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

def separate_lines(lines, height):
    left_points = []
    right_points = []

    if lines is None:
        return left_points, right_points

    for line in lines:
        x1, y1, x2, y2 = line[0]
        if x2 == x1:
            continue
        slope = (y2 - y1) / (x2 - x1)
        if abs(slope) < 0.5:
            continue  # skip near-horizontal lines
        if slope < 0:
            left_points.extend([(x1, y1), (x2, y2)])
        else:
            right_points.extend([(x1, y1), (x2, y2)])

    return left_points, right_points

Fitting Lane Lines with Polynomial Regression

Raw Hough segments are noisy. Fitting a first-degree polynomial through all the points on each side gives you a single clean line. For curved roads, bump it up to second degree.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

def fit_lane_line(points, height, degree=1):
    if len(points) < 2:
        return None

    points = np.array(points)
    xs = points[:, 0]
    ys = points[:, 1]

    coeffs = np.polyfit(ys, xs, degree)
    poly = np.poly1d(coeffs)

    y_start = height
    y_end = int(height * 0.6)
    y_vals = np.linspace(y_start, y_end, num=50).astype(int)
    x_vals = poly(y_vals).astype(int)

    return list(zip(x_vals, y_vals))

We fit x as a function of y (not the other way around) because lane lines are nearly vertical in image space. Fitting y = f(x) would blow up when the line is steep.

Drawing the Lane Overlay

Once you have left and right lane points, fill the area between them with a translucent green polygon:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

def draw_lane_overlay(frame, left_lane, right_lane):
    overlay = frame.copy()

    if left_lane is None or right_lane is None:
        return frame

    lane_polygon = np.array(
        left_lane + right_lane[::-1],
        dtype=np.int32
    )
    cv2.fillPoly(overlay, [lane_polygon], (0, 200, 0))
    return cv2.addWeighted(frame, 0.7, overlay, 0.3, 0)

Putting It All Together on Video

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41

def process_frame(frame):
    height, width = frame.shape[:2]
    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    blur = cv2.GaussianBlur(gray, (5, 5), 0)
    edges = cv2.Canny(blur, 50, 150)

    mask = np.zeros_like(edges)
    polygon = np.array([[
        (int(width * 0.1), height),
        (int(width * 0.45), int(height * 0.6)),
        (int(width * 0.55), int(height * 0.6)),
        (int(width * 0.9), height)
    ]], np.int32)
    cv2.fillPoly(mask, polygon, 255)
    roi = cv2.bitwise_and(edges, mask)

    lines = cv2.HoughLinesP(roi, rho=1, theta=np.pi / 180, threshold=50,
                             minLineLength=40, maxLineGap=150)

    left_pts, right_pts = separate_lines(lines, height)
    left_lane = fit_lane_line(left_pts, height)
    right_lane = fit_lane_line(right_pts, height)

    return draw_lane_overlay(frame, left_lane, right_lane)


cap = cv2.VideoCapture("driving_video.mp4")
fourcc = cv2.VideoWriter_fourcc(*"mp4v")
out = cv2.VideoWriter("output.mp4", fourcc, 30.0,
                       (int(cap.get(cv2.CAP_PROP_FRAME_WIDTH)),
                        int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))))

while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break
    result = process_frame(frame)
    out.write(result)

cap.release()
out.release()

This classical pipeline runs at hundreds of FPS on a CPU. It handles clear highway footage well. It falls apart on curved roads, poor lighting, and missing lane markings. That is where deep learning comes in.

YOLOv8 Segmentation for Lane Detection

YOLOv8-seg outputs pixel-level segmentation masks. The pretrained COCO model does not include a “lane” class, but you can fine-tune it on a lane dataset like TuSimple, CULane, or BDD100K. Even without fine-tuning, the road segmentation mask from a model trained on driving data gives you the drivable area, which you can post-process into lane boundaries.

Here is the inference workflow with a fine-tuned or custom-trained YOLOv8-seg model:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40

from ultralytics import YOLO
import cv2
import numpy as np

model = YOLO("yolov8n-seg.pt")  # swap for your fine-tuned weights

cap = cv2.VideoCapture("driving_video.mp4")

while cap.isOpened():
    ret, frame = cap.read()
    if not ret:
        break

    results = model(frame, verbose=False)

    for result in results:
        if result.masks is None:
            continue

        masks = result.masks.data.cpu().numpy()
        classes = result.boxes.cls.cpu().numpy().astype(int)

        for mask, cls_id in zip(masks, classes):
            # Filter for your lane/road class index
            # With a lane-trained model, check for your specific lane class ID
            binary_mask = (mask > 0.5).astype(np.uint8)
            binary_mask = cv2.resize(binary_mask,
                                      (frame.shape[1], frame.shape[0]),
                                      interpolation=cv2.INTER_NEAREST)

            colored_mask = np.zeros_like(frame)
            colored_mask[binary_mask == 1] = (0, 200, 0)
            frame = cv2.addWeighted(frame, 0.7, colored_mask, 0.3, 0)

    cv2.imshow("Lane Detection", frame)
    if cv2.waitKey(1) & 0xFF == ord("q"):
        break

cap.release()
cv2.destroyAllWindows()

Training YOLOv8-seg on Lane Data

If you have lane annotations in YOLO segmentation format (polygon points per image), training is straightforward:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

from ultralytics import YOLO

model = YOLO("yolov8n-seg.pt")

model.train(
    data="lane_dataset.yaml",
    epochs=100,
    imgsz=640,
    batch=16,
    name="lane_seg_model",
)

Your lane_dataset.yaml needs train and val image paths plus a names dictionary mapping class indices to labels like lane_marking or drivable_area. The TuSimple dataset is a solid starting point – it has 6,408 annotated clips of US highway driving.

Overlaying Detected Lanes on Video Frames

The strongest pipeline uses both approaches. Run YOLOv8-seg for drivable area detection, then apply classical Hough lines within that mask to find precise lane boundaries. This gives you the reliability of deep learning with the geometric precision of traditional CV:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29

def hybrid_lane_detection(frame, model):
    results = model(frame, verbose=False)
    road_mask = np.zeros(frame.shape[:2], dtype=np.uint8)

    for result in results:
        if result.masks is None:
            continue
        masks = result.masks.data.cpu().numpy()
        for mask in masks:
            binary = (mask > 0.5).astype(np.uint8)
            binary = cv2.resize(binary,
                                 (frame.shape[1], frame.shape[0]),
                                 interpolation=cv2.INTER_NEAREST)
            road_mask = cv2.bitwise_or(road_mask, binary)

    gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
    blur = cv2.GaussianBlur(gray, (5, 5), 0)
    edges = cv2.Canny(blur, 50, 150)
    masked_edges = cv2.bitwise_and(edges, edges, mask=road_mask)

    lines = cv2.HoughLinesP(masked_edges, rho=1, theta=np.pi / 180,
                             threshold=50, minLineLength=40, maxLineGap=150)

    height = frame.shape[0]
    left_pts, right_pts = separate_lines(lines, height)
    left_lane = fit_lane_line(left_pts, height)
    right_lane = fit_lane_line(right_pts, height)

    return draw_lane_overlay(frame, left_lane, right_lane)

For temporal smoothing across frames, keep a running average of the polynomial coefficients. This eliminates jitter without adding much latency:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

from collections import deque

class LaneSmoother:
    def __init__(self, window=5):
        self.left_history = deque(maxlen=window)
        self.right_history = deque(maxlen=window)

    def smooth(self, left_pts, right_pts, height):
        left_lane = fit_lane_line(left_pts, height)
        right_lane = fit_lane_line(right_pts, height)

        if left_lane is not None:
            self.left_history.append(np.array(left_lane))
        if right_lane is not None:
            self.right_history.append(np.array(right_lane))

        avg_left = None
        avg_right = None
        if self.left_history:
            avg_left = np.mean(self.left_history, axis=0).astype(int).tolist()
            avg_left = [tuple(p) for p in avg_left]
        if self.right_history:
            avg_right = np.mean(self.right_history, axis=0).astype(int).tolist()
            avg_right = [tuple(p) for p in avg_right]

        return avg_left, avg_right

Use this smoother in your video loop by replacing the direct fit_lane_line calls with smoother.smooth(). A window of 5 frames works well at 30 FPS – it gives you about 165ms of smoothing, which is enough to kill flicker without introducing visible lag.

Common Errors and Fixes

Canny detects too many or too few edges. The two threshold parameters control sensitivity. Start with (50, 150) and adjust. A 1:3 ratio between the low and high threshold is a reasonable default. If you get too much noise, raise both. If lanes disappear, lower them.

HoughLinesP returns None. This happens when no line segments pass the threshold. Lower the threshold parameter or reduce minLineLength. Also check that your ROI mask actually covers the lane area – save the mask image with cv2.imwrite("debug_mask.png", roi) and visually inspect it.

polyfit raises “poorly conditioned” warnings. You have too few points or they are nearly collinear in a degenerate way. Add a check: only call np.polyfit when you have at least 4 points. Wrap it in a try/except and fall back to the previous frame’s lane line:

1
2
3
4

try:
    coeffs = np.polyfit(ys, xs, degree)
except np.RankWarning:
    return previous_lane  # reuse last good detection

YOLOv8 mask resolution does not match frame size. YOLOv8 outputs masks at the model’s internal resolution (often 160x160). Always resize with cv2.resize() and cv2.INTER_NEAREST interpolation to preserve hard mask boundaries. Bilinear interpolation creates soft edges that mess up downstream thresholding.

Lane overlay flickers between frames. Use the LaneSmoother class from the previous section. A window of 5-10 frames eliminates jitter. If lanes still jump around, increase the minLineLength parameter in HoughLinesP to reject short spurious segments.

Video output is empty or corrupted. Make sure the VideoWriter codec matches your system. On Linux, mp4v works with .mp4 files. On macOS, try avc1. Always verify that cap.isOpened() returns True before entering the loop, and check that the frame dimensions you pass to VideoWriter match what VideoCapture actually returns.

CUDA out of memory with YOLOv8. Resize frames before inference with cv2.resize(frame, (640, 480)), or force CPU inference by setting the device: results = model(frame, device="cpu", verbose=False). The nano model (yolov8n-seg) is your best bet for constrained hardware – it runs at 40+ FPS on a modern CPU.