KDEAmbi/lessons_learned.md

Image Transformation Algorithm (HyperionGrabber)

The HyperionGrabber is responsible for acquiring video frames, transforming them, and sending them to the Hyperion server (via HyperionClient). The core image transformation logic is found in the _processFrame method of hyperiongrabber.cpp.

Algorithm:

Function _processFrame(input_frame: QVideoFrame):
    // 1. Validate the input frame
    If NOT input_frame.isValid():
        Return

    // 2. Increment frame counter and apply frame skipping if configured
    _frameCounter_m = _frameCounter_m + 1
    If _frameskip_m > 0 AND (_frameCounter_m MOD (_frameskip_m + 1) != 0):
        Return

    // 3. Convert QVideoFrame to QImage
    image = input_frame.toImage()
    If image.isNull():
        Log "Failed to convert QVideoFrame to QImage."
        Return

    // 4. Calculate target size for scaling based on _scale_m (default 8)
    target_width = image.width() / _scale_m
    target_height = image.height() / _scale_m
    target_size = QSize(target_width, target_height)
    If NOT target_size.isValid():
        Log "Invalid target size for scaling."
        Return

    // 5. Scale the image using Qt's optimized scaling with smooth transformation
    scaled_image = image.scaled(target_size, Qt::IgnoreAspectRatio, Qt::SmoothTransformation)

    // 6. Convert image format to RGB888 if necessary (24-bit RGB)
    If scaled_image.format() IS NOT QImage::Format_RGB888:
        scaled_image = scaled_image.convertToFormat(QImage::Format_RGB888)

    // 7. Send the processed image dimensions and data to the Hyperion client
    _hclient_p.setImgSize(scaled_image.width(), scaled_image.height())
    _hclient_p.sendImage(scaled_image.constBits(), scaled_image.sizeInBytes())

End Function

New Strategy: Direct WLED Communication

Goal: Modify the Hyperion_Grabber_X11_QT application to send image data directly to a WLED device, bypassing the Hyperion.ng server.

Current Understanding:

• The HyperionGrabber successfully captures and processes screen images (scaling, format conversion).
• The HyperionClient currently sends these processed images to a Hyperion.ng server using JSON-RPC over TCP.

Proposed Plan:

1. Research WLED API for Image Data:

   • WLED supports several UDP protocols for live image data, including DDP (Distributed Display Protocol) on UDP port 4048 and WLED UDP Realtime (WARLS) on UDP port 21324.
   • DDP appears to be a good fit for sending raw RGB pixel data efficiently.

2. Develop a WLED Client (WledClient):

   • Created a new C++ class (WledClient) that uses QUdpSocket to send data.
   • Implemented the DNRGB (WLED UDP Realtime) protocol for sending raw RGB pixel data.
   • The client is configured with the WLED device's IP address and DDP port (4048).

3. Integrate WLED Client into HyperionGrabber:

   • Modified HyperionGrabber to use WledClient instead of HyperionClient.
   • Updated the _processFrame method to call WledClient's sendImage method.

4. Remove HyperionClient Dependency:

   • Removed the HyperionClient class and its related code.

Step-by-Step: Screen to LEDs (Direct WLED Communication)

1. Screen Capture (HyperionGrabber):

   • The WaylandGrabber (or an X11 grabber in the X11 version) continuously captures frames from your screen.
   • Each captured frame is received by the HyperionGrabber's _processFrame method as a QVideoFrame.

2. Image Processing (HyperionGrabber):

   • The QVideoFrame is converted into a QImage.
   • Scaling: The QImage is scaled down. The _scale_m parameter (default 8) determines the scaling factor. For example, if your screen is 1920x1080 and _scale_m is 8, the image will be scaled to 240x135. This is done using Qt::SmoothTransformation for quality.
   • Format Conversion: The scaled image is converted to QImage::Format_RGB888 (24-bit RGB), which is a standard format for LED data.
   • Frame Skipping (Optional): If configured, some frames might be skipped to reduce the processing load.

3. Direct WLED Data Transmission (WledClient):

   • The HyperionGrabber passes the processed (scaled and formatted) QImage directly to the WledClient's sendImage method.
   • The WledClient then:
     • Iterates through the pixels of the QImage.
     • Constructs DNRGB packets (WLED UDP Realtime protocol):
       • Each packet starts with a 4-byte header:
         • Byte 0: Protocol type (4 for DNRGB).
         • Byte 1: Timeout value (e.g., 1 second, after which WLED returns to its normal mode if no more packets are received).
         • Byte 2 & 3: 16-bit starting LED index (low byte, then high byte). This allows sending parts of the image in multiple packets if the image is too large for a single UDP datagram.
       • The RGB data for each pixel (3 bytes per pixel: Red, Green, Blue) is appended to the header.
     • Sends these UDP datagrams to the configured WLED device's IP address and port (default 4048).

4. WLED Device Processing:

   • The WLED device receives the DNRGB UDP packets.
   • It interprets the pixel data and updates the connected LEDs accordingly.
   • Any internal WLED effects or configurations (like color correction, gamma, or specific LED mapping) will be applied by the WLED firmware itself.

Parameters for Testing with Your LED Setup:

The Hyperion_Grabber_X11_QT application now directly communicates with WLED.

Parameters you need to consider for the Hyperion_Grabber_X11_QT application:

• --address or -a: The IP address or hostname of your WLED device. (e.g., 192.168.1.177 as set in the code, but you should use your WLED's actual IP).
• --port or -p: The UDP port of your WLED device for the UDP Realtime protocol. (Default: 4048 for DDP, or 21324 for WARLS, but we are using DDP/DNRGB which is typically 4048).
• --scale or -s: The scaling factor for the captured image. This is crucial for performance vs. accuracy.
  • Recommendation: Start with the default 8. If the performance is good and you want more detail, you can try 4 or 2. If performance is an issue, you might even go higher (e.g., 16).
• --frameskip or -f: Number of frames to skip between processing. (Default: 0 - no skipping). You can increase this if you experience performance issues.
• --inactive or -i: Time in seconds after which the grabber logs an inactivity message. (Default: 0 - no inactivity timer). Note: This currently only logs a message; it does not send a black frame or clear LEDs to WLED. This functionality would need to be added to WledClient if desired.

Example Command to run the grabber:

./Hyperion_Grabber_X11_QT -a <YOUR_WLED_IP_ADDRESS> -p 4048 -s 8 -f 0

Replace <YOUR_WLED_IP_ADDRESS> with the actual IP address of your WLED device.

For your LED configuration (30 at the bottom, 20 at the right, 70 at the top, 20 at the left, and another 41 at the bottom):

This configuration is still primarily handled within the WLED device itself. You will need to configure the LED segments and their order within the WLED web interface to match your physical LED layout.

• Total LEDs: 30 + 20 + 70 + 20 + 41 = 181 LEDs.
• Ensure your WLED device is configured for this total number of LEDs and that the segments are correctly defined to match your physical setup.
• Enable "UDP Realtime" in WLED settings and confirm it's listening on port 4048 (or the port you specify with -p).

Summary of what you need to do for testing:

1. Configure WLED Device:
   • Ensure your WLED device has the correct total number of LEDs configured.
   • Set up LED segments in WLED to match your physical layout (bottom, right, top, left).
   • Enable "UDP Realtime" in WLED settings and confirm it's listening on the correct port (default 4048).
2. Build the Grabber: You will need to compile the modified Hyperion_Grabber_X11_QT project.
3. Run the Grabber: Execute the compiled application with the appropriate WLED IP address and port.

Color Order Troubleshooting and Resolution

Problem: Initial tests showed only red LEDs, despite sending a sequence of red, green, blue, white, and black.

Diagnosis:

• WLED's /json/cfg endpoint showed hw.led.ins[0].order: 0, which typically means RGB color order.
• Our WledClient was initially sending RGB, but then a test was performed with a GRB swap based on user input. This caused a mismatch.

Resolution:

• Reverted the R and G channel swap in wledclient.cpp to ensure data is sent in RGB order.
• Confirmed that WLED is listening on UDP port 21324 for Realtime data (from /json/cfg if.sync.port0).
• After rebuilding and re-running wled_test, the LEDs successfully displayed the full color sequence (Red, Green, Blue, White, Black) as expected.

Conclusion: The issue was a mismatch between the sent color order (RGB) and the WLED's expected color order (RGB, confirmed by configuration). The wled_test program now correctly demonstrates full color control.

Current Problem: Flickering with Static Screen

Observation: When running Hyperion_Grabber_Wayland_QT, some LEDs change color even when the screen is visually static, leading to flickering.

Diagnosis:

• Analysis of output.log revealed that the scaledImage (the input to WledClient) is not perfectly static, even when the screen appears still. The WaylandGrabber or underlying system introduces subtle pixel variations, causing the image checksum to constantly change.
• Our initial image difference threshold (_changeThreshold_m = 100) was too low, allowing these minor, visually imperceptible changes to trigger new frames to be sent to WLED.

Proposed Solution:

• Increase the _changeThreshold_m significantly to filter out the noise from the screen capture. This will prevent sending new frames to WLED unless there's a visually significant change on the screen.

Ambilight TV Concept

Ambilight TV, primarily a Philips technology, enhances the viewing experience by projecting dynamic, colored light onto the wall behind the television. This is achieved using intelligent LED lights positioned around the edges of the screen.

Key Principles:

1. Real-time Color Matching: The core idea is to analyze the colors displayed on the TV screen in real-time and project corresponding hues onto the surrounding wall. For example, if the left side of the screen shows a bright blue sky, the LEDs on that side will emit a matching blue light.
2. Extended Immersion: This creates the illusion that the picture extends beyond the physical boundaries of the TV, making the screen appear larger and drawing the viewer deeper into the content.
3. Reduced Eye Strain: By softening the contrast between the bright TV screen and a dark room, Ambilight can help reduce eye strain and fatigue.
4. Dynamic and Adaptive: The lights are not static; they change dynamically with the on-screen content, reacting to color shifts, movement, and even audio (in some modes).

How it Works (Generalizing for our project):

• Screen Analysis: The system continuously captures and analyzes the video frame displayed on the screen.
• Edge/Region Sampling: Instead of analyzing every single pixel, the system typically samples colors from specific regions or "zones" along the edges of the screen. These zones correspond to the physical placement of the LEDs.
• Color Averaging/Dominant Color Extraction: For each sampled zone, a dominant color or an average color is calculated.
• LED Control: This calculated color is then sent to the corresponding LED(s) or LED segment(s) on the back of the TV, which illuminate the wall with that color.

Relevance to Hyperion_Grabber_X11_QT

Our Hyperion_Grabber_X11_QT project aims to replicate this Ambilight functionality by:

• Screen Capture: The WaylandGrabber (or X11 grabber) continuously captures frames from your screen.
• Image Processing (HyperionGrabber): The captured QVideoFrame is converted to a QImage and then scaled down (scaledImage). This scaledImage represents a lower-resolution version of the entire screen.
• Color Extraction/Mapping (Implicit in WledClient): This is where the crucial step of mapping the scaledImage to the individual LEDs happens. The WledClient iterates through the pixels of the scaledImage and sends them as RGB data to the WLED device.

The Problem with Left-Side LEDs:

Given the Ambilight concept, if the LEDs on the left side are not showing colors that match the left side of the screen, it strongly suggests an issue in one of these areas:

1. Incorrect Sampling/Mapping in WledClient: The WledClient might be iterating through the scaledImage pixels in an order that doesn't correspond to the physical layout of the LEDs. For example, it might be reading pixels from the top-left to bottom-right of the scaledImage and sending them sequentially, but the physical LEDs might be arranged starting from the bottom-left, then up the left side, across the top, and down the right.
2. WLED Configuration Mismatch: Even if the WledClient sends the data correctly, the WLED device itself needs to be configured to interpret that incoming stream of RGB data correctly for its physical LED layout. If WLED expects a different order (e.g., starting from the top-left LED and going clockwise), but our client sends data starting from the bottom-left, there will be a mismatch.
3. Scaling/Aspect Ratio Issues: While less likely to cause a complete side mismatch, incorrect scaling or aspect ratio handling could subtly distort the sampled colors.

WLED API Information

Based on research of WLED API documentation (JSON and UDP Realtime):

• WLED JSON API (HTTP - /json/info):

  • Provides detailed, read-only information about the WLED device via a GET request to http://<WLED_IP_ADDRESS>/json/info.
  • Key information available:
    • leds.count: Total number of configured LEDs.
    • info.udpport: The UDP port WLED is listening on for Realtime data (default 21324, but can be changed).

• WLED UDP Realtime API (WARLS/DNRGB):

  • Confirms that DNRGB (protocol type 4) is the correct protocol for sending LED data.
  • The default UDP port for Realtime data is 21324, not 4048 (which is DDP). This is a critical finding and a potential mismatch with our current code, which uses 4048 by default.

Implications for Diagnostic Script:

To improve the diagnostic script, we can:

1. Fetch json/info: Use web_fetch to get the json/info from the WLED device.
2. Extract leds.count and info.udpport: Parse the JSON response to get the total LED count and the configured UDP port.
3. Use extracted values: Use these values in our diagnostic flashing commands, potentially overriding user-provided -p or providing a default count for --flash.

Confirmed WLED Configuration

• IP Address: 192.168.178.69
• Total LEDs: 181
• UDP Port for Realtime Data: 21324

New Strategy: Dedicated WLED Configuration & LED Testing Tool

Goal: Create a standalone tool that can reliably connect to a WLED device, read its configuration, save it, and provide interactive LED testing (flashing in steps/batches).

Phase 1: WLED Configuration Retrieval

• Step 1: Implement WLED Configuration Client (C++):
  • Create a new C++ class (e.g., WledConfigClient) that focuses solely on interacting with the WLED JSON API over HTTP.
  • This client will be responsible for making GET requests to endpoints like /json/info and parsing the JSON response.
  • It will handle potential network errors and provide clear feedback.
• Step 2: Create a Command-Line Tool for Configuration Retrieval:
  • Develop a new executable (e.g., wled_config_tool) that uses WledConfigClient.
  • This tool will take the WLED IP address as a command-line argument.
  • It will attempt to retrieve the WLED configuration (total LEDs, UDP port, segment info if available) and print it to stdout.
  • It will also have an option to save this configuration to a file (e.g., wled_config.json).

Phase 2: Interactive LED Testing

• Step 3: Integrate LED Testing into wled_config_tool:
  • Extend wled_config_tool to include functionality for flashing LEDs.
  • It will reuse the WledClient (or a modified version of it) to send DNRGB packets.
  • The tool will take arguments for startIndex, count, and color, similar to our previous --flash attempt.
  • It will use the retrieved WLED configuration (total LEDs, UDP port) as defaults or for validation.
• Step 4: Interactive Flashing and Mapping:
  • You will run wled_config_tool with flashing commands.
  • You will report which physical LEDs illuminate, and we will use this to build a precise mapping of LED indices to physical locations.
  • This mapping will be documented in lessons_learned.md.

LED Demo Mode Implementation

Goal: Implement a demo mode in wled_config_tool to sequentially light up LEDs with random colors, turn them off, and repeat.

Challenges & Solutions:

1. Blocking Delays (QThread::msleep()):

   • Problem: Initial attempts used QThread::msleep() for delays, which blocked the Qt event loop, preventing UDP packets from being sent and processed correctly by the WLED device. This resulted in only a brief, single flash of all LEDs.
   • Solution: Implemented a non-blocking delay() helper function using QEventLoop and QTimer. This allows the Qt event loop to continue processing events (like network communication) during delays.

2. Sequential Lighting (Single LED Update):

   • Problem: Initially, the demo mode only lit up one LED at a time because WledClient::setLedsColor was called for individual LEDs, and the WLED device would time out and turn off previous LEDs before the next one was lit.
   • Solution: Modified WledClient::setLedsColor to accept a QVector<QColor> representing the entire LED strip's state. In runDemoMode, a QVector<QColor> (currentColors) is maintained, and in each iteration, the color of the current LED is added to this vector. Then, client.setLedsColor(currentColors) is called, sending a single DDP packet with the state of all currently lit LEDs. This ensures that previously lit LEDs remain on.

Implementation Details:

• Added --demo (-d) command-line option to wled_config_tool.
• Implemented runDemoMode helper function to encapsulate the demo logic.
• Used std::srand and std::rand for random color generation.
• The demo mode lights up LEDs sequentially with random colors, then turns them all off, and repeats indefinitely until Ctrl+C is pressed.

Phase 3: Integrate into HyperionGrabber

• Step 5: Refine HyperionGrabber's LED Mapping: Once the LED mapping is definitively established and documented, we will integrate this knowledge into HyperionGrabber to correctly sample screen regions and send data to WLED.

The Case of the Vanishing Code: A Cautionary Tale

Problem: During development, the implementation of the --demo and interactive flashing functionality in wled_config_tool was lost. The features were well-documented in this file, but the corresponding code disappeared from all existing branches.

Diagnosis: Extensive searching of the git history using various commands (git log -S, git log -p, etc.) failed to locate the commit that removed the code. The code was not present on the master branch or the wled-color-fix feature branch, where it was expected to be. The most likely cause is that the branch containing the feature was either deleted or reset, or the commit was reverted in a non-obvious way.

Resolution: The functionality had to be re-implemented from scratch, using this lessons_learned.md file as the specification.

Lessons Re-Learned:

• Documentation is not a substitute for code: While detailed documentation is invaluable, it's useless if the code it describes doesn't exist.
• Git hygiene is critical: Ensure that feature branches are not deleted prematurely and that commits are not lost. Use git push to back up branches to a remote repository.
• When searching for lost code:
  • git log -S is a powerful tool, but it can be misled by documentation files. Use pathspec exclusions (:(exclude)filename) to narrow the search.
  • If searching fails, a direct inspection of the files on relevant branches (git checkout <branch> && cat <file>) is the most reliable way to determine what code is actually present.

Re-implementing the WLED Config Tool Features

Goal: Re-implement the --demo and --flash functionality in wled_config_tool.

Process:

1. Extended WledClient: Added a new method setLedsColor(const QVector<QColor> &colors) to WledClient. This method sends the state of the entire LED strip in a single UDP packet, which is crucial for avoiding flickering and state loss issues, as learned from the previous implementation.

2. Re-implemented wled_config_tool features:

   • Added the --demo and --flash command-line options.
   • Implemented the runDemoMode function to sequentially light up LEDs with random colors, using a non-blocking delay.
   • Implemented the runFlashMode function to light up a user-defined range of LEDs with a specific color.

3. Build System Fixes: The build process revealed several issues that needed to be addressed:

   • Missing Qt Module: The wled_config_tool target was missing a dependency on the Qt6::Gui module, which was required for QImage and QColor. This was fixed by adding Qt6::Gui to the target_link_libraries in CMakeLists.txt.
   • Deprecated Functions: The code was using the deprecated Qt functions qrand() and qsrand(). These were replaced with the standard C++ functions rand() and srand() from <cstdlib>.
   • Linker Errors: The wled_config_tool was not being linked with wledclient.cpp, resulting in "undefined reference" errors. This was resolved by adding wledclient.cpp to the add_executable section for wled_config_tool in CMakeLists.txt.

   Outcome: The wled_config_tool is now fully functional with the re-implemented demo and flash modes, and the build system is more robust.

Further Refinements to WLED Config Tool: Stability, Flickering, and Visualization

Goal: Address recurring segmentation faults, eliminate LED flickering, and provide a visual ASCII representation of the configured LED layout.

Problems Encountered & Solutions:

1. Persistent Segmentation Faults:

   • Cause: The interactive layout configuration, initially implemented with a state machine and asynchronous input, suffered from critical memory management issues. Specifically, QObject instances (like WledConfigClient, QSocketNotifier, WledClient, QVector, QTimer) were being created on the heap but were not properly parented to a QObject (like QCoreApplication app), leading to premature destruction or dangling pointers when accessed from asynchronous callbacks (lambdas). Additionally, QTextStream objects were being captured by value in lambdas, making them const and preventing non-const operations.
   • Resolution:
     • All dynamically allocated QObject instances within the interactive configuration were explicitly parented to the QCoreApplication app object (new MyObject(&app)). This ensures their lifetime is managed by the Qt event loop and they are automatically deleted when the application exits.
     • The interactive input logic was simplified. Instead of a complex state machine with asynchronous input for each number, the tool now prompts for all LED counts and the offset upfront using blocking cin >> calls. This simplifies the control flow and eliminates the need for complex asynchronous state management during input, making the code more robust.
     • QTextStream objects (cin and cout) are now created locally within the QSocketNotifier::activated lambda, ensuring they are always valid when used.

2. LED Flickering in Interactive Mode:

   • Cause: The previous implementation used a QTimer to send a full LED data packet every second to keep the WLED device in real-time mode. This constant refreshing, even with identical data, caused a noticeable flicker on the LEDs.
   • Resolution:
     • Research into the WLED UDP Realtime protocol confirmed that the second byte of the DNRGB packet specifies a timeout in seconds.
     • The WledClient::setLedsColor method was modified to accept a timeout parameter.
     • In the interactive layout configuration, the LED data is now sent only once with a significantly longer timeout (e.g., 60 seconds). This eliminates the need for periodic refresh packets and thus resolves the flickering. The QTimer previously used for refreshing was removed.

3. ASCII Layout Visualization:

   • Goal: Provide a clear, terminal-based visual representation of the configured LED layout.
   • Implementation: A new function printAsciiLayout was added. This function takes the LED counts for each side, the offset, and the total number of LEDs. It generates a rectangular ASCII art representation of the layout using ANSI escape codes to display colored squares (Red for Bottom, Green for Right, Blue for Top, Yellow for Left). This visualization is printed to the console after the LEDs are lit up on the physical device.

Outcome: The wled_config_tool is now significantly more stable, the LED flickering issue is resolved, and users can visually confirm their configured layout directly in the terminal.