LocatAI 1.3.4-beta

LocatAI: AI-Powered Web Element Locator

<div align="center"> <img src="./assets/locatai-logo.png" alt="LocatAI Logo" width="200"/> <br> <strong>Find any web element using natural language</strong> <br> <br> </div>

📑 Table of Contents

• Overview
• Why LocatAI?
• Key Features
• Getting Started
• Installation
• Configuration
• Basic Usage
• How LocatAI Works
• Advanced Features
• Best Practices
• Troubleshooting
• Metrics and Analytics
• FAQ
• Security Considerations
• License
• Contributing
• Support

📖 Overview

LocatAI is a revolutionary Selenium extension that leverages Large Language Models to dynamically locate web elements with unprecedented accuracy. Traditional element location methods like XPath and CSS selectors break when web UIs change, but LocatAI adapts intelligently by understanding elements based on their context, appearance, and purpose.

<table> <tr> <th>Traditional Approach</th> <th>LocatAI Approach</th> </tr> <tr> <td>

// Complex, brittle XPath
driver.FindElement(By.XPath("//div[@class='login-form']//button[contains(@class, 'btn-primary') and contains(text(), 'Login')]")).Click();

// Breaks if class names change
var productItems = driver.FindElements(By.CssSelector(".inventory_item"));
foreach (var item in productItems) {
    var title = item.FindElement(By.CssSelector(".inventory_item_name")).Text;
    if (title == "Sauce Labs Backpack") {
        item.FindElement(By.CssSelector(".btn_inventory")).Click();
        break;
    }
}

</td> <td>

// Simple and readable
driver.FindElementByLocatAI("login button").Click();

// Resilient to UI changes
driver.FindElementByLocatAI("add to cart button for Sauce Labs Backpack").Click();

</td> </tr> </table>

🚀 Getting Started in 5 Minutes

1. Install the package

   dotnet add package LocatAI
   

2. Add minimal configuration (create appsettings.json)

   {
     "AI": {
       "Provider": "openai",
       "ApiKey": "your-api-key"
     }
   }
   

3. Use LocatAI in your code

   using LocatAI.Extensions;
   
   // Initialize with configuration
   ElementFinder.Initialize(configuration);
   
   // Find elements by description
   driver.FindElementByLocatAI("login button").Click();
   

👉 That's it! Continue reading for advanced features and best practices.

📊 Approach Comparison

🌟 Key Features

• 🤖 AI-Powered Element Finding: Locate elements with natural language descriptions
• 🌐 Multi-Provider Support: Works with OpenAI, Anthropic, Google Gemini, and Ollama
• 💾 Intelligent Caching: Dramatically reduces API costs and speeds up execution
• 🔍 Smart Context Analysis: Understands page structure and element relationships
• 🔄 Self-Healing Locators: Adapts to website changes automatically
• 📊 Performance Metrics: Tracks success rates and execution times
• 🔒 Confidence Scoring: Configurable thresholds for reliable element matching
• 🚦 Element Verification: Ensures correct elements are found
• 🎨 Visual Highlighting: See which elements are found during execution

🔄 Compatibility

📦 Installation

NuGet Package

dotnet add package LocatAI

Package References

<ItemGroup>
  <PackageReference Include="LocatAI" Version="1.0.0" />
</ItemGroup>

⚙️ Configuration

<details> <summary><b>📚 Complete Configuration Reference</b></summary>

{
  "AI": {
    "Provider": "openai",         // Options: "openai", "anthropic", "gemini", "ollama"
    "ApiKey": "your-api-key-here",
    "Model": "gpt-4",            // Options depend on provider selected
    "BaseUrl": "https://api.openai.com/v1",  // Optional custom endpoint
    "Temperature": 0.2,          // Lower values = more deterministic outputs
    "MaxTokens": 2048,           // Maximum response length
    "RequestTimeoutSeconds": 30   // API request timeout
  },
  "Browser": {
    "Default": "Chrome",         // Options: "Chrome", "Firefox", "Edge", "Safari"
    "GlobalOptions": {
      "Headless": false,
      "Arguments": ["--disable-gpu", "--no-sandbox"],
      "BinaryLocation": "",      // Optional custom browser path
      "ImplicitWaitSeconds": 5
    },
    "Chrome": {                  // Browser-specific options
      "Arguments": ["--start-maximized"]
    }
  },
  "LocatAI": {
    "EnableCache": true,         // Enable element locator caching
    "CacheTTLMinutes": 60,      // Cache time-to-live
    "EnableDetailedLogging": true,
    "LogLevel": "Debug",         // Options: "Trace", "Debug", "Info", "Warning", "Error"
    "LogHtmlContent": false,     // Log page HTML (caution: verbose)
    "LogAIMessages": true,       // Log prompts and responses
    "MinimumConfidence": 0.8,    // Confidence threshold (0.0-1.0)
    "MaxLocatorsToTry": 3,       // Maximum number of locators to attempt
    "TimeoutMilliseconds": 10000, // Timeout for finding elements
    "EnableLocatorRetry": true,  // Retry with AI if element not found
    "MaxRetryAttempts": 2,       // Maximum retry attempts
    "EnableHighlighting": true,  // Highlight found elements
    "HighlightDurationMs": 300,  // Highlight duration
    "HighlightBorderWidth": 2,   // Highlight border width
    "UseSmartHighlighting": true, // Adapt highlight color to element
    "EnablePerformanceTracking": true,
    "LogPerformanceMetrics": true
  }
}

</details>

Programmatic Configuration

// Create options programmatically
var options = new LocatAIOptions {
    MinimumConfidence = 0.8,
    MaxLocatorsToTry = 3,
    EnableDetailedLogging = true,
    TimeoutMilliseconds = 10000,
    EnableCache = true,
    EnableHighlighting = true,
    HighlightDurationMs = 300
};

// Create AI provider
var aiProvider = new OpenAIProvider(
    apiKey: "your-api-key",
    model: "gpt-4",
    temperature: 0.2,
    maxTokens: 2048
);

// Initialize with options and provider
ElementFinder.Initialize(aiProvider, options);

🚀 Basic Usage

Dependency Injection Setup

// Program.cs or Startup.cs
using Microsoft.Extensions.Configuration;
using Microsoft.Extensions.DependencyInjection;
using LocatAI.DependencyInjection;

var configuration = new ConfigurationBuilder()
    .AddJsonFile("appsettings.json")
    .Build();

var services = new ServiceCollection()
    .AddLocatAI(configuration)    // Add LocatAI services
    .AddBrowser(configuration);   // Add Browser factory

var serviceProvider = services.BuildServiceProvider();

Finding Elements with Natural Language

using LocatAI.Extensions;
using OpenQA.Selenium;

// Get WebDriver from service provider
var browserFactory = serviceProvider.GetRequiredService<IBrowserFactory>();
var driver = browserFactory.CreateDriver();

try {
    // Navigate to website
    driver.Navigate().GoToUrl("https://www.saucedemo.com/");

    // Find elements by natural language description
    IWebElement usernameField = driver.FindElementByLocatAI("username input field");
    usernameField.SendKeys("standard_user");

    IWebElement passwordField = driver.FindElementByLocatAI("password field");
    passwordField.SendKeys("secret_sauce");

    IWebElement loginButton = driver.FindElementByLocatAI("login button");
    loginButton.Click();

    // Find elements with more context
    IWebElement inventoryItem = driver.FindElementByLocatAI("add to cart button for Sauce Labs Backpack");
    inventoryItem.Click();

    // Descriptive element finding
    IWebElement cartBadge = driver.FindElementByLocatAI("the shopping cart badge showing number of items");
    Assert.AreEqual("1", cartBadge.Text);
}
finally {
    driver.Quit();
    driver.Dispose();
}

Asynchronous Element Finding

// Async version for better performance
IWebElement usernameField = await driver.FindElementByLocatAIAsync("username input field");

// Find multiple elements
IReadOnlyCollection<IWebElement> productItems = await driver.FindElementsByLocatAIAsync("all product items on the page");

🧠 How LocatAI Works

LocatAI uses a sophisticated approach to locate elements:

1. Context Analysis: Captures current page HTML and analyzes the DOM structure
2. AI Prompt Engineering: Constructs optimized prompts with your element description
3. Locator Generation: Uses AI to generate multiple potential locators (XPath, CSS selectors)
4. Confidence Scoring: Assigns confidence scores to each locator based on uniqueness and specificity
5. Verification Loop: Tests locators in order of confidence, validates found elements
6. Smart Caching: Stores successful locators with their page context fingerprints
7. Adaptive Healing: When existing locators fail, adapts by generating new ones

🔍 Advanced Features

Element Verification

// Configure verification for better reliability
driver.ConfigureLocatAIVerification(new VerificationOptions {
    VerifyVisibility = true,          // Check if element is visible
    RequireVisibility = true,         // Require element to be visible
    VerifyText = true,                // Verify text content matches description 
    VerifyAttributes = true,          // Verify attributes match description
    VerifyElementType = true,         // Verify element type matches description
    EnableVisualVerification = false, // Use visual AI verification (experimental)
    MaxVerificationAttempts = 3       // Maximum verification attempts
});

// Find element with verification enabled
var submitButton = driver.FindElementByLocatAI("submit button");

// Temporarily disable verification
using (driver.DisableLocatAIVerification())
{
    var invisibleElement = driver.FindElementByLocatAI("hidden element in page");
}

Smart Waiting

// Wait for element to be present
driver.WaitForElementByLocatAI("confirmation message", timeoutSeconds: 10);

// Wait for element with condition
driver.WaitForElementByLocatAI("submit button", e => e.Enabled, "Button to be enabled");

// Custom wait conditions
driver.WaitUntilElementIsClickable("submit button");
driver.WaitUntilElementContainsText("status message", "Completed");
driver.WaitUntilElementDisappears("loading spinner");
driver.WaitUntilPageContainsText("Welcome to your account");

Advanced Browser Interactions

// Page interactions
driver.RefreshPageSafely();
driver.NavigateAndWaitForLoad("https://example.com");
driver.WaitForPageToLoad();
driver.ExecuteJavaScript("return document.title");

// Scrolling
driver.ScrollToElement(element);
driver.ScrollToTop();
driver.ScrollToBottom();
driver.ScrollElementIntoView(element);

// Mouse interactions
driver.HoverOverElement(element);
driver.ClickAndHold(element);
driver.RightClick(element);
driver.DoubleClick(element);
driver.DragAndDrop(sourceElement, targetElement);
driver.DragAndDropByOffset(element, 100, 50);

// Form interactions
driver.SelectByText(dropdownElement, "Option Text");
driver.SelectByValue(dropdownElement, "value1");
driver.ClearAndType(inputElement, "New Text");
driver.TypeSlowly(inputElement, "Typing with delay", 50); // 50ms delay

// Tab/window management
driver.OpenNewTab();
driver.SwitchToTab(1);
driver.CloseCurrentTab();
driver.SwitchToWindowByTitle("Payment Page");

// Alert handling
driver.AcceptAlert();
driver.DismissAlert();
driver.GetAlertText();
driver.SendKeysToAlert("text");

// Frame handling
driver.SwitchToFrameByLocatAI("payment iframe");
driver.SwitchToDefaultContent();

Performance Optimization

// Cache management
ElementFinder.OptimizeCache();       // Remove low-performing entries
ElementFinder.ClearCache();          // Clear entire cache
ElementFinder.ExportCache("cache.json"); // Export cache for persistence
ElementFinder.ImportCache("cache.json"); // Import previously saved cache

// Performance reporting
var metrics = ElementFinder.GetPerformanceMetrics();
Console.WriteLine($"Success Rate: {metrics.SuccessRate}");
Console.WriteLine($"Average Location Time: {metrics.AverageLocationTimeMs}ms");
Console.WriteLine($"Cache Hit Rate: {metrics.CacheHitRate}");

Test Framework Integration

// NUnit integration
[Test]
public void TestLogin()
{
    using (driver.SetNUnitContext()) // Captures test context
    {
        // Test code here
    }
}

// MSTest integration
[TestMethod]
public void TestLoginMSTest()
{
    using (driver.SetMSTestContext(TestContext))
    {
        // Test code here
    }
}

// xUnit integration
[Fact]
public void TestLoginXUnit()
{
    using (driver.SetXUnitContext(GetType().Name, nameof(TestLoginXUnit)))
    {
        // Test code here
    }
}

AI Provider Customization

// Using different AI providers

// OpenAI
ElementFinder.Initialize(new OpenAIProvider(
    apiKey: "your-openai-key",
    model: "gpt-4",
    temperature: 0.2
));

// Anthropic (Claude)
ElementFinder.Initialize(new AnthropicProvider(
    apiKey: "your-anthropic-key",
    model: "claude-3-sonnet-20240229"
));

// Google Gemini
ElementFinder.Initialize(new GeminiProvider(
    apiKey: "your-gemini-key",
    model: "gemini-pro"
));

// Ollama (local model)
ElementFinder.Initialize(new OllamaProvider(
    baseUrl: "http://localhost:11434",
    model: "llama3"
));

// Custom provider
public class CustomAIProvider : IAIProvider
{
    public async Task<string> GenerateResponseAsync(
        string userPrompt, 
        string systemPrompt, 
        CancellationToken cancellationToken = default)
    {
        // Custom implementation
        return "AI response";
    }
}

ElementFinder.Initialize(new CustomAIProvider());

📝 Logging

// Configure custom logger
LocatAILogger.Initialize(
    new LoggerConfiguration()
        .MinimumLevel.Debug()
        .WriteTo.Console(
            outputTemplate: "{Timestamp:yyyy-MM-dd HH:mm:ss} [{Level:u3}] {Message:lj}{NewLine}{Exception}"
        )
        .WriteTo.File("logs/locatai-.log", rollingInterval: RollingInterval.Day)
        .CreateLogger()
);

// Set log level at runtime
LocatAILogger.SetLogLevel(LogLevel.Debug);

// Enable specific logging features
LocatAILogger.EnableHtmlLogging(true);  // Log HTML snippets (verbose)
LocatAILogger.EnableAILogging(true);    // Log AI prompts and responses
LocatAILogger.EnableMetricsLogging(true); // Log performance metrics

🧪 Best Practices

Element Description Guidelines

• Be Specific: "login button" → "blue Sign In button at top-right corner"
• Include Context: "price" → "product price in the checkout summary"
• Mention Attributes: "submit button" → "submit button with 'Complete Order' text"
• Describe Appearance: "menu" → "hamburger menu icon with three horizontal lines"
• Use Location: "link" → "Help link in the footer navigation"

Performance Optimization

1. Enable Caching: Keep EnableCache set to true for repeated tests
2. Adjust Confidence: Lower MinimumConfidence (e.g., 0.6) for non-critical elements
3. Limit Retries: Set MaxRetryAttempts to 1 for faster execution
4. Use Async Methods: Prefer FindElementByLocatAIAsync for better performance
5. Balance Detail: More detailed descriptions improve accuracy but require more AI processing

Reliability Tips

1. Wait Strategically: Use WaitForElementByLocatAI instead of fixed Thread.Sleep
2. Enable Verification: Keep verification enabled for critical workflows
3. Handle Popups: Check for and dismiss common overlays before interactions
4. Use Smart Retries: Configure EnableLocatorRetry to handle intermittent failures
5. Log Failures: Enable detailed logging to troubleshoot problematic elements

🛠️ Troubleshooting

Common Issues

Debugging

// Enable detailed logging
LocatAILogger.SetLogLevel(LogLevel.Debug);

// Log HTML for troubleshooting
LocatAILogger.EnableHtmlLogging(true);

// Get detailed information about a specific element
var debugInfo = driver.GetLocatAIElementDebugInfo("login button");
Console.WriteLine(debugInfo);

// Track specific element location attempts
using (LocatAILogger.CreateContextScope("Login Button Finder"))
{
    var loginButton = driver.FindElementByLocatAI("login button");
}

📊 Metrics and Analytics

// Get performance metrics
var metrics = ElementFinder.GetPerformanceMetrics();

Console.WriteLine($"Total Elements Located: {metrics.TotalElementsLocated}");
Console.WriteLine($"Success Rate: {metrics.SuccessRate * 100}%");
Console.WriteLine($"Average Location Time: {metrics.AverageLocationTimeMs}ms");
Console.WriteLine($"Cache Hit Rate: {metrics.CacheHitRate * 100}%");
Console.WriteLine($"API Calls Made: {metrics.TotalAPICalls}");
Console.WriteLine($"API Cost Estimate: ${metrics.EstimatedAPICost}");

// Export metrics to file
ElementFinder.ExportMetrics("metrics.json");

📖 FAQ

Q: Does LocatAI work with any website?
A: Yes, LocatAI works with any website that Selenium supports, including modern single-page applications, legacy sites, and complex dynamic UIs.

Q: How much does it cost to use LocatAI with OpenAI's API?
A: With intelligent caching enabled, most tests cost only a few cents to run. After initial runs, cached locators reduce API calls by up to 95%.

Q: Can I use LocatAI with local AI models?
A: Yes, LocatAI supports Ollama for running local models like Llama 3, eliminating API costs and addressing privacy concerns.

Q: Will LocatAI slow down my tests?
A: Initial element location may take 1-2 seconds, but cached locators are as fast as traditional selectors. Overall, tests become more reliable and require less maintenance.

Q: How does LocatAI handle dynamic content and AJAX?
A: LocatAI intelligently waits for page updates and can generate locators that adapt to dynamic content changes.

🔒 Security Considerations

• API keys are never logged or exposed in error messages
• HTML content is processed locally with sensitive data masking options
• Support for local AI models via Ollama removes cloud API requirements
• Cache encryption is available for sensitive test environments

📜 License

This project is licensed under the MIT License - see the LICENSE.md file for details.

🤝 Contributing

Contributions are welcome! Please read CONTRIBUTING.md for details on our code of conduct and the process for submitting pull requests.

📬 Support

For questions, issues, or feature requests:

• Open an issue
• Join our Discord community
• Email us at support@locatai.dev

<div align="center"> <p>Made with ❤️ by the LocatAI Team</p> <p>⭐ Star us on GitHub if you find this project useful! ⭐</p> </div>