[AI Dev Tools] LLM Security Scanner, Code Efficiency Benchmark, and Repository Analysis

Agentic Security is an open-source tool designed to scan and test Large Language Models (LLMs) for vulnerabilities and potential security risks.

• Performs comprehensive fuzzing and employs various attack techniques to test LLMs, including customizable rule sets and agent-based attacks.
• Integrates multiple existing security tools and datasets, such as Garak, InspectAI, and llm-adaptive-attacks, to provide a robust testing framework.
• Offers LLM API integration and stress testing capabilities, allowing users to test their own LLM implementations.
• Provides a user-friendly interface for managing scans and visualizing results.
• Allows users to add custom datasets and extend the tool's functionality through CSV files or by implementing new data loaders.
• Can be integrated into CI/CD pipelines for automated security checks during development processes.

ENAMEL (EfficeNcy AutoMatic EvaLuator) is a benchmark for evaluating the efficiency of code generated by LLMs, addressing a gap in existing evaluation frameworks that primarily focus on functional correctness.

• The benchmark introduces a new efficiency metric called eff@k, generalizing the pass@k metric from correctness to efficiency and handling right-censored execution time.
• An unbiased and variance-reduced estimator of eff@k is derived using Rao--Blackwellization, with a numerically stable implementation provided.
• ENAMEL employs human expert-designed algorithms and implementations as reference solutions, setting a high standard for efficiency evaluation.
• Rigorous evaluation is ensured through human-curated test case generators that filter out incorrect code and differentiate suboptimal algorithms.
• An extensive study across 30 popular LLMs using ENAMEL reveals that current models struggle with designing advanced algorithms and implementation optimization, falling short of generating expert-level efficient code.